PAPI FAQ

General Questions (FAQ)

ptools-perfapi@icl.utk.edu

Please note, this is a moderated group and if you are not subscribed to it, there may be a delay till the moderators get to your message and approve it.

Please see INSTALL.txt in the papi root directory.

First, read this document and the PAPI Documentation thoroughly. Then consult the PAPI Home Page at http://icl.utk.edu/papi. If that doesn't help, then search the archives as mentioned below. If that fails, then send mail to one of the two mailing lists, the users group at ptools-perfapi@icl.utk.edu or the PAPI-developers group perfapi-devel@icl.utk.edu. The former is a group for general announcements, questions and miscellaneous topics. The latter is is a discussion group for the developers of PAPI and it receives all CVS update messages. (which can be a significant amount of mail!)

There are currently two mailing lists, ptools-perfapi, which is a group for general announcements, questions and miscellaneous topics and perfapi-devel, which is a discussion group for the developers of PAPI and it receives all CVS update messages (which can be a significant amount of mail!)

To subscribe to or maintain your subscription to either of the above groups, go to:
https://groups.google.com/a/icl.utk.edu/forum/#!forum/ptools-perfapi  or https://groups.google.com/a/icl.utk.edu/forum/#!forum/perfapi-devel

The archives for the general PAPI mailing list are located at https://groups.google.com/a/icl.utk.edu/forum/#!forum/ptools-perfapi. The archives for the developers list are located at https://groups.google.com/a/icl.utk.edu/forum/#!forum/perfapi-devel.

See the Supported Architectures at https://github.com/icl-utk-edu/papi/wiki/Supported-Architectures.

Some of the more popular tools using PAPI are PaRSEC, Caliper, Kokkos, TAU, HPCToolkit , Score-P, Vampir, and Scalasca. If you have a tool to be posted, send it to the mailing list.

No. There is no PAPI version with MacOS support for CPU hardware counter monitoring. In addition to not supporting timer_create() and other POSIX realtime timers, MacOS does not export hardware counters that PAPI can read.

The PAPI Library

The warning message here is benign, but since it occurs on the last file to be compiled, it often looks like the build has been aborted. The reason the message occurs is that the compiler thinks it is trying to stuff too many bits into an integer value. You can fix it by rearranging the code a little bit. Or just download the latest copy of fmultiplex2.F from the cvs tree.

PAPI 3 represents a major upgrade to the PAPI library. Because of this, there have been a number of interface changes. The process to upgrade from PAPI 2 to PAPI 3 is straightforward, and documented in the PAPI Conversion Cookbook.

To compile with debugging, define CFLAGS to include -DDEBUG in the corresponding Makefile or Rules. file.

To enable debugging messages at run time, set the PAPI_DEBUG environment variable to one or more of the following with any character as a separator.

SUBSTRATE
API
INTERNAL
THREADS
MULTIPLEX
OVERFLOW
PROFILE
ALL

Also, see the man page for PAPI_set_debug().

On most systems, overflow must be emulated in software by PAPI. Only on the UltraSparc III, Itanium and IRIX does the operating system support true interrupt on overflow. Therefore the user is advised on most platforms to make sure the overflow value is no more than 1/1000th the clock rate. The emulation handler in PAPI runs every millisecond, therefore the goal of the tool designer should be to pick an value that will overflow frequently but not too frequently. Not following these guidelines could result in either the overflows never occurring or overflows occurring on every interrupt and thus resulting in a flat profile.

Call PAPI_stop, and then call PAPI_overflow, PAPI_profile or PAPI_sprofil with a threshold value of 0. Since PAPI 3 can overflow and profile on multiple events, you must call the above routines for EACH event that had been previously enabled for overflow or profile.

PAPI only tracks 'hardware events', the occurrence of signals onboard the microprocessor. It does not count system calls, software interrupts or other software events. The user should remember that by default, PAPI only measures events that occur in User Space.

Currently, PAPI only supports thread level measurements with kernel or bound threads. Each thread must create, manipulate and read its own counters. When a thread is created, it inherits no PAPI events or information from the calling thread.

When a process is created, it inherits no PAPI information from the calling thread.

Yes, but the counts will reflect the total events for the process. Measurements done in other threads will all get the same values, namely those counts for the total process. For non-bound threads, it is not necessary to call PAPI_thread_init. But in most scenarios like with SMP or OpenMP compiler directives, bound threads will be the default. For those using Pthreads, the user should take care to set the scope of each thread to the PTHREAD_SCOPE_SYSTEM attribute, unless the system is known to have a non hybrid thread library implementation, like Linux.

In PAPI2.0: Unless otherwise stated in the FAQ section for your platform, the encoding is as follows:

event = ((reg_code & 0xffffff) << 8 | (reg_num & 0xff))

In PAPI3.0: Just find the native event name and then call PAPI_event_name_to_code. The code returned can be added directly to an event set. The native events can be listed with the test case 'native_avail' in the ctests directory.

There are numerous patches designed to provide access to the Intel CPU performance counters. As PAPI began, we used the original Beowulf patch (perf) by David Hendriks. However, as PAPI progressed, we needed some addition features, which I graciously added. This patch used a system call approach and has proven to be exceedingly stable. Yes, no crashes reported. I knew that there was a better way to designed a performance counter kernel patch, one that used mmap() to provide direct access to the virtual counts. Mikael Pettersson provided me with exactly that in the form of the perfctr patch. It is also very, very stable. It can be found at http://user.it.uu.se/~mikpe/linux/perfctr. If you're starting with PAPI for the first time, we recommend the perfctr patch as included in the papi source distribution.

This is not a question, but I'll help you. We the PAPI developers cannot be experts on the 1000's of events found across all supported platforms. However, if you are using a PAPI preset, the first thing to do is to look up the corresponding native event code using the test case 'avail'. Then the best bet is to always go to the vendor's technical documentation site and check the processor reference manual. If you're convinced everything is kosher, then please feel free to send a message to the mailing list and one of the members may be able to help you.

My program runs fine with 1 or 2 counters, but when I add more I get a -8, PAPI_ECNFLCT error code. The error text says, "Event exists. but cannot be counted due to hardware resource limitations". What does this mean?

Many systems have only a few hardware performance counter registers thus you can only measure a few metrics at once. Some platforms may support counter multiplexing, which gives the user the illusion of a larger number of registers by time sharing the performance registers. On the R10K series, the IRIX kernel supports multiplexing, allowing up to 32 events to be counted at once. Don't take fine grained measurements when multiplexing, unless you know what you're doing.

You have either exceeded the number of available hardware counters or two or more of the events you want to count need the same resources. This can be particularly annoying on machines like the Pentium 4. Although the P4 has 18 nominal counter registers, many events require resources that are restricted to 2 or 3 of these counters. In practice it is often difficult to count more than 4 or 5 simultaneous events on this platform. One way around limited counter resources is to use multiplexing.

Many systems have only a few hardware performance counter registers; thus you can only measure a few metrics at once. Some platforms may support counter multiplexing, which gives the user the illusion of a larger number of registers by time sharing the performance registers. On the MIPS R10K series, the IRIX kernel supports multiplexing, allowing up to 32 events to be counted at once. On other platforms PAPI does the multiplexing itself, swapping events in and out of the counters based on a timer interrupt. Don't take fine grained measurements when multiplexing, unless you know what you're doing.

PAPI multiplexing currently always uses one hardware counter for Total Cycles. If you are trying to multiplex a derived event on hardware with only two physical counters then you will get a PAPI_ECNFLCT error. This happens on the Intel Pentium IIIs for example.

Also, enabling multiplexing is a two-step process. You must call PAPI_multiplex_init() to initialize multiplexing system-wide. You must also call PAPI_set_multiplex() for *each* event set that you want to count in multiplexed mode. If you try to add too many events to an event set where multiplexing has not been set, a PAPI_ECNFLCT error will result.

Hardware counters count low level events that can be directly measured in the hardware. Often these low level events must be combined to form meaningful PAPI preset events. This linear combination of low level events is called a derived PAPI event. Derived events are usually formed by adding or subtracting 2 'native' events, but occasionally derived events can contain 4 or more terms.

Hardware counters are a limited resource. Some PAPI preset events are derived, and require the use of more than one hardware counter. For example, Solaris has 2 counters, both of which are needed to count Floating point instructions. Flops also uses total cycles to measure time. On Solaris this would mean using 3 counters, and those resources aren't available.
If you get this error on any platform, run the avail program in the ctests directory and see how many native events have to be monitored. PAPI_num_counters() can be used to determine how many counters exist on your platform. If there are more native events than counters, then this is the reason you are getting the error.

The Fortran header file you include has to be preprocessed before the Fortran file can use it. To have the cpp process the file before sending the file to the compiler, add the -F flag. For example:

f90 -F test.F -o test

According to the Intel documentation, the counts from this event are not intuitive relating to it's description. Older releases of PAPI had this preset available in the Intel ports, but no longer. It does appear to work on the AMD Athlon.

SOme versions of GCC have a bug that is triggered by a statement in PAPI 3.0. This (one character) is fixed in the current tar ball, but may not be in the one you downloaded.

If you see an INTERNAL ERROR from GCC when compiling multiplex.c, do 2 things.

1) edit multiplex.c, line 1021 to have 2 equal signs instead of 1.

2 optional) send a message to your local gcc maintainer and complain.

The actual culprit is:

assert(retval = PAPI_OK) and it should be assert(retval == PAPI_OK)

Of course, both are legal C and nothing should trigger an internal compiler error, but hey...

P.S. If your current release compiled with GCC, you're still ok. As the statement above NEVER gets triggered. It is there as an artifact from the original multiplex.c implementation. So you don't need to change or upgrade your PAPI or gcc.

The EventSet MUST be set to PAPI_NULL before it is passed into PAPI_create_eventset.

TLS is thread local storage, a high performance mechanism in later GCC's/GLIBC/pthread to do constant time access to thread local storage. PAPI uses this if available.

However, many systems (especially IA64 running Debian or SuSE) provide very poor/buggy/non-existent support for this. If you're getting an error during compile (or seg faults on every program during the run), then please rebuild using ./configure.

Other systems don't bother to ship a gcc with this turned on, so you'll get the above error.

./configure has a test to make sure that the thread support is working on your platform.

If you find a case where configure did not detect a broken __thread implementation, please report it to us.

The PAPI GIT Source Repository

Yes. The latest copy of the PAPI source tree is viewable through a web based source browser accessible from the GitHub repository at:
https://github.com/icl-utk-edu/papi

Make sure git is installed on your machine. You can download a copy of git here.

Download the PAPI repository the first time with the following command:

> git clone https://github.com/icl-utk-edu/papi.git

This creates a complete copy of the papi git repository on your computer in a folder called 'papi'.

To make sure your copy is up to date with the repository:

> cd papi
> git pull 

You can always commit changes to your local copy of the PAPI respository using the "git commit" command and its variations. You cannot push those changes to the master copy of the repository without obtaining credentials from the PAPI team.

The web has a variety of resources targetted at teaching you how to use GIT. A good place to start is the official GIT site.
History and background of GIT can be found on Wikipedia.
This user-friendly introduction might help "git" you started.

PAPI on AIX POWER Processors

If you are running papi-3.0 on aix5.2 & power4 combo, and seeing failure. It is
most likely caused by the BUG in the KERNEL. You need look for efix for APAR IY57280, or
contact papi team at papi@cs.utk.edu for the fix. Here is the more precise info
from IBM:
the problam was introduced in 5.2 ML3, and fixed in 5.2 ML4 and 5.3.
 
To use PAPI in 64-bit mode on power4:
    make -f Makefile.aix-power4-64bit
        link your program with libpapi64.a or libpapi64.so
         
See: /usr/lpp/pmtoolkit/lib/<arch>.evs for POWER3;
     /usr/pmapi/lib/POWER4.evs and /POWER4.gps for POWER4
 
For threaded programs, you had better:
 
setenv AIXTHREAD_SCOPE S

AIX 4.3.x:
The current source and Makefile is for pmtoolkit 1.3.
If you have pmtoolkit 1.2 the test cases will fail. For example:
 
      ./tests/avail
      IOT trap
 
This can be remdied by recompiling the PAPI library with the option
-DPMTOOLKIT_1_2 set.
 
AIX 5.x:
The current source is for pmapi 1.4
 
The aix-power substrate is contained in a single source file, but targets
three different configurations.
Conditional compilation directed by three different make files determines
which configuration is targetted. Make sure you select the Makefile that
matches your configuration:
- Makefile.aix-power    for AIX 4.3.x on POWER3
- Makefile.aix5-power3  for AIX 5.x   on POWER3
- Makefile.aix-power4   for AIX 5.x   on POWER4

The POWER3 and POWER4 have a FMADD instruction. Although this instruction
performs two Floating Point operations, it is counted as one Floating Point
instruction. Because of this, there are situations where PAPI_FP_INS may
produce fewer Floating Point counts than expected.
Further, the Floating Point Instruction event on POWER3 and POWER4 also
counts Floating Point Stores, leading to higher Floating Point counts than
expected. There are occasions where these two effects can cancel each other
out, to produce the right result for the wrong reason!
Note that POWER3 and POWER4 also support an FMA counter (PAPI_FMA_INS).
Thus, a more accurate count of Floating Point Operations can be obtained
by PAPI_FP_INS + PAPI_FMA_INS.
Correcting for the overcount by Floating Point Stores is more difficult,
requiring the use of the native events: PM_FPU_LD_ST_ISSUES and PM_FPU_LD.
The complete expression for Floating Point Operations then becomes:
PAPI_FP_INS + PAPI_FMA_INS - (PM_FPU_LD_ST_ISSUES - PM_FPU_LD)

The POWER architecture supports up to 8 counters. However, in many cases
events are mutually exclusive and can't be counted simultaneously.
 
On POWER4, events are available only as members of predefined groups.
For more on these groups, see /usr/pmapi/lib/POWER4.gps.
 
The following table, submitted by Joel Malard, indicates
events that cannot be counted simultaneously on POWER3:

It is very important that you set the environment variable AIXTHREAD_SCOPE to "S", which disables user level threads.

Linux-IA64

This version of the substrate always scales PME_FP_OPS_RETIRED_HI, hex code 0xa, even if you are using it as a NATIVE event. Previous versions of PAPI did not scale this event and could produce erroneously low counts for PAPI_FP_OPS or PAPI_FP_INS.

PAPI_CA_SNP
PAPI_CA_INV
 Only counts snoops and invalidations from the local processor.
PAPI_TLB_TL
 Counts "real" TLB misses, i.e. misses that cause a VHPT walk or a TLB
 miss trap to the OS. Misses in the L1 TLBs are not counted.
PAPI_FP_STAL
 Counts stalls due to register dependencies and load latencies.
 If the FP pipeline can stall for some other reason (I don't know)
 then those stall cycles won't be counted.

Redhat broke the perfmon kernel interface in their kernels and thus only enabled it for root. In some kernels, its disabled entirely. You can test this by running your papi as root, if it then works, guess what, you have a broken kernel.

The fix is supposed to be in the latest update to RHEL3 and RHEL4. The best thing to do would be to download a kernel.org kernel, rebuild and go.

You are probably on an Altix or a system with a Redhat kernel. The solution for the later is replace the kernel you have with a patched kernel.org kernel, discussed in this section.

Please send us the kernel version if this happens to you. You'll notice it by running the profile_pthreads test case.

If you're an Altix user, then it's best to complain to SGI. But please let us know also.

The problem is not in PAPI, but in libpfm 3.x. When this library is built using icc, the file pfmlib_gen_ia64.c generates a series of errors. One workaround for this may be to make the libpfm library separately using gcc and then build PAPI with icc. Or just use gcc.

Linux-Perfctr

For Linux kernels more recent than 2.6.32, the perf_events interface is built into the kernel and can be used directly.

For Linux kernels before 2.6.32, PAPI requires your Linux kernel to be patched with either the PerfCtr patch or the Perfmon patch. For compatability reasons, we have included both of these patches in the tarball. You should patch your kernel with PerfCtr using the distribution found in the papi/src/perfctr-2.6.x directory for x86 hardware, and the papi/src/perfctr-2.7.x directory for IBM POWER hardware. If you prefer Perfmon for kernels older than 2.6.30, you should use the distribution found in the papi/src/libpfm-3.y. Prefmon is no longer supported as a Linux patch. The most recent Perfctr distribution can  be obtained from Mikael Petterson's web site although it is no longer actively supported and not guaranteed to work. http://user.it.uu.se/~mikpe/linux/perfctr/
If you're not sure how to patch, recompile and reinstall your linux kernel, there are a variety of resources on the web. Here's one that should help: http://answers.oreilly.com/topic/36-how-to-patch-a-linux-kernel/.

cd perfctr
more INSTALL
If you're getting compilation errors regarding not being able to find include files, then you're probably running a broken redhat installation.

Edit the path to your kernel include files at the top of either Makefile.linux-perfctr

If you have a properly functioning Perfctr patch from a previous release of PAPI, you will obviously not want to repatch your kernel. PAPI is compatible with PerfCtr 2.4.x and Perfctr 2.6.x.

The x86 Makefiles:
Makefile.linux-perfctr-p3
Makefile.linux-perfctr-p4
Makefile.linux-athlon
Makefile.linux-opteron

To recompile PAPI *not* using the included PerfCtr distribution, you simply pass the PERFCTR argument to the appropriate Makefile.

make -f Makefile.linux-perfctr-p3
PERFCTR=/usr/src/perfctr-2.4.x

To use Perfctr 2.6.x, simply type:
make -f Makefile.linux-perfctr-p3

To use the older version:
make -f Makefile.linux-perfctr-p3 VERSION=2.4.x

Easy huh?

See the INSTALL file in papi/src/perfctr-2.6.x. The instructions are very, very simple. Do not use perfctr-2.4.x unless you have to. There is no link of perfctr version to linux kernel version!

You are probably running udev, which is not smart enough to know the permissions of dynamically created devices. To fix this, find your udev/devices directory, often /lib/udev/devices or /etc/udev/devices and perform the following actions.

mknod perfctr c 10 182
chmod 644 perfctr

On Ubuntu 6.06 (and probably other debian distros), add a line to /etc/udev/rules.d/40-permissions.rules like this:

KERNEL=="perfctr", MODE="0666"

On SuSE, you may need to add something like the following to /etc/udev/rules.d/50-udev-default.rules: (SuSE does not have the 40-permissions.rules file in it.]

# cpu devices
KERNEL=="cpu[0-9]*", NAME="cpu/%n/cpuid"
KERNEL=="msr[0-9]*", NAME="cpu/%n/msr"
KERNEL=="microcode", NAME="cpu/microcode", MODE="0600"
KERNEL=="perfctr", NAME="perfctr", MODE="0644"

These lines tell udev to always create the device file with the appropriate permissions. Use 'perfex -i' from the perfctr distribution to test this fix.

YOU MUST COMPILE YOUR KERNEL WITH APIC SUPPORT IF YOU WANT INTERRUPT SUPPORT!
With Perfctr 2.3.3 or later it is possible to make the performance counters generate an interrupt when the counter reaches a certain count. This requires support in the Linux kernel, Perfctr, PAPI and the CPU to work properly.
The necessary kernel support is available if your kernel is compiled with SMP APIC support or uni-processor APIC support compiled in. This is true for 2.4-ac kernels and kernels 2.4.10 or later. This topic is discussed in more detail in Mikael Pettersson's installation instructions for PerfCtr.
Your CPU must be a Pentium 686/AMD K7 or similar which can generate APIC interrupts for performance counter events. This is _not_ true for some mobile Pentiums and early revisions of the AMD K7 or Athlon.
You can verify that all is working by running the perfctr/examples/perfex program with the -i flag. If you do not see "pcint" as one of the flags, you need to recompile your kernel or buy a real CPU. ;-)

Counting memory load and store instructions on the Pentium 4 is a two step process. First the desired events are tagged at the front of the pipeline. Then tagged events are counted as they graduate from the end of the pipeline. Unfortunately, the tags are all the same 'color' and can't be differentiated as they exit the pipe. Thus, you can correctly measure LD instructions, or correctly measure SR instructions, but if you try to measure them both at once, you will always get the sum of both operations in both counters. The same applies to PAPI_LST_INS.

This behavior is demonstrated in the test program ctests/p4_lst_ins.c.

The moral of the story is to always use these three events one-at-a-time on Pentium 4 machines.

The Pentium 4 can generate floating point instructions either through the x87 floating point unit or with SSE instructions.
Furthermore SSE can generate either packed (multiple operands in one 128-bit register) or unpacked (signal operand in one 128-bit register) instructions.
Depending on your compiler and settings you will get different instruction mixes.
 
PAPI provides 2 preset events to count floating point operations:
- PAPI_FP_INS counts intstructions passing through the floating point unit;
- PAPI_FP_OPS counts something closer to theoretical floating point operations.
 
To minimize the overlap and maximize the usefulness of these two events on Pentium 4, we have made the following choices:
- PAPI_FP_INS always counts only x87 floating point operations.
- PAPI_FP_OPS counts can be customized as discussed below.
 
Further complicating things is that the Pentium 4 hardware is too restrictive to count all these modes at once, so a decision must be made about what to count.
In order to enable PAPI to count these various mixes, we support 2 methods.
 
1) The PAPI_PENTIUM4_FP_xxx defines.
 
   Set these in the EVENTFLAGS of either the Makefile.linux-perfctr-p4 or
   Makefile.linux-perfctr-em64t.
 
   -DPAPI_PENTIUM4_FP_X87
   -DPAPI_PENTIUM4_FP_X87_SSE_SP
   -DPAPI_PENTIUM4_FP_X87_SSE_DP
   -DPAPI_PENTIUM4_FP_SSE_SP_DP
 
   The predefined value for Nocona/EM64T/Pentium 4 Model 3 is:
 
         -DPAPI_PENTIUM4_FP_X87_SSE_DP.
 
   The predefined value for anything else is:
 
         -DPAPI_PENTIUM4_FP_X87.
 
   If nothing is defined, the substrate defaults to:
 
         -DPAPI_PENTIUM4_FP_X87_SSE_DP.
 
2) The PAPI_PENTIUM4_FP environment variable.
 
   Set this to one or two of the following, and it will change the
   behavior of PAPI_FP_OPS.
 
   X87: count all x87 instructions
   SSE_SP: count all unpacked SSE single precision instructions
   SSE_DP: count all unpacked SSE double precision instructions
 
   Due to the design of the register set, only 2 of the three are countable
   at one time. Sorry folks.

PAPI can count 2 different types of vector instructions on the Pentium 4.
Either MMX instructions or packed SSE floating point instructions. These are supported with 2 methods, in a similar fashion to floating point events described above.
 
1) The PAPI_PENTIUM4_VEC_xxx defines.
 
   Set these in the EVENTFLAGS of either the Makefile.linux-perfctr-p4 or
   Makefile.linux-perfctr-em64t.
 
   -DPAPI_PENTIUM4_VEC_MMX
   -DPAPI_PENTIUM4_VEC_SSE
 
   The current default for all platforms is:
 
         -DPAPI_PENTIUM4_VEC_SSE.
 
   If nothing is defined, the substrate defaults to:
 
         -DPAPI_PENTIUM4_VEC_SSE.
 
2) The PAPI_PENTIUM4_VEC environment variable.
 
   Set this to either of the following, and it will change the
   behavior of PAPI_VEC_INS.
 
   SSE: count all packed SSE SP and DP instructions
   MMX: count all 64 and 128 bit MMX instructions

This is a known issue and we are looking in to the cause. Currently, we have no fix or work around.

(The following discussion does not apply to newer quad-core and higher Opteron processors)

The AMD Opteron is the first chip series from AMD that can measure and report floating point operations. Two native events measure floating point activity. One measures speculative operations that enter the FP units; the other measures operations that retire from the FP units.

The retired event generates precise event counts that scale with the amount of work done. However, it measures data movement as well as floating point operations, resulting in counts that are consistently significantly higher than the expected theoretical counts, often by factors of 2 or more.

The speculative event can be configured to generate counts of only the operations typically of interest. Since these counts are speculative, they tend to be higher by often widely variable amounts than expected theoretical counts, especially on complex production codes.

PAPI provides 2 preset events to count floating point operations:

- PAPI_FP_INS counts intstructions passing through the floating point unit;
- PAPI_FP_OPS is intended to count something closer to theoretical floating point operations.

To minimize the overlap and maximize the usefulness of these two events on AMD Opteron, we have made the following choices:

- PAPI_FP_INS always counts retired floating point operations. This value will be precise and accurate, but will include FP loads and stores as well as computations.

- PAPI_FP_OPS counts speculative computation operations by default, but can be customized as discussed below.

As an alternative to counting speculative computations, PAPI_FP_OPS can be configured to retired operations corrected for data movement. Unfortunately, the correction factors themselves are speculative, and can lead to undercounting errors similar in magnitude to those seen in the pure speculative counts.

Two methods are provided to allow customization of PAPI_FP_OPS:

1) The PAPI_OPTERON_FP_xxx defines.

Set these in the CFLAGS variable of Makefile.linux-perfctr-opteron.

-DPAPI_OPTERON_FP_RETIRED
-DPAPI_OPTERON_FP_SSE_SP
-DPAPI_OPTERON_FP_SSE_DP
-DPAPI_OPTERON_FP_SPECULATIVE

The default value is equivalent to:

-DPAPI_OPTERON_FP_SPECULATIVE.

2) The PAPI_OPTERON_FP environment variable.

Set this to one of the following, and it will change the behavior of PAPI_FP_OPS.

RETIRED: count all retired FP instructions
SSE_SP: correct retired counts optimized for single precision
SSE_DP: correct retired counts optimized for double precision
SPECULATIVE: count speculative computations (default)

Solaris-Ultra

Assembler stubs for get_tick() and cpu_sync() as well as the following defines have been blatantly stolen from the perfmon code. The author of the package "perfmon" is Richard J. Enbody and the home page for "perfmon" is http://www.cse.msu.edu/~enbody/perfmon.html. For *all* the native event names, run native_avail in the ctests subdirectory. For how to use the native event names, see native.c

1) Ultra I/II/III/III+ are currently supported;

2) Some of the cache events have documented bugs, see the Sun UltraSparc hardware reference manual.

3) WARNING FOR PEOPLE USING MULTITHREADED LIBRARIES ON SOLARIS 2.8: There is a bug that prevents setitimer() from being called after the process has called pthread() create at any point in time. Therefore if you suspect your communication library is multithreaded, you had better start the instrumentation before initializing it. See multiplex3_pthreads for details.

You didn't check the PAPI Supported Architectures. The hardware counters on SunOS withUltraSparc are only available on Sun OS 5.8 and above. That's Solaris 2.8 for you SVR4 people.