Skip to content

Top-down analysis with the perf tool

Note, the description here is for Linux 6.1. Earlier versions of the perf tool supported top-down analysis but due to naming inconsistencies in old versions, Linux 6.1 is focused upon on this page.

What is top-down analysis?

Top-down analysis is an approach for identifying software performance bottlenecks. It is described in A Top-Down method for performance analysis and counters architecture but is often come across in Appendix B of Intel's Optimization Reference Manual in their software development manuals. The approach gathers a group of metrics and from these a problematic metric is identified. A group of metrics associated with the problematic metric are then measured drilling down to the metric/issue that identifies the performance problem.

More recent Intel processors feature improvements for top-down analysis, the most recent being Timed Process Event Based Sampling (TPEBS). A description for using perf from Intel is given here.

Starting at the top

In this tutorial perf bench mem memcpy will be used for the <benchmark>. First of all gather the level 1 metrics:

  $ perf stat -M TopdownL1 <benchmark>

The output will be like:

  $ perf stat -M TopdownL1 perf bench mem memcpy

  # Running 'mem/memcpy' benchmark:
  # function 'default' (Default memcpy() provided by glibc)
  # Copying 1MB bytes ...

         5.677689 GB/sec
  # function 'x86-64-unrolled' (unrolled memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         5.425347 GB/sec
  # function 'x86-64-movsq' (movsq-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         6.141903 GB/sec
  # function 'x86-64-movsb' (movsb-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         7.454676 GB/sec

   Performance counter stats for 'perf bench mem memcpy':

          72,813,380      TOPDOWN.SLOTS                    #     25.5 %  tma_retiring        
                                                    #     44.3 %  tma_backend_bound   
                                                    #     24.3 %  tma_frontend_bound  
                                                    #      6.0 %  tma_bad_speculation 
          18,560,273      topdown-retiring                 #     25.5% Retiring              
          17,703,645      topdown-fe-bound                 #     24.3% Frontend Bound        
              43,171      INT_MISC.UOP_DROPPING                                              
          31,980,778      topdown-be-bound                 #     43.9% Backend Bound         
              55,052      cpu/INT_MISC.RECOVERY_CYCLES,cmask=1,edge/                                   
           4,568,682      topdown-bad-spec                 #      6.3% Bad Speculation       

         0.012820482 seconds time elapsed

         0.004290000 seconds user
         0.008581000 seconds sys

The TopdownL1 metrics may be computed by default if no metric is provided:

  $ perf stat perf bench mem memcpy
  # Running 'mem/memcpy' benchmark:
  # function 'default' (Default memcpy() provided by glibc)
  # Copying 1MB bytes ...

        16.837284 GB/sec
  # function 'x86-64-unrolled' (unrolled memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

        16.551907 GB/sec
  # function 'x86-64-movsq' (movsq-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

        19.148284 GB/sec

   Performance counter stats for 'perf bench mem memcpy':

               21.38 msec task-clock                       #    0.927 CPUs utilized             
                   7      context-switches                 #  327.406 /sec                      
                   0      cpu-migrations                   #    0.000 /sec                      
               6,247      page-faults                      #  292.187 K/sec                     
          94,684,914      cycles                           #    4.429 GHz                       
         117,788,965      instructions                     #    1.24  insn per cycle            
          25,949,650      branches                         #    1.214 G/sec                     
             252,622      branch-misses                    #    0.97% of all branches           
                          TopdownL1                 #     45.8 %  tma_backend_bound      
                                                    #     11.2 %  tma_bad_speculation    
                                                    #     18.2 %  tma_frontend_bound     
                                                    #     24.8 %  tma_retiring           

         0.023069998 seconds time elapsed

         0.000000000 seconds user
         0.023455000 seconds sys

On the right of the counter values are the metrics. For TopdownL1 there are the metrics tma_retiring, tma_backend_bound, tma_frontend_bound and tma_bad_speculation. For the benchmark tma_backend_bound is the largest. We can drill down into this metric by adding the suffix _group to the metric name:

  $ perf stat -M tma_backend_bound_group perf bench mem memcpy

  # Running 'mem/memcpy' benchmark:
  # function 'default' (Default memcpy() provided by glibc)
  # Copying 1MB bytes ...

         6.781684 GB/sec
  # function 'x86-64-unrolled' (unrolled memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         5.033827 GB/sec
  # function 'x86-64-movsq' (movsq-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         6.688784 GB/sec
  # function 'x86-64-movsb' (movsb-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         6.829108 GB/sec

   Performance counter stats for 'perf bench mem memcpy':

          69,746,420      TOPDOWN.SLOTS                    #     24.9 %  tma_core_bound      
                                                    #     17.5 %  tma_memory_bound    
          18,599,045      topdown-retiring                 #     26.7% Retiring              
           1,253,770      EXE_ACTIVITY.BOUND_ON_STORES                                       
          17,504,983      topdown-fe-bound                 #     25.1% Frontend Bound        
           1,769,443      EXE_ACTIVITY.1_PORTS_UTIL                                          
          29,266,144      topdown-be-bound                 #     42.0% Backend Bound         
              55,050      cpu/INT_MISC.RECOVERY_CYCLES,cmask=1,edge/                                   
           2,934,845      CYCLE_ACTIVITY.STALLS_MEM_ANY                                      
           6,667,954      CYCLE_ACTIVITY.STALLS_TOTAL                                        
           1,775,168      EXE_ACTIVITY.2_PORTS_UTIL                                          
           4,376,245      topdown-bad-spec                 #      6.3% Bad Speculation       

         0.012655913 seconds time elapsed

         0.008451000 seconds user
         0.004225000 seconds sys

This time tma_core_bound is the largest TMA metric and so we drill down in to it:

  $ perf stat -M tma_core_bound_group perf bench mem memcpy
  # Running 'mem/memcpy' benchmark:
  # function 'default' (Default memcpy() provided by glibc)
  # Copying 1MB bytes ...

         6.510417 GB/sec
  # function 'x86-64-unrolled' (unrolled memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         6.065606 GB/sec
  # function 'x86-64-movsq' (movsq-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         7.512019 GB/sec
  # function 'x86-64-movsb' (movsb-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         6.781684 GB/sec

   Performance counter stats for 'perf bench mem memcpy':

        70,285,910      TOPDOWN.SLOTS                    #     31.2 %  tma_ports_utilization
        18,467,278      topdown-retiring                 #     26.3% Retiring              
         2,165,618      cpu/EXE_ACTIVITY.3_PORTS_UTIL,umask=0x80/                                   
        17,364,754      topdown-fe-bound                 #     24.7% Frontend Bound        
         1,754,496      EXE_ACTIVITY.1_PORTS_UTIL                                          
        30,043,781      topdown-be-bound                 #     42.7% Backend Bound         
        14,057,182      CPU_CLK_UNHALTED.THREAD          #      0.1 %  tma_divider         
         3,054,356      CYCLE_ACTIVITY.STALLS_MEM_ANY                                      
         6,685,779      CYCLE_ACTIVITY.STALLS_TOTAL                                        
         1,767,046      EXE_ACTIVITY.2_PORTS_UTIL                                          
         4,410,096      topdown-bad-spec                 #      6.3% Bad Speculation       
             9,354      ARITH.DIVIDER_ACTIVE                                               

       0.011282941 seconds time elapsed

       0.000000000 seconds user
       0.011349000 seconds sys

And then tma_ports_utilization:

  $ perf stat -M tma_ports_utilization_group perf bench mem memcpy

  # Running 'mem/memcpy' benchmark:
  # function 'default' (Default memcpy() provided by glibc)
  # Copying 1MB bytes ...

         6.554111 GB/sec
  # function 'x86-64-unrolled' (unrolled memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...  

         5.710892 GB/sec
  # function 'x86-64-movsq' (movsq-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         6.467301 GB/sec
  # function 'x86-64-movsb' (movsb-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

         6.300403 GB/sec

   Performance counter stats for 'perf bench mem memcpy':

         1,812,959      RESOURCE_STALLS.SCOREBOARD       #     16.6 %  tma_ports_utilized_0  (34.73%)
         1,991,726      cpu/EXE_ACTIVITY.3_PORTS_UTIL,umask=0x80/                                     (34.73%)
        14,159,441      CPU_CLK_UNHALTED.THREAD                                              (34.73%)
         6,689,757      CYCLE_ACTIVITY.STALLS_TOTAL                                          (34.73%)
         3,838,402      CYCLE_ACTIVITY.STALLS_MEM_ANY                                        (34.73%)
         3,282,823      UOPS_EXECUTED.CYCLES_GE_3        #     22.9 %  tma_ports_utilized_3m  (52.98%)
        14,324,185      CPU_CLK_UNHALTED.THREAD                                              (52.98%)
        14,599,955      CPU_CLK_UNHALTED.THREAD          #     12.5 %  tma_ports_utilized_2  (65.27%)
         1,823,495      EXE_ACTIVITY.2_PORTS_UTIL                                            (65.27%)
         1,819,926      EXE_ACTIVITY.1_PORTS_UTIL        #     12.5 %  tma_ports_utilized_1  (79.65%)
        14,591,940      CPU_CLK_UNHALTED.THREAD                                              (79.65%)

       0.012931961 seconds time elapsed

       0.008647000 seconds user
       0.004323000 seconds sys

For tma_ports_utilization_group we can see numbers like (34.73%) that indicate there were insufficient performance counters to gather the metric and the performance counters had to be multiplexed during the benchmark run. Multiplexing lowers accuracy and can be worked around by just measuring the metric on its own:

  $ perf stat -M tma_ports_utilized_0 perf bench mem memcpy
  ...
         2,268,815      RESOURCE_STALLS.SCOREBOARD       #     19.8 %  tma_ports_utilized_0

Finally we see tma_ports_utilized_3m as the largest metric. Looking at perf list, (sometimes perf list -v) we can see the metrics meaning:

  • tma_ports_utilized_3m

    This metric represents fraction of cycles CPU executed total of 3 or more uops per cycle on all execution ports (Logical Processor cycles since ICL, Physical Core cycles otherwise). Sample with: UOPS_EXECUTED.CYCLES_GE_3

The 'Sample with' event can be used with perf record to identify where in the benchmark the performance bottleneck is:

  $ perf record -e UOPS_EXECUTED.CYCLES_GE_3 perf bench mem memcpy; perf report

  # Running 'mem/memcpy' benchmark:
  # function 'default' (Default memcpy() provided by glibc)
  # Copying 1MB bytes ...

        23.251488 GB/sec
  # function 'x86-64-unrolled' (unrolled memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

        23.251488 GB/sec
  # function 'x86-64-movsq' (movsq-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

        28.722426 GB/sec
  # function 'x86-64-movsb' (movsb-based memcpy() in arch/x86/lib/memcpy_64.S)
  # Copying 1MB bytes ...

        27.901786 GB/sec
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.034 MB perf.data (1 samples) ]

  # To display the perf.data header info, please use --header/--header-only options.  
  #
  #
  # Total Lost Samples: 0  
  #
  # Samples: 1  of event 'UOPS_EXECUTED.CYCLES_GE_3/period=1000000/'
  # Event count (approx.): 2000003 
  #
  # Overhead  Command     Shared Object  Symbol         
  # ........  ..........  .............  ...............  
  #
     100.00%  mem-memcpy  perf           [.] memcpy_orig