Skip to content
Snippets Groups Projects
Select Git revision
  • riscv-sqoscfg-rfc-v2
  • riscv-cbqri-rfc-v2
  • riscv-cbqri-rfc
  • master default protected
  • v6.3-rc6
  • v6.3-rc5
  • v6.3-rc4
  • v6.3-rc3
  • v6.3-rc2
  • v6.3-rc1
  • v6.2
  • v6.2-rc8
  • v6.2-rc7
  • v6.2-rc6
  • v6.2-rc5
  • v6.2-rc4
  • v6.2-rc3
  • v6.2-rc2
  • v6.2-rc1
  • v6.1
  • v6.1-rc8
  • v6.1-rc7
  • v6.1-rc6
  • v6.1-rc5
24 results
Search by author
  • Any Author
  • Adrien Ricciardi Adrien Ricciardi RICCIARDI-Adrien-BayLibre
  • Alexandre Bailon Alexandre Bailon abailon
  • Andrew Stubbs Andrew Stubbs ams
  • Anne Françoise Gros Anne Françoise Gros afgros
  • BayLibreCI BayLibreCI baylibreCI
  • Bernhard Rosenkränzer Bernhard Rosenkränzer bero
  • Carlo Caione Carlo Caione carlo.caione
  • Christophe Baylibre Christophe Baylibre BLChristopheCBO
  • Drew Fustini Drew Fustini dfustini
  • Guillaume LA ROQUE Guillaume LA ROQUE glaroque
  • Guillaume Pain Guillaume Pain gpain
  • Guillaume Ranquet Guillaume Ranquet granquet
  • Jeremy Robinson Jeremy Robinson jrobinson
  • Jerome Brunet Jerome Brunet jbrunet
  • Julien Masson Julien Masson JulienMasson
  • Julien Morin Julien Morin jmorin2
  • Julien Stephan Julien Stephan jstephan
  • Jérémie Garcia Jérémie Garcia jgarcia
  • Kevin Hilman Kevin Hilman khilman
  • Markus Schneider-Pargmann Markus Schneider-Pargmann msp8
20 authors
  1. Feb 18, 2023
  3. Feb 14, 2023
    • Daniel Bristot de Oliveira's avatar
      rtla: Add hwnoise tool · 1f428356
      Daniel Bristot de Oliveira authored 
      The hwnoise tool is a special mode for the osnoise top tool.

hwnoise dispatches the osnoise tracer and displays a summary of the noise.
The difference is that it runs the tracer with the OSNOISE_IRQ_DISABLE
option set, thus only allowing only hardware-related noise, resulting in
a simplified output. hwnoise has the same features of osnoise.

An example of the tool's output:

 # rtla hwnoise -c 1-11 -T 1 -d 10m -q
                                           Hardware-related Noise
 duration:   0 00:10:00 | time is in us
 CPU Period       Runtime        Noise  % CPU Aval   Max Noise   Max Single          HW          NMI
   1 #599       599000000          138    99.99997           3            3           4           74
   2 #599       599000000           85    99.99998           3            3           4           75
   3 #599       599000000           86    99.99998           4            3           6           75
   4 #599       599000000           81    99.99998           4            4           2           75
   5 #599       599000000           85    99.99998           2            2           2           75

Link: https://lkml.kernel.org/r/2d6f49a6f3a4f8b51b2c806458b1cff71ad4d014.1675805361.git.bristot@kernel.org



Signed-off-by: default avatarDaniel Bristot de Oliveira <bristot@kernel.org>
Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Clark Williams <williams@redhat.com>
Cc: Bagas Sanjaya <bagasdotme@gmail.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      1f428356
  5. Feb 02, 2023
    • Daniel Bristot de Oliveira's avatar
      rtla/timerlat: Add auto-analysis support to timerlat top · 5def33df
      Daniel Bristot de Oliveira authored 
      Currently, timerlat top displays the timerlat tracer latency results, saving
the intuitive timerlat trace for the developer to analyze.

This patch goes a step forward in the automaton of the scheduling latency
analysis by providing a summary of the root cause of a latency higher than
the passed "stop tracing" parameter if the trace stops.

The output is intuitive enough for non-expert users to have a general idea
of the root cause by looking at each factor's contribution percentage while
keeping the technical detail in the output for more expert users to start
an in dept debug or to correlate a root cause with an existing one.

The terminology is in line with recent industry and academic publications
to facilitate the understanding of both audiences.

Here is one example of tool output:
 ----------------------------------------- %< -----------------------------------------------------
  # taskset -c 0 timerlat -a 40 -c 1-23 -q
                                     Timer Latency
    0 00:00:12   |          IRQ Timer Latency (us)        |         Thread Timer Latency (us)
  CPU COUNT      |      cur       min       avg       max |      cur       min       avg       max
    1 #12322     |        0         0         1        15 |       10         3         9        31
    2 #12322     |        3         0         1        12 |       10         3         9        23
    3 #12322     |        1         0         1        21 |        8         2         8        34
    4 #12322     |        1         0         1        17 |       10         2        11        33
    5 #12322     |        0         0         1        12 |        8         3         8        25
    6 #12322     |        1         0         1        14 |       16         3        11        35
    7 #12322     |        0         0         1        14 |        9         2         8        29
    8 #12322     |        1         0         1        22 |        9         3         9        34
    9 #12322     |        0         0         1        14 |        8         2         8        24
   10 #12322     |        1         0         0        12 |        9         3         8        24
   11 #12322     |        0         0         0        15 |        6         2         7        29
   12 #12321     |        1         0         0        13 |        5         3         8        23
   13 #12319     |        0         0         1        14 |        9         3         9        26
   14 #12321     |        1         0         0        13 |        6         2         8        24
   15 #12321     |        1         0         1        15 |       12         3        11        27
   16 #12318     |        0         0         1        13 |        7         3        10        24
   17 #12319     |        0         0         1        13 |       11         3         9        25
   18 #12318     |        0         0         0        12 |        8         2         8        20
   19 #12319     |        0         0         1        18 |       10         2         9        28
   20 #12317     |        0         0         0        20 |        9         3         8        34
   21 #12318     |        0         0         0        13 |        8         3         8        28
   22 #12319     |        0         0         1        11 |        8         3        10        22
   23 #12320     |       28         0         1        28 |       41         3        11        41
  rtla timerlat hit stop tracing
  ## CPU 23 hit stop tracing, analyzing it ##
  IRQ handler delay:				      	    27.49 us (65.52 %)
  IRQ latency:						    28.13 us
  Timerlat IRQ duration:				     9.59 us (22.85 %)
  Blocking thread:					     3.79 us (9.03 %)
			objtool:49256    		     3.79 us
    Blocking thread stacktrace
		-> timerlat_irq
		-> __hrtimer_run_queues
		-> hrtimer_interrupt
		-> __sysvec_apic_timer_interrupt
		-> sysvec_apic_timer_interrupt
		-> asm_sysvec_apic_timer_interrupt
		-> _raw_spin_unlock_irqrestore
		-> cgroup_rstat_flush_locked
		-> cgroup_rstat_flush_irqsafe
		-> mem_cgroup_flush_stats
		-> mem_cgroup_wb_stats
		-> balance_dirty_pages
		-> balance_dirty_pages_ratelimited_flags
		-> btrfs_buffered_write
		-> btrfs_do_write_iter
		-> vfs_write
		-> __x64_sys_pwrite64
		-> do_syscall_64
		-> entry_SYSCALL_64_after_hwframe
  ------------------------------------------------------------------------
    Thread latency:					    41.96 us (100%)

  The system has exit from idle latency!
    Max timerlat IRQ latency from idle: 17.48 us in cpu 4
  Saving trace to timerlat_trace.txt
 ----------------------------------------- >% -----------------------------------------------------

In this case, the major factor was the delay suffered by the IRQ handler
that handles timerlat wakeup: 65.52 %. This can be caused by the
current thread masking interrupts, which can be seen in the blocking
thread stacktrace: the current thread (objtool:49256) disabled interrupts
via raw spin lock operations inside mem cgroup, while doing write
syscall in a btrfs file system.

A simple search for the function name on Google shows that this is
a legit case for disabling the interrupts:

  cgroup: Use irqsave in cgroup_rstat_flush_locked()
  lore.kernel.org/linux-mm/20220301122143.1521823-2-bigeasy@linutronix.de/

The output also prints other reasons for the latency root cause, such as:

  - an IRQ that happened before the IRQ handler that caused delays
  - The interference from NMI, IRQ, Softirq, and Threads

The details about how these factors affect the scheduling latency
can be found here:

   https://bristot.me/demystifying-the-real-time-linux-latency/

Link: https://lkml.kernel.org/r/3d45f40e630317f51ac6d678e2d96d310e495729.1675179318.git.bristot@kernel.org



Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: default avatarDaniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      5def33df
    • Daniel Bristot de Oliveira's avatar
      rtla/timerlat: Add auto-analysis core · 27e348b2
      Daniel Bristot de Oliveira authored 
      Currently, timerlat displays a summary of the timerlat tracer results
saving the trace if the system hits a stop condition.

While this represented a huge step forward, the root cause was not
that is accessible to non-expert users.

The auto-analysis fulfill this gap by parsing the trace timerlat runs,
printing an intuitive auto-analysis.

Link: https://lkml.kernel.org/r/1ee073822f6a2cbb33da0c817331d0d4045e837f.1675179318.git.bristot@kernel.org



Cc: Daniel Bristot de Oliveira <bristot@kernel.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: default avatarDaniel Bristot de Oliveira <bristot@kernel.org>
Signed-off-by: default avatarSteven Rostedt (Google) <rostedt@goodmis.org>
      27e348b2
  7. Feb 01, 2023
  9. Dec 10, 2022
  11. Aug 10, 2022
  13. Jul 31, 2022
  15. Jul 12, 2022
  17. May 26, 2022
  19. Mar 15, 2022
  21. Feb 26, 2022
  23. Feb 15, 2022
    • Masahiro Yamada's avatar
      kbuild: replace $(if A,A,B) with $(or A,B) · 5c816641
      Masahiro Yamada authored 
      
$(or ...) is available since GNU Make 3.81, and useful to shorten the
code in some places.

Covert as follows:

  $(if A,A,B)  -->  $(or A,B)

This patch also converts:

  $(if A, A, B) --> $(or A, B)

Strictly speaking, the latter is not an equivalent conversion because
GNU Make keeps spaces after commas; if A is not empty, $(if A, A, B)
expands to " A", while $(or A, B) expands to "A".

Anyway, preceding spaces are not significant in the code hunks I touched.

Signed-off-by: default avatarMasahiro Yamada <masahiroy@kernel.org>
Reviewed-by: default avatarNicolas Schier <nicolas@fjasle.eu>
      5c816641