IRQs: the Hard, the Soft, the Threaded and the Preemptible - Alison Chaiken, Peloton Technology Interrupt handlers manage responses to asynchronous hardware events, keep time and queue deferred work. Hard IRQs are straightforward functions that read status and schedule a slower "bottom half". The difficulty lies with management of the deferred work. Soft IRQS and tasklets are two different mechanisms. Tasklets handle most deferred work, but network, timer and block tasks are managed directly by softirqs. Confusion comes from the realization that the softirq interface via ksoftirqd also manages tasklets. In the RT kernel, hard IRQs themselves run as preemptible threads. What precisely are softirqs and why does the kernel have them? How does handling of deferred work differ with the PREEMPT_RT_FULL kernel? These and related mysteries will be illustrated via results from tools like bpf, event tracers and mpstat that show how to detect IRQ scheduling problems in x86, arm and arm64. About Alison Chaiken Alison has been an automotive systems programmer and kernel engineer since 2011 and has worked at Nokia, Mentor Graphics, and Peloton Technology. In 2014-2015, she collaborated on-site with a customer in Germany. Alison has spoken at events including ELC and ELCE, USENIX, Automotive Linux Summit, Southern California Linux Expo and Maker Faire. She organizes the monthly meetings of the 1800+-member, 15-year-old Silicon Valley Automotive Open Source Group.