Averages hide the tail
One healthy average can coexist with a handful of slow events that users actually feel.
eBPF-powered Linux performance diagnostics
Find the wait behind the latency.
rand-fast measures scheduler latency for a specific process, turning a vague slowdown into a signal you can investigate.
$ sudo fast sched --pid 1234 --duration 10s
Example output
PID: api-server (1234)Duration: 10s
Scheduler latency
p5021 µs
p95840 µs
p997.2 ms
max84.1 ms
Slow events> 1ms 421> 10ms 38> 50ms 4
From runnable to running, measured at the kernel boundary.
The problem
Latency is the symptom.
The queue is the clue.
CPU graphs can tell you that a service is busy. They rarely tell you why one request waited longer than the next. Scheduler latency makes that hidden wait measurable.
Host-wide data loses the culprit
Start with the process that regressed, then follow its threads and the CPUs they wait for.
Kernel signals add context
Wakeups and context switches show the gap between work becoming runnable and actually running.
How it works
Follow the scheduling path.
Three kernel events. One answer to the question: how long did this process wait?
sched_wakeupWork becomes runnable
rand-fast records when a target thread is woken and ready to compete for CPU time.
runnableThe wait is measured
The elapsed time between wakeup and execution becomes a scheduler latency event.
sched_switchWork starts running
Events are aggregated by percentile, threshold, process, and CPU for a useful signal.
v0.1 / Scheduler
A focused signal for a noisy system.
Start with the process you care about. Keep the output small enough to act on.
01Process-first
Measure the workload that hurts
Focus on one PID and its threads instead of drowning in host-wide averages.
02Tail-aware
See the latency tail
p50, p95, p99, and max expose the waits that an average keeps hidden.
03CPU context
Keep the CPU in the picture
Break scheduler latency down by CPU to reveal where runnable work is waiting.
04Kernel signal
Observe the scheduling path
Trace wakeups and switches with eBPF, without adding application instrumentation.
Quick start
Start with one process.
Point fast at a PID, give it a measurement window, and inspect the scheduler latency tail.
shell
$ sudo fast sched --pid 1234 --duration 10sThe roadmap
One signal at a time.
One diagnosis closer.
Scheduler latency is the first instrument. The roadmap expands rand-fast from a focused probe into a complete Linux performance flight recorder.
Scheduler
Measure per-process scheduler latency and CPU contention.
- Scheduler latency
- PID-level CLI
- Percentile statistics
CPU
Connect CPU saturation, hot stacks, and scheduler pressure.
- CPU usage
- On-CPU profiling
- Hot stacks
I/O
Find slow devices and process-level I/O bottlenecks.
- Disk latency
- Block devices
- Process I/O
Network
Follow connection latency from the process to the remote endpoint.
- TCP RTT
- Retransmissions
- Connection latency
Off-CPU
Show where threads wait when they are not running.
- Futex waits
- Lock contention
- Off-CPU stacks
Memory
Connect memory pressure to real application slowdowns.
- PSI
- Page faults
- Swap pressure
Diagnose
Rank multiple signals and explain the most likely cause.
- Signal correlation
- Ranked causes
- One command
Flight Recorder
Keep low-cost observability running until an incident needs detail.
- Continuous capture
- Incident preservation
- Low overhead
The first measurement is the fastest way in.