Linux kernel panic root cause analysis
A per-case service for teams whose servers crash and whose vendors cannot or will not say why. You send what you have. We send back the cause, the evidence, and the next step.
What we analyse
- Kernel panics and oopses
- BUG and kernel BUG at, general protection fault, NULL pointer dereference, list corruption, invalid opcode, and page faults in kernel mode.
- Lockups and stalls
- Soft and hard lockups, hung tasks, RCU stalls, NMI watchdog resets, and hosts that freeze without producing a dump.
- Memory and I/O failures
- Out-of-memory kills in unexpected places, slab corruption, storage and network driver faults, and filesystem shutdowns.
- Spontaneous reboots
- Resets with no vmcore. We work from dmesg, journal, IPMI system event logs, pstore and serial console output to narrow the cause.
What we need from you
Start with a case summary. The more of the list below you can provide, the faster the answer.
- –A vmcore from kdump, or vmcore-dmesg.txt if the full dump is too large to share
- –Kernel version and distribution, and the Modules linked in line or lsmod output
- –dmesg, journal or console output from the minutes before the crash
- –Any out-of-tree modules in use: security sensors, backup drivers, live patching, monitoring agents
- –What changed recently: kernel update, agent upgrade, new hardware, new workload
What you get
- Written RCA report
- Annotated backtrace, the failing code path, and a plain-language account of what happened and why.
- Named responsible component
- The module, driver or kernel subsystem with its version, and the evidence that rules the others out.
- Escalation package
- A summary you can forward to the vendor, or to Red Hat, Canonical or SUSE support, written so their engineer can act on it.
- Mitigation and fix
- What to change today to stop the recurrence, and the permanent fix: a vendor patch, a kernel update or a configuration change.
How it works
- 01
Describe the crash
Submit a short case summary. An engineer is assigned and replies with a case ID, what to share, how to share it, and the charges. A vmcore from kdump is ideal; dmesg, console output or a photo of the panic screen is a fine start.
- 02
Hybrid analysis
Our tooling resolves symbols, parses the backtrace, and inspects locks and memory state. A kernel engineer validates the finding.
- 03
RCA delivered
You receive a short written report: annotated backtrace, root cause, fix, and interim mitigation such as a kernel parameter or driver pin.
How engagements work
- –Per case, no retainer. The assigned engineer reviews your summary and replies with the charges before any work starts.
- –A preliminary finding is shared as soon as the cause is clear, typically within hours of receiving the data. The full report follows.
- –Follow-up questions on the report are welcome. If the vendor pushes back, we help you answer them.
- –Patch development and ongoing kernel maintenance are separate engagements, quoted on request.
Common questions
- Do I need a vmcore?
- No. A vmcore gives the most definitive answer, but dmesg, console logs or a photo of the panic screen are enough to start, and often enough to name the culprit.
- Our kernel is tainted and the OS vendor declined the case. Can you still help?
- Yes. Tainted kernels are most of our work. The taint is the reason the vendor stops looking, not the reason the host crashed.
- Which distributions do you cover?
- RHEL, Rocky, AlmaLinux, Ubuntu, Debian, SUSE, Amazon Linux, Oracle Linux and mainline kernels.
- How is the vmcore transferred?
- The assigned engineer sends upload instructions with your case ID. Transfers are encrypted, and the dump is deleted 14 days after the case closes.
- Can you reproduce the crash or write the fix?
- We provide a reproduction recipe when the data allows it, and describe the fix precisely enough for the vendor or your own kernel team to apply it.
- Do you work with cloud instances and VMs?
- Yes. Most cloud providers support kdump in guests. Where a dump cannot be captured, we work from serial console output and the provider's instance logs.
Data security
A vmcore contains production memory. We handle it accordingly.
- –Dumps are encrypted in transit and at rest. Only the engineers on your case can read them.
- –vmcore files and extracted data are deleted 14 days after the case closes, or earlier on request.
- –A mutual NDA is available before you send anything.
Get in touch
Describe the crash or send a business query. Nothing to upload at this stage. An assigned engineer replies with a case ID, upload instructions and service charges.