Messages in this thread | | | From | (Eric W. Biederman) | Date | Tue, 27 Mar 2018 10:59:50 -0500 | Subject | Re: [PATCH net-next v2 0/2] kernel: add support to collect hardware logs in crash recovery kernel |
| |
Rahul Lakkireddy <rahul.lakkireddy@chelsio.com> writes:
> On Tuesday, March 03/27/18, 2018 at 18:47:34 +0530, Eric W. Biederman wrote: >> Rahul Lakkireddy <rahul.lakkireddy@chelsio.com> writes: >> >> > On Saturday, March 03/24/18, 2018 at 20:50:52 +0530, Eric W. Biederman wrote: >> >> >> >> Rahul Lakkireddy <rahul.lakkireddy@chelsio.com> writes: >> >> >> >> > On production servers running variety of workloads over time, kernel >> >> > panic can happen sporadically after days or even months. It is >> >> > important to collect as much debug logs as possible to root cause >> >> > and fix the problem, that may not be easy to reproduce. Snapshot of >> >> > underlying hardware/firmware state (like register dump, firmware >> >> > logs, adapter memory, etc.), at the time of kernel panic will be very >> >> > helpful while debugging the culprit device driver. >> >> > >> >> > This series of patches add new generic framework that enable device >> >> > drivers to collect device specific snapshot of the hardware/firmware >> >> > state of the underlying device in the crash recovery kernel. In crash >> >> > recovery kernel, the collected logs are exposed via /sys/kernel/crashdd/ >> >> > directory, which is copied by user space scripts for post-analysis. >> >> > >> >> > A kernel module crashdd is newly added. In crash recovery kernel, >> >> > crashdd exposes /sys/kernel/crashdd/ directory containing device >> >> > specific hardware/firmware logs. >> >> >> >> Have you looked at instead of adding a sysfs file adding the dumps >> >> as additional elf notes in /proc/vmcore? >> >> >> > >> > I see the crash recovery kernel's memory is not present in any of the >> > the PT_LOAD headers. So, makedumpfile is not collecting the dumps >> > that are in crash recovery kernel's memory. >> > >> > Also, are you suggesting exporting the dumps themselves as PT_NOTE >> > instead? I'll look into doing it this way. >> >> Yes. I was suggesting exporting the dumps themselves as PT_NOTE >> in /proc/vmcore. I think that will allow makedumpfile to collect >> your new information without modification. >> > > If I export the dumps themselves as PT_NOTE in /proc/vmcore, can the > crash tool work without modification; i.e can crash tool extract these > notes?
I believe crash would need to be taught about these notes. This is something new.
However "readelf -a random_elf_file" does display elf notes, and elf notes in general are not hard to extract.
What I expect from an enconding in ELF core dump format is a way to captuer the data, a way to encode the data, and a way to transport the data to the people who care. Analysis tools are easy enough after the fact.
Eric
| |