Messages in this thread | ![/](/images/icornerl.gif) | | From | Karim Yaghmour <> | Subject | Re: The case for a standard kernel debugger | Date | Mon, 9 Oct 2000 05:10:16 -0400 |
| |
richardj_moore@uk.ibm.com wrote: > > Completely agree - co-operation+integration is the order of the day.
Cool :)
> They > other thing I didn't mention was that the GKHI was substantially coded > before we discovered your hook capability. Part of the GKHI is also to > allow hooks to be dynamicaly defined i.e. to allow kernel modules to > declare hooks themselves, though I haven't yet implemented that - the > design is done.
Same here, as I said it was next on my list.
> The MP complient remark refers to the fact that the hook > mechanism must work under MP. Our hooks are implemented by modifying code > dynamically - that has certain serialisation requrements: you need to > ensure that other processors see a consistent view of memory, which means > that you need to stop them while you're change code dynamically and also > flush their I-fetch caches. There are some very odd (H/W) behavious that > can occur with self-modifying code if the appropriate measures are not > taken.
Well, this explains why I didn't understand what you meant by "MP compliant". LTT has no self-modifying code. Hence, there is no need to make special cases for that.
> > Our uniprocessor version of GKHI is being tested as I write. We hope to > release it in the next couple of weeks. As soon as we are happy with it I > will send you a copy so you can evaluate it against you hook methodology > and we can see what ecconomies can be established.
That's fine by my book.
> > And talking further of co-operation. I'd like to make DProbes drive your > trace facility.
See below.
> Did you see the announcement post I sent to LTT?
Yes I did get the announcement, though I was far too busy at the time to respond anything meaningful more than: "Cool stuff" :) So I said to myself that I'd take the time later to write something more intelligible ... I never got to the later part ...
> The idea > is that DProbes is an enabler for other RAS facilities. We can dynamically > insert a probe anywhere into memory (user and kernel) without the need for > re-compilation of the source. From the RPN program that's driven by the > probe event handler we can initiate other facilities such as entering SGI's > kernel debugger or invoking Crash Dump or forcing a core dump. Now, DProbes > came from OS/2 and was called dynamic trace. Its original purpose was to > implement tracepoints on the fly. We can still do that with DProbes, > provided we have a tracing mechanism we can feed into. That's where you > come in. Can you provide an interface we can call from kernel space to log > a trace event with a variable length buffer?
(I begin with a paragraph on the "philosophy" behind LTT and related trace projects to make sure everyone is aware of those issues).
OK, I think there's a design issue that ought to be addressed right away. LTT wasn't designed to help anyone delve into or debug kernel internals. That was never a design goal and will not be for any foreseeable future. The main purpose of LTT is to provide it's user with an accurate reconstruction of the dynamic behavior of the system being observed. That said, LTT __can also__ be used to understand kernel internals and track internal operations. The generic hooking mechanism provided, the capability of creating new trace points and the event-driven state machine engine are all part of this "internals" functionality. Even though these are not an essential part of LTT (the essential part being as discussed above), they do form a very interesting development avenue and this is where I believe the projects I am putting forward and Dprobes cross the same paths. In each, there's a desire to empower __kernel developers__ or __kernel power users__ (if such a thing exists) with the tools necessary to track and act upon the occurrence of key events. Where the current LTT implementation of things only regards a fixed set of key kernel events, Dprobes leaves definitions open to the programer. I don't think that either is better than the other, I think both serve their purpose. By providing a pre-defined set of key events, LTT "defines" which events are interesting for system behavior reconstruction and which aren't (notice that I speak of "system behavior" not "kernel behavior", the former being the combined behavior of the kernel and the applications). By providing a way to insert arbitrary probes, Dprobes enables system developers to rapidly isolate different sort of bugs __without__ having to modify the original source code.
So, my point is that although Dprobes can make a world of a difference to help developers isolate bugs and fast, it cannot, by itself, serve as the basis for the system behavior reconstruction infrastructure. For that, and many other purposes, there must be a pre-defined set of key events. An example of "other purposes" is using the pre-defined trace points to implement C2 security in the kernel (this was suggested by Alan).
Hence, yes I can provide an interface from the kernel to log a trace event with a variable length buffer, but I don't think that taking away the statically defined trace points is the right thing to do. (I might have gotten this completely wrong, though ... My presumption about your suggestion of using Dprobes to "drive" LTT, is that you mean that all events should come from Dprobes and Drpobes alone. I could be wrong).
So here's what I suggest: There's already two event types within the events recognized by LTT which had been planned for this type of usage. They are: "New event" and "Custom event". The first is used to declare a new event type and the second is used to log all such events. To declare a new event, the caller would call upon an event ID creation function providing it with an event size. The function would use the "New event" type to declare a new event in the log and would return a unique event ID. Thereafter, the normal tracing function, already available through the LTT kernel patch, could be used to log the new events.
This could be used by Dprobes to enable dynamically inserted probe points to be logged within a normal trace and, thereafter, be part of trace analysis.
Does this fit your needs?
> > Richard Moore - RAS Project Lead - Linux Technology Centre (PISC). > > http://oss.software.ibm.com/developerworks/opensource/linux > Office: (+44) (0)1962-817072, Mobile: (+44) (0)7768-298183 > IBM UK Ltd, MP135 Galileo Centre, Hursley Park, Winchester, SO21 2JN, UK > > Karim Yaghmour <karym@opersys.com> on 06/10/2000 09:16:12 > > Please respond to Karim Yaghmour <karym@opersys.com> > > To: Richard J Moore/UK/IBM@IBMGB > cc: linux-kernel@vger.kernel.org > Subject: Re: The case for a standard kernel debugger > > Hello Richard, > > Part of your analysis is correct. The hooks were designed to take care of > static tracepoints only. That said, dynamic allocation of event IDs was > next on my list and the hooking mechanism would have been modified > consequently. > > As for "multiple exits registered per hook", if you mean that you can have > more > than one function called back for each event, then this is already > possible. > The other items you mention such as atomicity and prioritization seem > interesting > indeed, although I am not sure what you mean by MP compliant as the only > thing that stops the current generalized hooking mechanism to be MP > compliant > is the insertion of correct locks during callback registration. > > Please understand that the purpose wasn't to discredit your work, but > rather > to stop duplication of work as efforts could be deployed elsewhere. I think > that your work and the work already done on LTT can be brought together in > a way that would profit all. This is what I was hinting to towards the end > of the posting. It was an invitation more than anything else. > > Apart from the hooking mechanism, there were other items which I mentioned > that merit discussion, such as the ability to enable dynamic probes to log > events in normal LTT traces or the event-driven state machine engine. > Hence, > if you are interested in joining forces to further enhance probing and > tracing > capabilities in Linux, I think this would be a good opportunity. > > Best regards > > Karim > > richardj_moore@uk.ibm.com wrote: > > > > Yes, we looked at that and it didn't seem to provide the generality we > > needed - multipe exits registered per hook, ability to arm a set of hooks > > atomically, ability to prioritise dispatching order of a hook exit, MP > > complient. I may be wrong but the Linux Trace Toolkit hooks like like > they > > were specifically designed to cater for inserting static tracepoints into > > the kernel. > > > > Richard Moore - RAS Project Lead - Linux Technology Centre (PISC). > > > > http://oss.software.ibm.com/developerworks/opensource/linux > > Office: (+44) (0)1962-817072, Mobile: (+44) (0)7768-298183 > > IBM UK Ltd, MP135 Galileo Centre, Hursley Park, Winchester, SO21 2JN, UK > > -- > =================================================== > Karim Yaghmour > karym@opersys.com > Operating System Consultant > (Linux kernel, real-time and distributed systems) > =================================================== > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > Please read the FAQ at http://www.tux.org/lkml/
-- =================================================== Karim Yaghmour karym@opersys.com Operating System Consultant (Linux kernel, real-time and distributed systems) =================================================== - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org Please read the FAQ at http://www.tux.org/lkml/
| ![\](/images/icornerr.gif) |