Messages in this thread | | | Date | Tue, 26 Oct 2021 21:15:11 -0400 | From | Steven Rostedt <> | Subject | Re: [PATCH v4 6/8] tracing/histogram: Optimize division by a power of 2 |
| |
On Tue, 26 Oct 2021 18:09:22 -0700 Kalesh Singh <kaleshsingh@google.com> wrote:
> > delta = mult * div / 2^20 > > > > That is if mult is a power of two, then there would be no rounding > > errors, and the delta is zero, making the max infinite:
That should have been (as shown in the algorithm)
delta = mult * div - 2 ^ 20
As mult is 2^20 / div; and the above should end up zero if there's no rounding issues, as it would be:
delta = (2^20 / div) * div - 2^20
-- Steve
| |