Messages in this thread | | | Subject | Re: [PATCH v8 09/14] iio: afe: rescale: fix accuracy for small fractional scales | From | Peter Rosin <> | Date | Thu, 2 Sep 2021 23:52:21 +0200 |
| |
On 2021-09-02 04:27, Liam Beguin wrote: > On Mon, Aug 30, 2021 at 04:30:54PM +0200, Peter Rosin wrote: >> >> Having said that, if 'scale' and 'offset' indeed are static, then the gcd >> cost can be mitigated by caching the result. Exact rescaling is always >> nice... >> >> If IIO_VAL_INT overflows while rescaling, we are SOL whichever way we turn, >> so ignore doing anything about that. > > I was thinking of using check_mul_overflow() to do something about the > overflow, but I'm happy to leave it out for now.
My mistake, you are right. A sufficiently large denominator can of course be used to dodge overflow in the numerator by sacrificing some accuracy even if our maximum scale is still limited to 32 bits.
I apparently didn't have my brains about when I wrote the above...
Cheers, Peter
| |