Messages in this thread | | | From | Eric Dumazet <> | Date | Fri, 26 Nov 2021 11:21:34 -0800 | Subject | Re: [PATCH v1] x86/lib: Optimize 8x loop and memory clobbers in csum_partial.c |
| |
On Fri, Nov 26, 2021 at 11:14 AM Noah Goldstein <goldstein.w.n@gmail.com> wrote: > > On Fri, Nov 26, 2021 at 12:50 PM Noah Goldstein <goldstein.w.n@gmail.com> wrote: > > > > On Fri, Nov 26, 2021 at 12:27 PM Eric Dumazet <edumazet@google.com> wrote: > > > > > > On Fri, Nov 26, 2021 at 10:17 AM Noah Goldstein <goldstein.w.n@gmail.com> wrote: > > > > > > > > > > > > > > > Makes sense. Although if you inline I think you definitely will want a more > > > > conservative clobber than just "memory". Also I think with 40 you also will > > > > get some value from two counters. > > > > > > > > Did you see the number/question I posted about two accumulators for 32 > > > > byte case? > > > > Its a judgement call about latency vs throughput that I don't really have an > > > > answer for. > > > > > > > > > > The thing I do not know is if using more units would slow down the > > > hyper thread ? > > Did some quick tests with the latency/throughput benchmarks running > in parallel on two hyperthreads on the same processors. The 32 byte case > latency advantage goes with 2 accum and there is still a slight regression > in throughput. The larger cases that hit the loop still still have improvements > both in tput and latency with 2 accum.
Great. I also played with rorx instruction, because it removes one "adcl $0,..." step
__wsum ipv6_csum_partial(const void *buff, int len, __wsum sum) { u64 temp64; u64 tmp; u32 res32;
if (unlikely(len != 40)) return csum_partial(buff, len, sum);
temp64 = (__force u64)sum; asm("addq 0*8(%[src]),%[temp64]\n\t" "adcq 1*8(%[src]),%[temp64]\n\t" "adcq 2*8(%[src]),%[temp64]\n\t" "adcq 3*8(%[src]),%[temp64]\n\t" "adcq 4*8(%[src]),%[temp64]\n\t" "mov %k[temp64],%[res32]\n\t" "rorx $32,%[temp64],%[temp64]\n\t" "adcl %k[temp64],%[res32]\n\t" "adcl $0,%[res32]" : [temp64] "+r" (temp64), [res32] "=r" (res32) : [src] "r" (buff) : "memory"); return (__force __wsum)res32; }
| |