Messages in this thread | | | From | Noah Goldstein <> | Date | Fri, 26 Nov 2021 13:50:32 -0600 | Subject | Re: [PATCH v1] x86/lib: Optimize 8x loop and memory clobbers in csum_partial.c |
| |
On Fri, Nov 26, 2021 at 1:21 PM Eric Dumazet <edumazet@google.com> wrote: > > On Fri, Nov 26, 2021 at 11:14 AM Noah Goldstein <goldstein.w.n@gmail.com> wrote: > > > > On Fri, Nov 26, 2021 at 12:50 PM Noah Goldstein <goldstein.w.n@gmail.com> wrote: > > > > > > On Fri, Nov 26, 2021 at 12:27 PM Eric Dumazet <edumazet@google.com> wrote: > > > > > > > > On Fri, Nov 26, 2021 at 10:17 AM Noah Goldstein <goldstein.w.n@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > Makes sense. Although if you inline I think you definitely will want a more > > > > > conservative clobber than just "memory". Also I think with 40 you also will > > > > > get some value from two counters. > > > > > > > > > > Did you see the number/question I posted about two accumulators for 32 > > > > > byte case? > > > > > Its a judgement call about latency vs throughput that I don't really have an > > > > > answer for. > > > > > > > > > > > > > The thing I do not know is if using more units would slow down the > > > > hyper thread ? > > > > Did some quick tests with the latency/throughput benchmarks running > > in parallel on two hyperthreads on the same processors. The 32 byte case > > latency advantage goes with 2 accum and there is still a slight regression > > in throughput. The larger cases that hit the loop still still have improvements > > both in tput and latency with 2 accum. > > Great. I also played with rorx instruction, because it removes one > "adcl $0,..." step
Bright :) but it will need a BMI support check.
> > __wsum ipv6_csum_partial(const void *buff, int len, __wsum sum) > { > u64 temp64; > u64 tmp; > u32 res32; > > if (unlikely(len != 40)) > return csum_partial(buff, len, sum); > > temp64 = (__force u64)sum; > asm("addq 0*8(%[src]),%[temp64]\n\t" > "adcq 1*8(%[src]),%[temp64]\n\t" > "adcq 2*8(%[src]),%[temp64]\n\t" > "adcq 3*8(%[src]),%[temp64]\n\t" > "adcq 4*8(%[src]),%[temp64]\n\t" > "mov %k[temp64],%[res32]\n\t" > "rorx $32,%[temp64],%[temp64]\n\t" > "adcl %k[temp64],%[res32]\n\t" > "adcl $0,%[res32]" > : [temp64] "+r" (temp64), [res32] "=r" (res32) > : [src] "r" (buff) > : "memory"); > return (__force __wsum)res32; > }
I actually get better performance in hyperthread benchmarks with 2 accum:
Used:
u64 res; temp64 = (__force uint64_t)sum; asm("movq 0*8(%[src]),%[res]\n\t" "addq 1*8(%[src]),%[res]\n\t" "adcq 2*8(%[src]),%[res]\n\t" "adcq $0, %[res]\n" "addq 3*8(%[src]),%[temp64]\n\t" "adcq 4*8(%[src]),%[temp64]\n\t" "adcq %[res], %[temp64]\n\t" "mov %k[temp64],%k[res]\n\t" "rorx $32,%[temp64],%[temp64]\n\t" "adcl %k[temp64],%k[res]\n\t" "adcl $0,%k[res]" : [temp64] "+r"(temp64), [res] "=&r"(res) : [src] "r"(buff) : "memory"); return (__force __wsum)res;
w/ hyperthread: size, 2acc lat, 1acc lat, 2acc tput, 1acc tput 40, 6.511, 7.863, 6.177, 6.157
w/o hyperthread: size, 2acc lat, 1acc lat, 2acc tput, 1acc tput 40, 5.577, 6.764, 3.150, 3.210
| |