lkml.org 
[lkml]   [2023]   [Oct]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH v9 1/5] lib: objpool added: ring-array based lockless MPMC
On 2023/9/25 17:42, Masami Hiramatsu (Google) wrote:
> Hi Wuqiang,
>
> On Tue, 5 Sep 2023 09:52:51 +0800
> "wuqiang.matt" <wuqiang.matt@bytedance.com> wrote:
>
>> The object pool is a scalable implementaion of high performance queue
>> for object allocation and reclamation, such as kretprobe instances.
>>
>> With leveraging percpu ring-array to mitigate the hot spot of memory
>> contention, it could deliver near-linear scalability for high parallel
>> scenarios. The objpool is best suited for following cases:
>> 1) Memory allocation or reclamation are prohibited or too expensive
>> 2) Consumers are of different priorities, such as irqs and threads
>>
>> Limitations:
>> 1) Maximum objects (capacity) is determined during pool initializing
>> and can't be modified (extended) after objpool creation
>> 2) The memory of objects won't be freed until objpool is finalized
>> 3) Object allocation (pop) may fail after trying all cpu slots
>
> I made a simplifying patch on this by (mainly) removing ages array.
> I also rename local variable to use more readable names, like slot,
> pool, and obj.
>
> Here the results which I run the test_objpool.ko.
>
> Original:
> [ 50.500235] Summary of testcases:
> [ 50.503139] duration: 1027135us hits: 30628293 miss: 0 sync: percpu objpool
> [ 50.510416] duration: 1047977us hits: 30453023 miss: 0 sync: percpu objpool from vmalloc
> [ 50.517421] duration: 1047098us hits: 31145034 miss: 0 sync & hrtimer: percpu objpool
> [ 50.524671] duration: 1053233us hits: 30919640 miss: 0 sync & hrtimer: percpu objpool from vmalloc
> [ 50.531382] duration: 1055822us hits: 3407221 miss: 830219 sync overrun: percpu objpool
> [ 50.538135] duration: 1055998us hits: 3404624 miss: 854160 sync overrun: percpu objpool from vmalloc
> [ 50.546686] duration: 1046104us hits: 19464798 miss: 0 async: percpu objpool
> [ 50.552989] duration: 1033054us hits: 18957836 miss: 0 async: percpu objpool from vmalloc
> [ 50.560289] duration: 1041778us hits: 33806868 miss: 0 async & hrtimer: percpu objpool
> [ 50.567425] duration: 1048901us hits: 34211860 miss: 0 async & hrtimer: percpu objpool from vmalloc
>
> Simplified:
> [ 48.393236] Summary of testcases:
> [ 48.395384] duration: 1013002us hits: 29661448 miss: 0 sync: percpu objpool
> [ 48.400351] duration: 1057231us hits: 31035578 miss: 0 sync: percpu objpool from vmalloc
> [ 48.405660] duration: 1043196us hits: 30546652 miss: 0 sync & hrtimer: percpu objpool
> [ 48.411216] duration: 1047128us hits: 30601306 miss: 0 sync & hrtimer: percpu objpool from vmalloc
> [ 48.417231] duration: 1051695us hits: 3468287 miss: 892881 sync overrun: percpu objpool
> [ 48.422405] duration: 1054003us hits: 3549491 miss: 898141 sync overrun: percpu objpool from vmalloc
> [ 48.428425] duration: 1052946us hits: 19005228 miss: 0 async: percpu objpool
> [ 48.433597] duration: 1051757us hits: 19670866 miss: 0 async: percpu objpool from vmalloc
> [ 48.439280] duration: 1042951us hits: 37135332 miss: 0 async & hrtimer: percpu objpool
> [ 48.445085] duration: 1029803us hits: 37093248 miss: 0 async & hrtimer: percpu objpool from vmalloc
>
> Can you test it too?

Sure, I'll do some testings locally. Also do a performance comparsion with
'entries' managed in objpool_head.

>
> Thanks,

Thank you for your time, I appreciate it.

>
> From f1f442ff653e329839e5452b8b88463a80a12ff3 Mon Sep 17 00:00:00 2001
> From: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
> Date: Mon, 25 Sep 2023 16:07:12 +0900
> Subject: [PATCH] objpool: Simplify objpool by removing ages array
>
> Simplify the objpool code by removing ages array from per-cpu slot.
> It chooses enough big capacity (which is a rounded up power of 2 value
> of nr_objects + 1) for the entries array, the tail never catch up to
> the head in per-cpu slot. Thus tail == head means the slot is empty.
>
> This also uses consistent local variable names for pool, slot and obj.
>
> Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> ---
> include/linux/objpool.h | 61 ++++----
> lib/objpool.c | 310 ++++++++++++++++------------------------
> 2 files changed, 147 insertions(+), 224 deletions(-)
>
> diff --git a/include/linux/objpool.h b/include/linux/objpool.h
> index 33c832216b98..ecd5ecaffcd3 100644
> --- a/include/linux/objpool.h
> +++ b/include/linux/objpool.h
> @@ -38,33 +38,23 @@
> * struct objpool_slot - percpu ring array of objpool
> * @head: head of the local ring array (to retrieve at)
> * @tail: tail of the local ring array (to append at)
> - * @bits: log2 of capacity (for bitwise operations)
> - * @mask: capacity - 1
> + * @mask: capacity of entries - 1
> + * @entries: object entries on this slot.
> *
> * Represents a cpu-local array-based ring buffer, its size is specialized
> * during initialization of object pool. The percpu objpool slot is to be
> * allocated from local memory for NUMA system, and to be kept compact in
> - * continuous memory: ages[] is stored just after the body of objpool_slot,
> - * and then entries[]. ages[] describes revision of each item, solely used
> - * to avoid ABA; entries[] contains pointers of the actual objects
> - *
> - * Layout of objpool_slot in memory:
> - *
> - * 64bit:
> - * 4 8 12 16 32 64
> - * | head | tail | bits | mask | ages[4] | ents[4]: (8 * 4) | objects
> - *
> - * 32bit:
> - * 4 8 12 16 32 48 64
> - * | head | tail | bits | mask | ages[4] | ents[4] | unused | objects
> + * continuous memory: CPU assigned number of objects are stored just after
> + * the body of objpool_slot.
> *
> */
> struct objpool_slot {
> - uint32_t head;
> - uint32_t tail;
> - uint32_t bits;
> - uint32_t mask;
> -} __packed;
> + uint32_t head;
> + uint32_t tail;
> + uint32_t mask;
> + uint32_t dummyr;
> + void * entries[];
> +};
>
> struct objpool_head;
>
> @@ -82,12 +72,11 @@ typedef int (*objpool_fini_cb)(struct objpool_head *head, void *context);
> * @obj_size: object & element size
> * @nr_objs: total objs (to be pre-allocated)
> * @nr_cpus: nr_cpu_ids
> - * @capacity: max objects per cpuslot
> + * @capacity: max objects on percpu slot
> * @gfp: gfp flags for kmalloc & vmalloc
> * @ref: refcount for objpool
> * @flags: flags for objpool management
> * @cpu_slots: array of percpu slots
> - * @slot_sizes: size in bytes of slots
> * @release: resource cleanup callback
> * @context: caller-provided context
> */
> @@ -100,7 +89,6 @@ struct objpool_head {
> refcount_t ref;
> unsigned long flags;
> struct objpool_slot **cpu_slots;
> - int *slot_sizes;
> objpool_fini_cb release;
> void *context;
> };
> @@ -108,9 +96,12 @@ struct objpool_head {
> #define OBJPOOL_FROM_VMALLOC (0x800000000) /* objpool allocated from vmalloc area */
> #define OBJPOOL_HAVE_OBJECTS (0x400000000) /* objects allocated along with objpool */
>
> +#define OBJPOOL_NR_OBJECT_MAX (1 << 24)
> +#define OBJPOOL_OBJECT_SIZE_MAX (1 << 16)
> +
> /**
> * objpool_init() - initialize objpool and pre-allocated objects
> - * @head: the object pool to be initialized, declared by caller
> + * @pool: the object pool to be initialized, declared by caller
> * @nr_objs: total objects to be pre-allocated by this object pool
> * @object_size: size of an object (should be > 0)
> * @gfp: flags for memory allocation (via kmalloc or vmalloc)
> @@ -128,47 +119,47 @@ struct objpool_head {
> * pop (object allocation) or do clearance before each push (object
> * reclamation).
> */
> -int objpool_init(struct objpool_head *head, int nr_objs, int object_size,
> +int objpool_init(struct objpool_head *pool, int nr_objs, int object_size,
> gfp_t gfp, void *context, objpool_init_obj_cb objinit,
> objpool_fini_cb release);
>
> /**
> * objpool_pop() - allocate an object from objpool
> - * @head: object pool
> + * @pool: object pool
> *
> * return value: object ptr or NULL if failed
> */
> -void *objpool_pop(struct objpool_head *head);
> +void *objpool_pop(struct objpool_head *pool);
>
> /**
> * objpool_push() - reclaim the object and return back to objpool
> * @obj: object ptr to be pushed to objpool
> - * @head: object pool
> + * @pool: object pool
> *
> * return: 0 or error code (it fails only when user tries to push
> * the same object multiple times or wrong "objects" into objpool)
> */
> -int objpool_push(void *obj, struct objpool_head *head);
> +int objpool_push(void *obj, struct objpool_head *pool);
>
> /**
> * objpool_drop() - discard the object and deref objpool
> * @obj: object ptr to be discarded
> - * @head: object pool
> + * @pool: object pool
> *
> * return: 0 if objpool was released or error code
> */
> -int objpool_drop(void *obj, struct objpool_head *head);
> +int objpool_drop(void *obj, struct objpool_head *pool);
>
> /**
> * objpool_free() - release objpool forcely (all objects to be freed)
> - * @head: object pool to be released
> + * @pool: object pool to be released
> */
> -void objpool_free(struct objpool_head *head);
> +void objpool_free(struct objpool_head *pool);
>
> /**
> * objpool_fini() - deref object pool (also releasing unused objects)
> - * @head: object pool to be dereferenced
> + * @pool: object pool to be dereferenced
> */
> -void objpool_fini(struct objpool_head *head);
> +void objpool_fini(struct objpool_head *pool);
>
> #endif /* _LINUX_OBJPOOL_H */
> diff --git a/lib/objpool.c b/lib/objpool.c
> index 22e752371820..f8e8f70d7253 100644
> --- a/lib/objpool.c
> +++ b/lib/objpool.c
> @@ -15,104 +15,55 @@
> * Copyright: wuqiang.matt@bytedance.com
> */
>
> -#define SLOT_AGES(s) ((uint32_t *)((char *)(s) + sizeof(struct objpool_slot)))
> -#define SLOT_ENTS(s) ((void **)((char *)(s) + sizeof(struct objpool_slot) + \
> - (sizeof(uint32_t) << (s)->bits)))
> -#define SLOT_OBJS(s) ((void *)((char *)(s) + sizeof(struct objpool_slot) + \
> - ((sizeof(uint32_t) + sizeof(void *)) << (s)->bits)))
> -#define SLOT_CORE(n) cpumask_nth((n) % num_possible_cpus(), cpu_possible_mask)
> -
> -/* compute the suitable num of objects to be managed per slot */
> -static int objpool_nobjs(int size)
> -{
> - return rounddown_pow_of_two((size - sizeof(struct objpool_slot)) /
> - (sizeof(uint32_t) + sizeof(void *)));
> -}
> -
> /* allocate and initialize percpu slots */
> static int
> -objpool_init_percpu_slots(struct objpool_head *head, int nobjs,
> - void *context, objpool_init_obj_cb objinit)
> +objpool_init_percpu_slots(struct objpool_head *pool, void *context,
> + objpool_init_obj_cb objinit)
> {
> - int i, j, n, size, objsz, cpu = 0, nents = head->capacity;
> -
> - /* aligned object size by sizeof(void *) */
> - objsz = ALIGN(head->obj_size, sizeof(void *));
> - /* shall we allocate objects along with percpu-slot */
> - if (objsz)
> - head->flags |= OBJPOOL_HAVE_OBJECTS;
> -
> - /* vmalloc is used in default to allocate percpu-slots */
> - if (!(head->gfp & GFP_ATOMIC))
> - head->flags |= OBJPOOL_FROM_VMALLOC;
> -
> - for (i = 0; i < head->nr_cpus; i++) {
> - struct objpool_slot *os;
> + int i, j, n, size, slot_size, cpu_count = 0;
> + struct objpool_slot *slot;
>
> + for (i = 0; i < pool->nr_cpus; i++) {
> /* skip the cpus which could never be present */
> if (!cpu_possible(i))
> continue;
>
> /* compute how many objects to be managed by this slot */
> - n = nobjs / num_possible_cpus();
> - if (cpu < (nobjs % num_possible_cpus()))
> + n = pool->nr_objs / num_possible_cpus();
> + if (cpu_count < (pool->nr_objs % num_possible_cpus()))
> n++;
> - size = sizeof(struct objpool_slot) + sizeof(void *) * nents +
> - sizeof(uint32_t) * nents + objsz * n;
> + cpu_count++;
> +
> + slot_size = struct_size(slot, entries, pool->capacity);
> + size = slot_size + pool->obj_size * n;
>
> /*
> * here we allocate percpu-slot & objects together in a single
> - * allocation, taking advantage of warm caches and TLB hits as
> - * vmalloc always aligns the request size to pages
> + * allocation, taking advantage on NUMA.
> */
> - if (head->flags & OBJPOOL_FROM_VMALLOC)
> - os = __vmalloc_node(size, sizeof(void *), head->gfp,
> + if (pool->flags & OBJPOOL_FROM_VMALLOC)
> + slot = __vmalloc_node(size, sizeof(void *), pool->gfp,
> cpu_to_node(i), __builtin_return_address(0));
> else
> - os = kmalloc_node(size, head->gfp, cpu_to_node(i));
> - if (!os)
> + slot = kmalloc_node(size, pool->gfp, cpu_to_node(i));
> + if (!slot)
> return -ENOMEM;
>
> /* initialize percpu slot for the i-th slot */
> - memset(os, 0, size);
> - os->bits = ilog2(head->capacity);
> - os->mask = head->capacity - 1;
> - head->cpu_slots[i] = os;
> - head->slot_sizes[i] = size;
> - cpu = cpu + 1;
> -
> - /*
> - * manually set head & tail to avoid possible conflict:
> - * We assume that the head item is ready for retrieval
> - * iff head is equal to ages[head & mask]. but ages is
> - * initialized as 0, so in view of the caller of pop(),
> - * the 1st item (0th) is always ready, but the reality
> - * could be: push() is stalled before the final update,
> - * thus the item being inserted will be lost forever
> - */
> - os->head = os->tail = head->capacity;
> -
> - if (!objsz)
> - continue;
> + memset(slot, 0, size);
> + slot->mask = pool->capacity - 1;
> + pool->cpu_slots[i] = slot;
>
> for (j = 0; j < n; j++) {
> - uint32_t *ages = SLOT_AGES(os);
> - void **ents = SLOT_ENTS(os);
> - void *obj = SLOT_OBJS(os) + j * objsz;
> - uint32_t ie = os->tail & os->mask;
> + void *obj = (void *)slot + slot_size + pool->obj_size * j;
>
> - /* perform object initialization */
> if (objinit) {
> int rc = objinit(obj, context);
> if (rc)
> return rc;
> }
> -
> - /* add obj into the ring array */
> - ents[ie] = obj;
> - ages[ie] = os->tail;
> - os->tail++;
> - head->nr_objs++;
> + slot->entries[j] = obj;
> + slot->tail++;
> }
> }
>
> @@ -120,164 +71,135 @@ objpool_init_percpu_slots(struct objpool_head *head, int nobjs,
> }
>
> /* cleanup all percpu slots of the object pool */
> -static void objpool_fini_percpu_slots(struct objpool_head *head)
> +static void objpool_fini_percpu_slots(struct objpool_head *pool)
> {
> int i;
>
> - if (!head->cpu_slots)
> + if (!pool->cpu_slots)
> return;
>
> - for (i = 0; i < head->nr_cpus; i++) {
> - if (!head->cpu_slots[i])
> - continue;
> - if (head->flags & OBJPOOL_FROM_VMALLOC)
> - vfree(head->cpu_slots[i]);
> - else
> - kfree(head->cpu_slots[i]);
> - }
> - kfree(head->cpu_slots);
> - head->cpu_slots = NULL;
> - head->slot_sizes = NULL;
> + for (i = 0; i < pool->nr_cpus; i++)
> + kvfree(pool->cpu_slots[i]);
> + kfree(pool->cpu_slots);
> }
>
> /* initialize object pool and pre-allocate objects */
> -int objpool_init(struct objpool_head *head, int nr_objs, int object_size,
> +int objpool_init(struct objpool_head *pool, int nr_objs, int object_size,
> gfp_t gfp, void *context, objpool_init_obj_cb objinit,
> objpool_fini_cb release)
> {
> int nents, rc;
>
> /* check input parameters */
> - if (nr_objs <= 0 || object_size <= 0)
> + if (nr_objs <= 0 || object_size <= 0 ||
> + nr_objs > OBJPOOL_NR_OBJECT_MAX || object_size > OBJPOOL_OBJECT_SIZE_MAX)
> + return -EINVAL;
> +
> + /* Align up to unsigned long size */
> + object_size = ALIGN(object_size, sizeof(unsigned long));
> +
> + /*
> + * To avoid filling up the entries array in the per-cpu slot,
> + * use the power of 2 which is more than N + 1. Thus, the tail never
> + * catch up to the pool, and able to use pool/tail as the sequencial
> + * number.
> + */
> + nents = roundup_pow_of_two(nr_objs + 1);
> + if (!nents)
> return -EINVAL;
>
> - /* calculate percpu slot size (rounded to pow of 2) */
> - nents = max_t(int, roundup_pow_of_two(nr_objs),
> - objpool_nobjs(L1_CACHE_BYTES));
> -
> - /* initialize objpool head */
> - memset(head, 0, sizeof(struct objpool_head));
> - head->nr_cpus = nr_cpu_ids;
> - head->obj_size = object_size;
> - head->capacity = nents;
> - head->gfp = gfp & ~__GFP_ZERO;
> - head->context = context;
> - head->release = release;
> -
> - /* allocate array for percpu slots */
> - head->cpu_slots = kzalloc(head->nr_cpus * sizeof(void *) +
> - head->nr_cpus * sizeof(int), head->gfp);
> - if (!head->cpu_slots)
> + /* initialize objpool pool */
> + memset(pool, 0, sizeof(struct objpool_head));
> + pool->nr_cpus = nr_cpu_ids;
> + pool->obj_size = object_size;
> + pool->nr_objs = nr_objs;
> + /* just prevent to fullfill the per-cpu ring array */
> + pool->capacity = nents;
> + pool->gfp = gfp & ~__GFP_ZERO;
> + pool->context = context;
> + pool->release = release;
> + /* vmalloc is used in default to allocate percpu-slots */
> + if (!(pool->gfp & GFP_ATOMIC))
> + pool->flags |= OBJPOOL_FROM_VMALLOC;
> +
> + pool->cpu_slots = kzalloc(pool->nr_cpus * sizeof(void *), pool->gfp);
> + if (!pool->cpu_slots)
> return -ENOMEM;
> - head->slot_sizes = (int *)&head->cpu_slots[head->nr_cpus];
>
> /* initialize per-cpu slots */
> - rc = objpool_init_percpu_slots(head, nr_objs, context, objinit);
> + rc = objpool_init_percpu_slots(pool, context, objinit);
> if (rc)
> - objpool_fini_percpu_slots(head);
> + objpool_fini_percpu_slots(pool);
> else
> - refcount_set(&head->ref, nr_objs + 1);
> + refcount_set(&pool->ref, nr_objs + 1);
>
> return rc;
> }
> EXPORT_SYMBOL_GPL(objpool_init);
>
> /* adding object to slot, abort if the slot was already full */
> -static inline int objpool_try_add_slot(void *obj, struct objpool_slot *os)
> +static inline int objpool_try_add_slot(void *obj, struct objpool_head *pool, int cpu)
> {
> - uint32_t *ages = SLOT_AGES(os);
> - void **ents = SLOT_ENTS(os);
> - uint32_t head, tail;
> + struct objpool_slot *slot = pool->cpu_slots[cpu];
> + uint32_t tail, next;
>
> do {
> - /* perform memory loading for both head and tail */
> - head = READ_ONCE(os->head);
> - tail = READ_ONCE(os->tail);
> - /* just abort if slot is full */
> - if (tail - head > os->mask)
> - return -ENOENT;
> - /* try to extend tail by 1 using CAS to avoid races */
> - if (try_cmpxchg_acquire(&os->tail, &tail, tail + 1))
> - break;
> - } while (1);
> + uint32_t head = READ_ONCE(slot->head);
>
> - /* the tail-th of slot is reserved for the given obj */
> - WRITE_ONCE(ents[tail & os->mask], obj);
> - /* update epoch id to make this object available for pop() */
> - smp_store_release(&ages[tail & os->mask], tail);
> + tail = READ_ONCE(slot->tail);
> + next = tail + 1;
> +
> + /* This must never happen because capacity >= N + 1 */
> + if (WARN_ON_ONCE((next > head && next - head > pool->nr_objs) ||
> + (next < head && next > head + pool->nr_objs)))
> + return -EINVAL;
> +
> + } while (!try_cmpxchg_acquire(&slot->tail, &tail, next));
> +
> + WRITE_ONCE(slot->entries[tail & slot->mask], obj);
> return 0;
> }
>
> /* reclaim an object to object pool */
> -int objpool_push(void *obj, struct objpool_head *oh)
> +int objpool_push(void *obj, struct objpool_head *pool)
> {
> unsigned long flags;
> - int cpu, rc;
> + int rc;
>
> /* disable local irq to avoid preemption & interruption */
> raw_local_irq_save(flags);
> - cpu = raw_smp_processor_id();
> - do {
> - rc = objpool_try_add_slot(obj, oh->cpu_slots[cpu]);
> - if (!rc)
> - break;
> - cpu = cpumask_next_wrap(cpu, cpu_possible_mask, -1, 1);
> - } while (1);
> + rc = objpool_try_add_slot(obj, pool, raw_smp_processor_id());
> raw_local_irq_restore(flags);
>
> return rc;
> }
> EXPORT_SYMBOL_GPL(objpool_push);
>
> -/* drop the allocated object, rather reclaim it to objpool */
> -int objpool_drop(void *obj, struct objpool_head *head)
> -{
> - if (!obj || !head)
> - return -EINVAL;
> -
> - if (refcount_dec_and_test(&head->ref)) {
> - objpool_free(head);
> - return 0;
> - }
> -
> - return -EAGAIN;
> -}
> -EXPORT_SYMBOL_GPL(objpool_drop);
> -
> /* try to retrieve object from slot */
> -static inline void *objpool_try_get_slot(struct objpool_slot *os)
> +static inline void *objpool_try_get_slot(struct objpool_slot *slot)
> {
> - uint32_t *ages = SLOT_AGES(os);
> - void **ents = SLOT_ENTS(os);
> /* do memory load of head to local head */
> - uint32_t head = smp_load_acquire(&os->head);
> + uint32_t head = smp_load_acquire(&slot->head);
>
> /* loop if slot isn't empty */
> - while (head != READ_ONCE(os->tail)) {
> - uint32_t id = head & os->mask, prev = head;
> + while (head != READ_ONCE(slot->tail)) {
>
> /* do prefetching of object ents */
> - prefetch(&ents[id]);
> -
> - /* check whether this item was ready for retrieval */
> - if (smp_load_acquire(&ages[id]) == head) {
> - /* node must have been udpated by push() */
> - void *node = READ_ONCE(ents[id]);
> - /* commit and move forward head of the slot */
> - if (try_cmpxchg_release(&os->head, &head, head + 1))
> - return node;
> - /* head was already updated by others */
> - }
> + prefetch(&slot->entries[head & slot->mask]);
> +
> + /* commit and move forward head of the slot */
> + if (try_cmpxchg_release(&slot->head, &head, head + 1))
> + /*
> + * TBD: check overwrap the tail/head counter and warn
> + * if it is broken. But this happens only if this
> + * process slows down a lot and another CPU updates
> + * the haed/tail just 2^32 + 1 times, and this slot
> + * is empty.
> + */
> + return slot->entries[head & slot->mask];
>
> /* re-load head from memory and continue trying */
> - head = READ_ONCE(os->head);
> - /*
> - * head stays unchanged, so it's very likely there's an
> - * ongoing push() on other cpu nodes but yet not update
> - * ages[] to mark it's completion
> - */
> - if (head == prev)
> - break;
> + head = READ_ONCE(slot->head);
> }
>
> return NULL;
> @@ -307,32 +229,42 @@ void *objpool_pop(struct objpool_head *head)
> EXPORT_SYMBOL_GPL(objpool_pop);
>
> /* release whole objpool forcely */
> -void objpool_free(struct objpool_head *head)
> +void objpool_free(struct objpool_head *pool)
> {
> - if (!head->cpu_slots)
> + if (!pool->cpu_slots)
> return;
>
> /* release percpu slots */
> - objpool_fini_percpu_slots(head);
> + objpool_fini_percpu_slots(pool);
>
> /* call user's cleanup callback if provided */
> - if (head->release)
> - head->release(head, head->context);
> + if (pool->release)
> + pool->release(pool, pool->context);
> }
> EXPORT_SYMBOL_GPL(objpool_free);
>
> -/* drop unused objects and defref objpool for releasing */
> -void objpool_fini(struct objpool_head *head)
> +/* drop the allocated object, rather reclaim it to objpool */
> +int objpool_drop(void *obj, struct objpool_head *pool)
> {
> - void *nod;
> + if (!obj || !pool)
> + return -EINVAL;
>
> - do {
> - /* grab object from objpool and drop it */
> - nod = objpool_pop(head);
> + if (refcount_dec_and_test(&pool->ref)) {
> + objpool_free(pool);
> + return 0;
> + }
> +
> + return -EAGAIN;
> +}
> +EXPORT_SYMBOL_GPL(objpool_drop);
> +
> +/* drop unused objects and defref objpool for releasing */
> +void objpool_fini(struct objpool_head *pool)
> +{
> + void *obj;
>
> - /* drop the extra ref of objpool */
> - if (refcount_dec_and_test(&head->ref))
> - objpool_free(head);
> - } while (nod);
> + /* grab object from objpool and drop it */
> + while ((obj = objpool_pop(pool)))
> + objpool_drop(obj, pool);
> }
> EXPORT_SYMBOL_GPL(objpool_fini);

\
 
 \ /
  Last update: 2023-10-08 21:05    [W:0.248 / U:1.024 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site