Messages in this thread | | | Date | Wed, 18 Oct 2023 15:18:52 -0400 | Subject | Re: [PATCH-cgroup 1/4] workqueue: Add workqueue_unbound_exclude_cpumask() to exclude CPUs from wq_unbound_cpumask | From | Waiman Long <> |
| |
On 10/18/23 09:41, Waiman Long wrote: > On 10/18/23 05:24, Tejun Heo wrote: >> Hello, >> >> On Fri, Oct 13, 2023 at 02:11:19PM -0400, Waiman Long wrote: >>> When the "isolcpus" boot command line option is used to add a set >>> of isolated CPUs, those CPUs will be excluded automatically from >>> wq_unbound_cpumask to avoid running work functions from unbound >>> workqueues. >>> >>> Recently cpuset has been extended to allow the creation of partitions >>> of isolated CPUs dynamically. To make it closer to the "isolcpus" >>> in functionality, the CPUs in those isolated cpuset partitions >>> should be >>> excluded from wq_unbound_cpumask as well. This can be done currently by >>> explicitly writing to the workqueue's cpumask sysfs file after creating >>> the isolated partitions. However, this process can be error prone. >>> Ideally, the cpuset code should be allowed to request the workqueue >>> code >>> to exclude those isolated CPUs from wq_unbound_cpumask so that this >>> operation can be done automatically and the isolated CPUs will be >>> returned >>> back to wq_unbound_cpumask after the destructions of the isolated >>> cpuset partitions. >>> >>> This patch adds a new workqueue_unbound_exclude_cpumask() to enable >>> that. This new function will exclude the specified isolated CPUs >>> from wq_unbound_cpumask. To be able to restore those isolated CPUs >>> back after the destruction of isolated cpuset partitions, a new >>> wq_user_unbound_cpumask is added to store the user provided unbound >>> cpumask either from the boot command line options or from writing to >>> the cpumask sysfs file. This new cpumask provides the basis for CPU >>> exclusion. >> The behaviors around wq_unbound_cpumask is getting pretty inconsistent: >> >> 1. Housekeeping excludes isolated CPUs on boot but allows user to >> override >> it to include isolated CPUs afterwards. >> >> 2. If an unbound wq's cpumask doesn't have any intersection with >> wq_unbound_cpumask we ignore the per-wq cpumask and falls back to >> wq_unbound_cpumask. >> >> 3. You're adding a masking layer on top with exclude which fails to >> set if >> the intersection is empty. >> >> Can we do the followings for consistency? >> >> 1. User's requested_unbound_cpumask is stored separately (as in this >> patch). >> >> 2. The effect wq_unbound_cpumask is determined by >> requested_unbound_cpumask >> & housekeeping_cpumask & cpuset_allowed_cpumask. The operation order >> matters. When an & operation yields an cpumask, the cpumask from the >> previous step is the effective one. > Sure. I will do that.
I have a second thought after taking a further look at that. First of all, cpuset_allowed_mask isn't relevant here and the mask can certainly contain offline CPUs. So cpu_possible_mask is the proper fallback.
With the current patch, wq_user_unbound_cpumask is set up initially as (HK_TYPE_WQ ∩ HK_TYPE_DOMAIN) house keeping mask and rewritten by any subsequent write to workqueue/cpumask sysfs file. So using wq_user_unbound_cpumask has the implied precedence of user-sysfs written mask, command line isolcpus or nohz_full option mask and cpu_possible_mask. I think just fall back to wq_user_unbound_cpumask if the operation fails should be enough.
Cheers, Longman
| |