Messages in this thread | | | Date | Mon, 23 Jan 2023 10:16:37 +0100 | Subject | Re: [PATCH v5 10/39] x86/mm: Introduce _PAGE_COW | From | David Hildenbrand <> |
| |
On 19.01.23 22:22, Rick Edgecombe wrote: > Some OSes have a greater dependence on software available bits in PTEs than > Linux. That left the hardware architects looking for a way to represent a > new memory type (shadow stack) within the existing bits. They chose to > repurpose a lightly-used state: Write=0,Dirty=1. So in order to support > shadow stack memory, Linux should avoid creating memory with this PTE bit > combination unless it intends for it to be shadow stack. > > The reason it's lightly used is that Dirty=1 is normally set by HW > _before_ a write. A write with a Write=0 PTE would typically only generate > a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* > generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which > supports shadow stacks will no longer exhibit this oddity. > > So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this, > in places where Linux normally creates Write=0,Dirty=1, it can use the > software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other > words, whenever Linux needs to create Write=0,Dirty=1, it instead creates > Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. > Further differentiated by VMA flags, these PTE bit combinations would be > set as follows for various types of memory: > > (Write=0,Cow=1,Dirty=0): > - A modified, copy-on-write (COW) page. Previously when a typical > anonymous writable mapping was made COW via fork(), the kernel would > mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This > happens in copy_present_pte(). > - A R/O page that has been COW'ed. The user page is in a R/O VMA, > and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault > handler creates a copy of the page and sets the new copy's PTE as > Write=0 and Cow=1. > - A shared shadow stack PTE. When a shadow stack page is being shared > among processes (this happens at fork()), its PTE is made Dirty=0, so > the next shadow stack access causes a fault, and the page is > duplicated and Dirty=1 is set again. This is the COW equivalent for > shadow stack pages, even though it's copy-on-access rather than > copy-on-write. > > (Write=0,Cow=0,Dirty=1): > - A shadow stack PTE. > - A Cow PTE created when a processor without shadow stack support set > Dirty=1. > > There are six bits left available to software in the 64-bit PTE after > consuming a bit for _PAGE_COW. No space is consumed in 32-bit kernels > because shadow stacks are not enabled there. > > Implement only the infrastructure for _PAGE_COW. Changes to start > creating _PAGE_COW PTEs will follow once other pieces are in place. > > Tested-by: Pengfei Xu <pengfei.xu@intel.com> > Tested-by: John Allen <john.allen@amd.com> > Co-developed-by: Yu-cheng Yu <yu-cheng.yu@intel.com> > Signed-off-by: Yu-cheng Yu <yu-cheng.yu@intel.com> > Signed-off-by: Rick Edgecombe <rick.p.edgecombe@intel.com> > --- > > v5: > - Fix log, comments and whitespace (Boris) > - Remove capitalization on shadow stack (Boris) > > v4: > - Teach pte_flags_need_flush() about _PAGE_COW bit > - Break apart patch for better bisectability > > v3: > - Add comment around _PAGE_TABLE in response to comment > from (Andrew Cooper) > - Check for PSE in pmd_shstk (Andrew Cooper) > - Get to the point quicker in commit log (Andrew Cooper) > - Clarify and reorder commit log for why the PTE bit examples have > multiple entries. Apply same changes for comment. (peterz) > - Fix comment that implied dirty bit for COW was a specific x86 thing > (peterz) > - Fix swapping of Write/Dirty (PeterZ) > > v2: > - Update commit log with comments (Dave Hansen) > - Add comments in code to explain pte modification code better (Dave) > - Clarify info on the meaning of various Write,Cow,Dirty combinations > > arch/x86/include/asm/pgtable.h | 78 ++++++++++++++++++++++++++++ > arch/x86/include/asm/pgtable_types.h | 59 +++++++++++++++++++-- > arch/x86/include/asm/tlbflush.h | 3 +- > 3 files changed, 134 insertions(+), 6 deletions(-) > > diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h > index b39f16c0d507..6d2f612c04b5 100644 > --- a/arch/x86/include/asm/pgtable.h > +++ b/arch/x86/include/asm/pgtable.h > @@ -301,6 +301,44 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) > return native_make_pte(v & ~clear); > } > > +/* > + * Normally COW memory can result in Dirty=1,Write=0 PTEs. But in the case > + * of X86_FEATURE_USER_SHSTK, the software COW bit is used, since the > + * Dirty=1,Write=0 will result in the memory being treated as shadow stack > + * by the HW. So when creating COW memory, a software bit is used > + * _PAGE_BIT_COW. The following functions pte_mkcow() and pte_clear_cow() > + * take a PTE marked conventionally COW (Dirty=1) and transition it to the > + * shadow stack compatible version of COW (Cow=1). > + */ > +static inline pte_t pte_mkcow(pte_t pte) > +{ > + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) > + return pte; > + > + pte = pte_clear_flags(pte, _PAGE_DIRTY); > + return pte_set_flags(pte, _PAGE_COW); > +} > + > +static inline pte_t pte_clear_cow(pte_t pte) > +{ > + /* > + * _PAGE_COW is unnecessary on !X86_FEATURE_USER_SHSTK kernels, since > + * the HW dirty bit can be used without creating shadow stack memory. > + * See the _PAGE_COW definition for more details. > + */ > + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) > + return pte; > + > + /* > + * PTE is getting copied-on-write, so it will be dirtied > + * if writable, or made shadow stack if shadow stack and > + * being copied on access. Set the dirty bit for both > + * cases. > + */ > + pte = pte_set_flags(pte, _PAGE_DIRTY); > + return pte_clear_flags(pte, _PAGE_COW); > +} > + > #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP > static inline int pte_uffd_wp(pte_t pte) > { > @@ -413,6 +451,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) > return native_make_pmd(v & ~clear); > } > > +/* See comments above pte_mkcow() */ > +static inline pmd_t pmd_mkcow(pmd_t pmd) > +{ > + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) > + return pmd; > + > + pmd = pmd_clear_flags(pmd, _PAGE_DIRTY); > + return pmd_set_flags(pmd, _PAGE_COW); > +} > + > +/* See comments above pte_mkcow() */ > +static inline pmd_t pmd_clear_cow(pmd_t pmd) > +{ > + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) > + return pmd; > + > + pmd = pmd_set_flags(pmd, _PAGE_DIRTY); > + return pmd_clear_flags(pmd, _PAGE_COW); > +} > + > #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP > static inline int pmd_uffd_wp(pmd_t pmd) > { > @@ -484,6 +542,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) > return native_make_pud(v & ~clear); > } > > +/* See comments above pte_mkcow() */ > +static inline pud_t pud_mkcow(pud_t pud) > +{ > + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) > + return pud; > + > + pud = pud_clear_flags(pud, _PAGE_DIRTY); > + return pud_set_flags(pud, _PAGE_COW); > +} > + > +/* See comments above pte_mkcow() */ > +static inline pud_t pud_clear_cow(pud_t pud) > +{ > + if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) > + return pud; > + > + pud = pud_set_flags(pud, _PAGE_DIRTY); > + return pud_clear_flags(pud, _PAGE_COW); > +} > + > static inline pud_t pud_mkold(pud_t pud) > { > return pud_clear_flags(pud, _PAGE_ACCESSED); > diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h > index 0646ad00178b..5c3f942865d9 100644 > --- a/arch/x86/include/asm/pgtable_types.h > +++ b/arch/x86/include/asm/pgtable_types.h > @@ -21,7 +21,8 @@ > #define _PAGE_BIT_SOFTW2 10 /* " */ > #define _PAGE_BIT_SOFTW3 11 /* " */ > #define _PAGE_BIT_PAT_LARGE 12 /* On 2MB or 1GB pages */ > -#define _PAGE_BIT_SOFTW4 58 /* available for programmer */ > +#define _PAGE_BIT_SOFTW4 57 /* available for programmer */ > +#define _PAGE_BIT_SOFTW5 58 /* available for programmer */ > #define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4 */ > #define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4 */ > #define _PAGE_BIT_PKEY_BIT2 61 /* Protection Keys, bit 3/4 */ > @@ -34,6 +35,15 @@ > #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */ > #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 > > +/* > + * Indicates a copy-on-write page. > + */ > +#ifdef CONFIG_X86_USER_SHADOW_STACK > +#define _PAGE_BIT_COW _PAGE_BIT_SOFTW5 /* copy-on-write */ > +#else > +#define _PAGE_BIT_COW 0 > +#endif > + > /* If _PAGE_BIT_PRESENT is clear, we use these: */ > /* - if the user mapped it with PROT_NONE; pte_present gives true */ > #define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL > @@ -117,6 +127,40 @@ > #define _PAGE_SOFTW4 (_AT(pteval_t, 0)) > #endif > > +/* > + * The hardware requires shadow stack to be read-only and Dirty. > + * _PAGE_COW is a software-only bit used to separate copy-on-write PTEs > + * from shadow stack PTEs:
Is that really required?
For anon pages, we have PG_anon_exclusive, that can tell you whether the page is "certainly exclusive" (now cow necessary) vs. "maybe shared" (cow maybe necessary).
Why isn't that sufficient to make the same decisions here?
-- Thanks,
David / dhildenb
| |