mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-09-18 22:19:30 +02:00
master
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Lorenzo Stoakes (ARM)
|
59351365ac |
scripts/mksysmap: fix escape of '$' in the __pi_ pattern
Commit |
||
|
Lorenzo Stoakes (ARM)
|
281b61d408 |
scripts/mksysmap: drop the MODULE_INFO() symbols from kallsyms
Commit |
||
|
Lorenzo Stoakes (ARM)
|
9e4a3ec341 |
x86/mm/pat: Allocate split page tables as kernel page tables
A PTE is allocated directly without going through the standard page table allocation routines (such as pte_alloc_one_kernel()) when the CPA code splits a large page (__split_large_page()). This means the page table constructor is never called nor is the page table marked as a kernel page table. The former results in the folio associated with the page table not being marked as a page table (__pagetable_ctor() is never called thus neither is __folio_set_pgtable()) nor are statistics updated to reflect it (lruvec_stat_add_folio() is never called). The latter issue of failing to mark the page table as a kernel page table (ptdesc_set_kernel() is never called) is far more problematic. Since commit: |
||
|
Lorenzo Stoakes (ARM)
|
d5d8b8662e |
x86/mm/pat: Acquire init_mm read lock on attribute changes to avoid UAF
A previous commit protected against races between ptdump and CPA collapse,
however one still exists between attribute changes and collapse as reported
by Denis V. Lunev (linked).
When an attribute change arises, a lockless page table walker obtains a PTE
entry, which is later written to via set_pte_atomic():
...
-> change_page_attr_set_clr()
-> __change_page_attr_set_clr()
-> __change_page_attr()
-> _lookup_address_cpa()
-> lookup_address_in_pgd_attr()
-> [ lockless page table walker ]
-> set_pte_atomic()
There is nothing preventing a concurrent CPA collapse which can free the
PTE that was retrieved here, resulting in a use-after-free.
With the mmap write lock taken on init_mm over CPA collapse, resolve this
race by acquiring an mmap read lock on init_mm over
__change_page_attr_set_clr().
This locks across the whole operation over which the walk and the PTE
entry write occurs, solving the race.
It is safe to do this here, as no spinlocks are held upon entry to
__change_page_attr_set_clr().
However, the lock must not be held over an allocation, as allocation can
trigger reclaim and shrinkers may call into CPA recursively, making
deadlocks possible (init_mm -> ... -> fs_reclaim -> init_mm).
A page table is allocated when a huge page needs to be split:
-> change_page_attr_set_clr()
-> __change_page_attr_set_clr()
-> __change_page_attr()
-> split_large_page()
[ pagetable_alloc() ]
-> __split_large_page()
Avoid deadlocks by dropping the mmap lock across pagetable_alloc() in
split_large_page() and track whether this is needed by adding a new
'init_mm_read_locked' flag to struct cpa_data.
This is safe as __split_large_page() (called with locks re-established)
revalidates that the page table entry is the same as it was prior to the
locks being dropped and __change_page_attr() repeats the entire page table
walk whenever a split occurs, so concurrent split and collapse are
accounted for.
Concurrent ptdump is also safe as the lock is only dropped over page table
allocation during which time the page table has not yet been modified.
The CPA_COLLAPSE flag is only set by set_memory_rox(), which exclusively
operates upon vmalloc ranges, and on x86 only within the module mapping
space.
This is important, because some callers directly invoke
__change_page_attr_set_clr(), bypassing this lock. However, none of these
operate within the module mapping space.
* cpa_process_alias() - a recursive helper called by
__change_page_attr_set_clr().
* __set_memory_enc_pgtable() - operates on the direct mapping and (via
__vmbus_establish_gpadl()) the vmalloc mapping space.
* __set_pages_[n]p() - called by set_direct_map_[invalid, default,
valid]_noflush(), __kernel_map_pages() - operates on the direct map.
* kernel_[un]map_pages_in_pgd() - operates on EFI ranges.
This work is based upon Denis V. Lunev's excellent analysis of the bug
with gratitude.
[ dhansen: move to imperative voice in changelog ]
Fixes:
|
||
|
Lorenzo Stoakes (ARM)
|
a1c7570ced |
x86/mm/pat: Acquire init_mm write lock on collapse to avoid UAF
x86 implements page attribute modification using its Change Page Attributes (CPA) mechanism. This tracks properties of ranges such as cache mode through x86 page attributes, and as part of that logic manipulates kernel page tables. Since commit: |
||
|
Lorenzo Stoakes (ARM)
|
6cc27d8219 |
mm/vma: correctly unaccount on mmap_prepare() failure
__mmap_setup() accounts memory for relevant mappings via:
security_vm_enough_memory_mm()
-> __vm_enough_memory()
-> vm_acct_memory()
If __mmap_setup() fails, this indicates that this accounting did not take
place, and thus it's appropriate for __mmap_region() to jump to
abort_munmap.
However if call_mmap_prepare() fails, it also jumps there and any accounted
memory is not correctly unaccounted.
Fix this by handling each error separately.
Link: https://lore.kernel.org/20260902-fix-unaccount-mmap_prepare-v1-1-ea070189fdfb@kernel.org
Fixes:
|
||
|
Lorenzo Stoakes (ARM)
|
397432cab1 |
mm/mremap: account mm->locked_vm correctly for MREMAP_DONTUNMAP
When a VMA is mremap()'d with MREMAP_DONTUNMAP set, that results in the VMA being copied, but the source VMA not being unmapped. If the VMA is mlock()'d this is a legal operation, though the source VMA has its VMA_LOCKED_BIT cleared. However this is done in dontunmap_complete(), after mm->locked_vm was incremented via vrm_stat_account(), resulting in double-counting. Worse, this is not even corrected when source VMA is unmapped, due to the VMA_LOCKED_BIT flag having been cleared. This all works fine in the usual mremap() case (without MREMAP_DONTUNMAP), as the source VMA is unmapped with VMA_LOCKED_BIT intact, at which time mm->locked_vm is decremented accordingly. Resolve the issue by invoking vrm_stat_account() only after dontunmap_complete() has run. Note that MREMAP_DONTUNMAP requires old_len == new_len, so no need to account for a delta in size in this case. The bug was introduced by commit |
||
|
Lorenzo Stoakes (ARM)
|
e384abeb55 |
mm/huge_memory: bypass THP tuneables for huge pfnmap mappings
The sysfs THP tuneables at /sys/kernel/mm/transparent_huge_pages/ rather confusingly only control the behaviour of THP in some instances. They are not applicable to MADV_COLLAPSE operations, nor to DAX mappings. Long-term, THP is predicated upon compaction being able to obtain large folios to populate THP ranges. However, vm_normal_folio() returns NULL for PFN map mappings, thus their reference count is maintained by the driver, not core mm. As a consequence, the folios are not subject to reclaim nor compaction, so are not truly part of the THP mechanism at all. However, since commit |
||
|
Lorenzo Stoakes (ARM)
|
97d34aa65c |
mm/secretmem: properly account locked pages
secretmem accounts folios by treating memory as if it were mlock()'d and
thus limited by the RLIMIT_MEMLOCK limit.
However the folios are unevictable and remain so until the inode is
evicted, eliminating usual mlock() semantics - mapping folios then
unmapping them does not clear their unevictable state, since it depends on
AS_UNEVICTABLE, not PG_mlocked.
A user can therefore easily work around the RLIMIT_MEMLOCK limit - simply
map then unmap and VmLck no longer counts the secretmem range. Worse,
folios are not accounted in the process's RSS, meaning the OOM killer
won't know to kill the process.
Repeatedly mapping/unmapping (or forking) can then result in the
consumption of all available system memory with unevictable folios and
cause system instability.
A secretmem fd can be passed between processes and over fork so a
per-process limit simply does not make sense, so follow the precedent set
by io_uring, perf, skbuff, iommufd and xdp by tracking the number of
locked pages in user_struct->locked_vm.
Since the scope tracked is actually inode lifetime, the RLIMIT_MEMLOCK
applies per-user not per-process, so it doesn't make sense to bypass for
users with CAP_IPC_LOCK, therefore remove this bypass.
There is simply no reason to carry on marking the mapping as mlock()'d
since it's misleading and the lifecycle is now correctly handled, so
remove this too.
Note that secretmem does not support any form of truncation (including
hole punching) and the folios are unreclaimable, so the folios need only
be accounted on fault and unaccounted on inode destruction.
__secretmem_account_pages() is more or less a duplicate of the code that
io_uring etc. use, but since this is a bug fix that needs backporting,
defer any de-duplication efforts to a follow-up.
test_mlock_limit() asserts mlock_future_ok() on mmap(), however this has
been removed, so remove the test altogether for the fix. A new test will
be sent separately for upstream.
Link: https://lore.kernel.org/20260826-secretmem-accounting-v3-1-94cb04399510@kernel.org
Fixes:
|
||
|
Lorenzo Stoakes (ARM)
|
35b0fb391b |
mm/mremap: reset unfaulted VMA page offset for MREMAP_DONTUNMAP
Uniquely an mremap() invocation using the MREMAP_DONTUNMAP flag can reset
a faulted VMA into an unfaulted one.
It does so after the page tables have been moved to the copied VMA with
MREMAP_DONTUNMAP leaving the old VMA in place which is naturally unfaulted
as the page tables it had are no longer present.
However, in doing so, it violates the invariant that the anonymous page
offset of an unfaulted VMA is vma->vm_start >> PAGE_SHIFT.
This is because a VMA may have been faulted in, mremap()'d (causing a
delta between its page offset and vma->vm_start >> PAGE_SHIFT), and then
mremap()'d again with MREMAP_DONTUNMAP resulting in the unfaulting.
This condition is a violation of a fundamental assumption in mm, but now
also triggers an assert in assert_sane_pgoff() which explicitly checks for
this condition.
Correct it by resetting the VMA's page offset at the point of completing
the MREMAP_DONTUNMAP operation.
Link: https://lore.kernel.org/20260825-fix-mremap-dontunmap-pgoff-v1-1-39a40b2c98b3@kernel.org
Fixes:
|
||
|
Lorenzo Stoakes (ARM)
|
627824f20f |
MAINTAINERS: remove Lorenzo as THP co-maintainer
Unfortunately my workload is such that I simply no longer have the time to give THP the focus that it deserves. So, at least temporarily, step down from the role. Link: https://lore.kernel.org/20260826-drop-thp-maintainership-v1-1-3d102748fa17@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Kiryl Shutsemau <kirill@shutemov.name> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
3372b6631b |
MAINTAINERS: add drivers/char/mem.c to mm misc, memory mapping sections
This file is a 'special' driver that implements /dev/zero and /dev/mem among other things. As such it makes sense for mm to be cc'd on mails and to have some say in how things are changed there, so add it to the mm misc section. Uniquely, it provides the 'old way' of obtaining an anonymous mapping - MAP_PRIVATE of /dev/zero - so is directly tied to memory mapping, therefore also add it to the memory mapping section. scripts/get_maintainer.pl copes perfectly fine with files in multiple sections so everything should work correctly. Link: https://lore.kernel.org/20260812-add-drivers-mem-to-mm-maintainers-v1-1-6218b861f4c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: SJ Park <sj@kernel.org> Reviewed-by: Anshuman Khandual<anshuman.khandual@arm.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Jann Horn <jannh@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
6fd3e592c0 |
mm: add some missing includes to mm-local headers
There are a number of internal headers local to mm/ which reference functions and data types without including the relevant headers. mm/vma.h is a special case that intentionally does not include additional headers, but the others are not. This breaks tooling like clangd (which is where I noticed this), though the build is OK due to the C files including the headers happening to include required dependencies. It's better to be explicit about dependencies anyway, so add the missing includes and fix clangd as a bonus. Link: https://lore.kernel.org/20260804-fix-some-local-headers-v1-1-a7beb173c116@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
fb580e1964 |
tools/testing/selftests/mm: test anonymous page offset merge behaviour
While maintaining anonymous page offsets for VMAs has no impact for most merge cases, it does impact MAP_PRIVATE-mapped file-backed mappings which happen to have matching page offset but not matching anonymous page offset. Assert this behaviour by attempting to map an unfaulted MAP_PRIVATE-memfd region with a faulted one with compatible file page offsets but incompatible anonymous page offsets. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-16-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
6b7460ad1a |
tools/testing/vma: expand VMA merge tests to assert anon pgoff
Now we have introduced the VMA anonymous page offset attribute and update it when VMAs are manipulated, update VMA merge tests to assert that the anonymous page offset is as expected. Also update instances where we could use vma_start_pgoff() to do so. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-15-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
93c0c8dc87 |
mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios
Currently anonymous folios belonging to CoW'd MAP_PRIVATE file-backed
mappings are indexed by their page offset within the file in which they
were originally mapped.
This differs from anonymous folios belonging to pure anon mappings which
are indexed by their anonymous page offset (the address at which they'd
belong in the VMA when first faulted).
This change fixes this inconsistency, always indexing anonymous folios by
their anonymous page offset regardless of the VMA to which they belong.
The foundations have been laid such that we need only switch this
functionality on such by:
* Using linear_anon_page_index() in __folio_set_anon() to assign the
folio's index to the anonymous linear index rather than the file-backed
one.
* Otherwise using linear_anon_page_index() in all instances where
anonymous folios are being referenced or manipulated.
* Replacing vma_address() with vma_filebacked_address() or
vma_anon_address() as appropriate.
* Updating the merging logic to check that anonymous page offsets are
aligned as well as filebacked ones for MAP_PRIVATE file-backed VMAs,
introducing needs_adjacent_anon_pgoff() to figure out when this is
required.
* Updating linear_folio_page_index() to invoke linear_anon_page_index()
if the folio is anonymous.
* Updating vma_address_end() to use the VMA's anonymous page offset when
pvmw->pgoff is anonymous.
* Correcting folio_within_range() to use anonymous page offset for
anonymous folios.
This will have no impact on merging of anonymous VMAs, whose page offset
and anonymous page offset are identical, nor will it impact shared
file-backed VMAs, which will continue to be merged based on the
file-backed page offset.
However, MAP_PRIVATE file-backed mappings must now be aligned on anonymous
page offset as well.
In most instances this should have no impact on merging of file-backed
mappings, which are usually not merged all that often, let alone
MAP_PRIVATE mapped ones, and rarely remapped and faulted before being
moved back in place (the case in which a merge may now fail).
One subtle impact of this change is in NUMA interleaving - since commit
|
||
|
Lorenzo Stoakes (ARM)
|
5f653f8b7a |
mm/migrate: calculate large folio page index using PFN
Rather than having to figure out the page index to use using linear_page_index(), calculate it using PFN. This is a more natural fit as the linear page index is immaterial to determining the folio page index. Derive the page index from the offset between migration entry PFN and folio PFN - pvmw.pfn (set via DEFINE_FOLIO_VMA_WALK() which uses folio_pfn() to obtain it). Additionally remove a not so useful comment and clean the code layout up. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-13-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Suggested-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
8e658ecc3b |
mm/huge_memory: update remove_migration_pmd() to accept a folio
This function does not need to accept a page and requiring it to is unnecessary and misleading. make_[writable, readable]_device_private_entry() must be passed a PMD-aligned PFN as they immediately used to obtain a softleaf PMD entry and the same argument applies to folio_add_[anon, file]_rmap_pmd(). While we are here, update a VM_BUG_ON() to a VM_WARN_ON_ONCE(). No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-12-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
e7006fbd60 |
mm: clean up vma_address_end()
vma_address_end() is a confusing function with a lot of moving parts so clean it up prior to extending it for anon page indexed mappings. Const-ify some variables and establish pgoff_vma_start and pgoff_end variables to clearly identify the page offset for the start of the VMA and the end of the page offset range specified by the page walk. This simplifies the function significantly and lays the groundwork for a future change to update this function to account for anonymously page indexed folios. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-11-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
50c5f35a64 |
mm/rmap: track whether the page VMA mapped pgoff is anonymous
Update the page_vma_mapped_walk structure to track whether the pgoff being tracked is an anonymous pgoff or not and update the comments to reflect this. This is necessary in order to determine the correct VMA page offset in vma_address_end() when pvmw->nr_pages > 1. Also document that pvmw->pgoff is meaningless for pvmw->nr_pages == 1 and for KSM. Do not set this field where pgoff is not specified. This is laying the groundwork for eventually using anonymous page offsets as the index for all anonymous folios. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-10-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
6a993c7fbc |
mm: propagate VMA anonymous page offset on map, remap, split + merge
We must correctly update VMA anonymous page offset state on all VMA operations that would result in it changing, with special attention given to remapping. We cover most cases by simply updating vma_set_range() to do so (with a new anonymous page offset parameter), but also notably must update the merging and mapping logic to propagate this parameter correctly. The remap logic remains the same - we may update the anonymous page offset if the VMA is unfaulted, but now this applies to MAP_PRIVATE file-backed mappings too, so we update the code to reflect this. Note that we use __linear_anon_page_index() upon remap as the VMA may be shared, in order that we update the field consistently regardless of VMA type. Similarly, pass through anon page offset to the merge logic, updating the vma_merge_struct struct to propagate it, and also use __linear_anon_page_index() to obtain the anonymous page index so it can be safely used for both shared and MAP_PRIVATE file-backed mappings. In copy_vma(), the anonymous page offset is updated regardless of whether the mapping is a CoW mapping or not. This is both to keep the anonymous page offset consistent even for non-CoW mappings (it is set so should at least remain correct) and makes the logic cleaner. A self-merge however remains permitted only for mappings which can have a populated vma->anon_vma and do not require alignment on a separate file offset - that is pure anonymous VMAs, so only set can_self_merge if vma_is_anonymous(). Finally, we update insert_vm_struct() to correctly set the anonymous page offset on insertion of a VMA. We simply ensure state is correctly propagated here, so no functional changes are intended. Also update VMA userland tests to reflect this change. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-9-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
746b9e0a47 |
tools/testing/vma: add tests for copy_vma() self-merge
Assert that a VMA can be moved backwards, forwards and between a preceding VMA and its old self. In the cases in which the VMA merges only with itself expect that to be achieved by expanding its old self, so assert that these function correctly. However in the case of a merge between a preceding VMA and itself the original VMA is removed, so assert that the preceding VMA replaces the one passed in as vmap and the merge is as expected. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-8-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
9998bc06d7 |
mm/vma: fix self-merge check in copy_vma()
The existing logic is very confusing so improve things. Firstly rename the confusing faulted_in_anon_vma variable to can_self_merge and update this when the page offset is updated. What is being checked for is a 'self-merge' - that is between the VMA being remapped and its prior VMA (remember that this is copy_vma() - if a non-MREMAP_DONTUNMAP remap the original VMA is only removed afterwards). This can happen if the VMA is moved immediately adjacent to itself, either before or after it: |----------------|----------------| | | | v | v |...............||---------------||...............| | new || old || new | |...............||---------------||---------------| In these cases the old VMA is simply expanded to cover the new range. It is also possible for the move to both self-merge and merge with a prior VMA if it is placed between a preceding VMA and its old self: |---------------| | | v | |---------------||...............||---------------| | prev || new || old | |---------------||...............||---------------| In this case, the old VMA is removed and 'prev' is expanded and replaces it. Since copy_vma_and_data() which calls copy_vma() intends to reference the old VMA after the merge, it must have this pointer updated. This kind of self-merge is not possible with a succeeding merge, as the merge always prefers to expand the preceding VMA if possible. copy_vma() accounts for this by explicitly checking to see if a self-merge occurred and updating the vmap pointer if so. However it incorrect did so even for a subsequent merge (this is simply a noop so it had no impact). So change this to only check for the case which matters - a backwards merge - and rearrange the parameters to make it clearer we're doing that - i.e. check new_vma->vm_start < old_vma_start (having already renamed vma_start to old_vma_start to make it clear this is the previous VMA). Also update the existing wall-of-text comment to be a lot clearer. While we're here, replace the VM_BUG_ON_VMA() with a VM_WARN_ON_ONCE_VMA() and update the VMA userland tests accordingly. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-7-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
dba10745e9 |
mm: introduce and use vma_filebacked_address()
In cases where we know that the VMA is file-backed, use vma_filebacked_address() rather than vma_address(). This lays the foundation for using the anonymous page offset via vma_anon_address(). Also add an assert to ensure that the VMA whose address is required is not anonymous. No functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-6-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Tested-by: syzbot@syzkaller.appspotmail.com Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
2a8de2d1d6 |
mm: update print_bad_page_map() to show anon index if appropriate
If the VMA is a CoW mapping page offset may differ from anon page offset, indicating different positions in the relevant rmap trees. Update print_bad_page_map() to reflect that - if the mapping is non-CoW or the indexes match, then output only one index as before, otherwise output both with (file) or (anon) suffixes to reflect which is which. It's not possible to give only one index as there is no folio available to perform folio_test_anon() upon (the page table entry is bad so this is unavailable). This is potentially useful debugging information and matches the existing page offset provided. Use the raw __linear_anon_page_index() function so as to always output this value regardless of whether the mapping is file-backed or not and to avoid asserts that shouldn't apply here. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-5-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Gregory Price (Meta) <gourry@gourry.net> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
c02fe674fe |
mm: abstract vma_address() and introduce vma_anon_address()
Introduce __vma_address() which abstracts the VMA start page offset field as pgoff_start, then update vma_address() to use it. Then introduce vma_anon_address() which does the equivalent of vma_address(), only using the anonymous page offset of the VMA rather than the file-backed one. Also add an assert to ensure that the function is not called for mappings which are file-backed but not MAP_PRIVATE to ensure it is only used in the correct places. This will be necessary for determining the address of a folio's index within a VMA when the folio belongs to a MAP_PRIVATE file-backed VMA but has been CoW'd, and thus is anonymous, once the anonymous VMA page offset field is used for the reverse mapping. No callers are updated, so no functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-4-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
7e6543d1f9 |
mm: introduce linear_anon_page_index()
This function provides the anonymous equivalent of linear_page_index(), instead offsetting based on the anonymous page offset of the VMA. It is valid only for anonymous or MAP_PRIVATE file-backed mappings, in other words CoW mappings. For pure anon VMAs, this will be equal to linear_page_index(). Assert that both of these invariants are true in linear_anon_page_index() and implement the algorithm in __linear_anon_page_index(). Note that MAP_PRIVATE-/dev/zero mappings will satisfy vma_is_anonymous() but not fulfill this invariant, so when asserting this we check vma->vm_file to account for this. We do not update callsites yet, so no functional change intended. Also const-ify vma_is_anonymous() to make it compatible with the const-ified linear_anon_page_index(). While we're here, update linear_page_index() to be more succinct. VMA userland tests are also updated accordingly. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-3-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Reviewed-by: Gregory Price (Meta) <gourry@gourry.net> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
51943a18ad |
mm: provide vma_[flags_]is_cow_mapping() and remove is_cow_mapping()
All remaining callers of is_cow_mapping() are invoking it in the form of is_cow_mapping(vma->vm_flags) or an indirected version of this. Therefore, provide a helper - vma_is_cow_mapping() to directly test the VMA. Additionally provide a new helper vma_flags_is_cow_mapping() which performs the check using the new vma_flags_t type, and share this logic between vma_is_cow_mapping() and vma_desc_is_cow_mapping(). With these changes, no callers of is_cow_mapping() remain, so remove it. Also update the userland VMA tests to reflect the change. No functional change intended. [akpm@linux-foundation.org: fix kerneldoc comment typo, per Lorenzo] Link: https://lore.kernel.org/aob1goSSPH6sTN9y@gremlin Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-2-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Gregory Price (Meta) <gourry@gourry.net> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
0bd14001eb |
mm/vma: introduce VMA anon page offset field and add helpers
Patch series "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous pgoff", v5. In memory management we've managed to manufacture a great deal of confusion around the concept of anonymous memory. We have: 1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and swap-backed (thus for reclaim purposes, treated as anonymous). These are simple enough. 2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so present in the page cache and mapped by an address_space object, but whose folios are also swap-backed (thus treated as anonymous for reclaim purposes). 3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have vma->vm_file set, but which clears vma->vm_ops to satisfy vma_is_anonymous(), resulting in VMAs that were mmap()'d referencing a file, but are in every other sense anonymous, including the folios. 4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs and have file-backed folios until CoW'd, at which point those CoW'd folios are anonymous. This series fixes issue 3. In order for us to traverse VMAs using the reverse mapping, we require two fields - folio->mapping and folio->index. The first tells the rmap code where to look for VMAs, and the second tells it at which offset the folio starts within the referenced object. For anonymous folios, folio->mapping points at an anon_vma object. For file-backed folios, it points at an address_space. And: * For file-backed folios folio->index is simply the page offset of the start of the folio within the file. * For anonymous folios belonging to pure anon mappings, folio->index is equal to the anonymous page offset of the folio. * For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file page offset. This series establishes a new anonymous page offset property of VMAs to allow us to map anonymous folios at their anonymous page offset, consistent with pure anon. The purpose of doing so is to lay the foundations for the scalable CoW work. This is necessary because scalable CoW looks in the maple tree for the VMA located at folio->index << PAGE_SHIFT, before falling back to looking up tracked remaps if necessary. The MAP_PRIVATE file-backed case means that folio indices will very often conflict with one another and this remap tracking becomes substantially more contended, and of course the fast path can never be used. This also makes it possible, in future, to unshare anonymously mapped folios with deep fork hierarchies on remap, eliminating the need for remap tracking in the vast majority of cases. Similar to page offset of pure anonymous VMAs, we update the anonymous page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd (i.e. vma->anon_vma is non-NULL). Overall, there is little impact on mergeability, which remains exactly the same for pure anonymous and shared file-backed mappings, with the only impact being on MAP_PRIVATE-mapped file-backed mappings, which must now match on anonymous page offset as well as file page offset to be merged. To fail to merge like this would require CoW'ing the mapping, then finding another VMA with identical file and compatible page offset to remap next to. This is therefore very much an edge case that should have very little impact (and which scalable CoW may very well address in any case). This patch (of 16): Establish fields in vm_area_struct to store the anonymous page offset of VMAs. Initially, the anonymous page offset of a VMA is vma->vm_start >> PAGE_SHIFT. When a VMA is remapped to new_address its anonymous page offset is either updated to new_address >> PAGE_SHIFT if unfaulted or, if faulted, remains equal to the anonymous page offset it had when first faulted. Currently, anonymous folios belonging to CoW'd MAP_PRIVATE-mapped file-backed VMAs are tracked by their file offsets. By adding anonymous offset as a property of VMAs, we can now track them by their anonymous page offset instead. By tracking this, we provide the means by which to eliminate this inconsistency, and more importantly lay the foundations for future work for the scalable CoW anonymous rmap rework. This patch simply adds the fields and some simple helpers. Subsequent patches will update mm code to make use of these fields correctly. The fields chosen are packed in the VMA such that, for 64-bit kernel builds, no additional space is taken up. The first field is present on cacheline 0 containing key VMA fields, and the second on cacheline 3, which contains file-backed reverse mapping fields. Given the relative time spent accessing reverse mapping fields as well as updating them, there shouldn't be any performance impact here from false sharing. Update the VMA userland tests to account for this change. No callsites are updated yet, so no functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-0-c21581c0c3c8@kernel.org Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-1-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Gregory Price (Meta) <gourry@gourry.net> Reviewed-by: Xu Xin <xu.xin16@zte.com.cn> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Alexander Deucher <alexander.deucher@amd.com> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Baoquan He <baoquan.he@linux.dev> Cc: Barry Song <baohua@kernel.org> Cc: Boris Brezillon <boris.brezillon@collabora.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Christan König <christian.koenig@amd.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: Dave Airlie <airlied@gmail.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Harry Yoo <harry@kernel.org> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Huang Ray <Ray.Huang@amd.com> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Ian Rogers <irogers@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: James Clark <james.clark@linaro.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Janosch Frank <frankja@linux.ibm.com> Cc: Jason Gunthorpe <jgg@ziepe.ca> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Hubbard <jhubbard@nvidia.com> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kees Cook <kees@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liviu Dudau <liviu.dudau@arm.com> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Cc: Marc Rutland <mark.rutland@arm.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Matthew Auld <matthew.auld@intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Maxime Ripard <mripard@kernel.org> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Namhyung kim <namhyung@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Nico Pache <npache@redhat.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rik van Riel <riel@surriel.com> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Price <steven.price@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Thomas Zimemrmann <tzimmermann@suse.de> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: xu xin <xu.xin16@zte.com.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
aec6a8d80e |
Bluetooth: hci_bcm4377: Ignore reserved PHY in ext adv reports on BCM4378
Commit |
||
|
Lorenzo Stoakes (ARM)
|
1b300f679a |
mm/mseal: remove further superfluous comments, do_mseal()
There's no need to abstract do_mseal() any longer so put the system call implementation in the system call declaration. The comment around do_mseal() is strangely formatted, overly long and adds a lot of superfluous information that the code already provides, so boil it down to the essentials. Link: https://lore.kernel.org/20260717-mseal-fixups-v2-3-0daa0014b813@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christian Brauner <brauner@kernel.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Kees Cook <kees@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
4c1c2d2418 |
mm/mseal: limit scope of mseal address zero to address zero
Commit
|
||
|
Lorenzo Stoakes (ARM)
|
14dad135d6 |
mm/mseal: remove superfluous comments, fix confusion around mm
Patch series "mm/mseal: further cleanups", v2. The mseal implementation is still rather confusing, so tighten things up a little. The only user of do_mseal() outside of the system call is the MMAP_PAGE_ZERO process personality - retain better control over how mseal is utilised by providing mseal_mmap_page_zero() for this instead. The comments are overly long and confusion, so cut them down so they're a lot clearer. Remove confusing mm_struct params (mseal can not be used on remote mm's) and wrap the actual system call logic into the system call declaration. This patch (of 3): Remove comment blocks that don't add value and eliminate any confusion about whether or not we permit mseal()'ing of remote mm's by not passing through an mm parameter but rather referencing current->mm in each function. Also while we're here, avoid an ugly goto by using an else branch, and move local parameters declarations into reverse xmas tree order. No functional change intended. Link: https://lore.kernel.org/20260717-mseal-fixups-v2-0-0daa0014b813@kernel.org Link: https://lore.kernel.org/20260717-mseal-fixups-v2-1-0daa0014b813@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christian Brauner <brauner@kernel.org> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Kees Cook <kees@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
f4a31afd96 |
mm/mremap: convert mremap code to use vma_flags_t
Replace use of the legacy vm_flags_t flags with vma_flags_t values throughout the mremap logic. Note that, in replacing vm_flags_clear() (which takes the VMA write lock) with vma_clear_flags() and vma_clear_flags_mask() (which do not) respectively in unmap_source_vma() and dontunmap_complete(), we do not add a VMA write lock to account for htis. This is because, in both cases, move_vma() is their calling function and this has already acquired the VMA write lock on vrm->vma whose VMA flags are being cleared. In the case of vma_set_flags() in unmap_source_vma() we do need to do this - as prev and next were not necessarily write locked at this point. Additionally update comments to reflect the changes to be consistent. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-13-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
31b50515a7 |
mm/mprotect: convert mprotect code to use vma_flags_t
Replace use of the legacy vm_flags_t flags with vma_flags_t values throughout the mprotect logic. Note that we retain the legacy vm_flags_t bit shifting code in do_mprotect_pkey(), deferring a vma_flags_t approach to this for the time being. Additionally update comments to reflect the changes to be consistent. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-12-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
2fa484226b |
mm/mlock: convert mlock code to use vma_flags_t
Replace use of the legacy vm_flags_t flags with vma_flags_t values throughout the mlock logic. Additionally update comments to reflect the changes to be consistent. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-11-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
df751fe672 |
mm/vma: convert miscellaneous uses of VMA flags in core mm
Update various uses of legacy flags in vma.c and mmap.c to the new vma_flags_t type, updating comments alongside them to be consistent. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-10-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
8e7501a361 |
mm/vma: update create_init_stack_vma() to use vma_flags_t
Replace use of the legacy vm_flags_t flags with vma_flags_t values in create_init_stack_vma(). As part of this change we add VMA_STACK_EARLY and VMA_STACK_INCOMPLETE vma_flags_t defines, and slightly rework create_init_stack_vma() for clarity. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-9-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
a2fdfd0080 |
mm: introduce vma_get_page_prot() and use it
There's a large number of vm_get_page_prot(vma->vm_flags) invocations. Make life easier by introducing vma_get_page_prot() parameterised by the VMA. This also makes converting vm_get_page_prot() to vma_flags_t easier. Also update the userland VMA tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-8-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Zi Yan <ziy@nvidia.com> Acked-by: Jani Nikula <jani.nikula@intel.com> # for i915 Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de> [DRM] Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
e28266b033 |
mm/vma: rename vma_get_page_prot to vma_flags_to_page_prot
Having vma_get_page_prot() refer to VMA flags and vma_set_page_prot() refer to a VMA is confusing. Rename vma_get_page_prot() to vma_flags_to_page_prot() to resolve this confusion. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-7-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
cc593403b1 |
mm/vma: convert vm_pgprot_modify() to use vma_flags_t and rename
Update vm_pgprot_modify() to use the new VMA flags type vma_flags_t and rename to vma_pgprot_modify() accordingly. This is part of the ongoing work to convert vm_flags_t to vma_flags_t, in order to eliminate the arbitrary limit of the number of bits in a system word on available VMA flags. Update VMA userland tests accordingly, updating vma_set_page_prot() to no longer inline vma_pgprot_modify(), rather we can simply define vma_pgprot_modify() as a static inline function and the tests will pick it up from vma.h. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-6-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
96c09ae60b |
mm: prefer mm->def_vma_flags in mm logic
Currently mm->def_flags (of type vm_flags_t) is union'd with mm->def_vma_flags (of type vma_flags_t). As part of the effort to convert vm_flags_t usage to vma_flags_t (in order to no longer be arbitrarily limited to a system word size for VMA flags), prefer mm->def_vma_flags to mm->def_flags throughout the mm logic. We update dump_mm() to use the %*pb format which means we make no assumption about the number of VMA flag bits on output when outputting default VMA flags. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-5-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
8d80043131 |
mm: update generic_get_unmapped_area[_topdown]() to use vma_flags_t
As part of the changes converting VMA flags from a system word size to a bitmap, extend this change to generic_get_unmapped_area() and generic_get_unmapped_area_topdown(), which also allows us to convert stack_guard_placement() as well. We retain arch_get_unmapped_area() and arch_get_unmapped_area_topdown() as-is for now, using legacy_to_vma_flags() as necessary to do so. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-4-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
2ec748152d |
mm: convert __get_unmapped_area() to use vma_flags_t
Update __get_unmapped_area() to be parameterised by vma_flags_t rather than vm_flags_t as part of the effort to move VMA flags from a system word to a bitmap. We cascade the changes up to arch_get_unmapped_area_topdown() and arch_get_unmapped_area(), where, for now, we use vma_flags_to_legacy() in order to propagate the VMA flags. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-3-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
1944a0183c |
mm/vma: update do_mmap() to use vma_flags_t
The core do_mmap() function accepts a vm_flags_t parameter which it then manipulates before passing to mmap_region() to do the heavy lifting of the memory mapping. Update do_mmap() to instead accept a vma_flags_t parameter, and adjust all the logic within do_mmap() to manipulate this instead. This is as part of the ongoing effort to convert VMA flags from a system word size to a bitmap type which allows us to unrestrict the number of VMA flags, as well as gain control over how VMA flag manipulation occurs. We do not cascade these changes to all functions which accept vm_flags_t, but rather use vma_flags_to_legacy() where necessary, specifically deferring converting calc_vm_prot_bits(), calc_vm_flag_bits() and __get_unmapped_area() to vma_flags_t. Also utilise the new vma_flags_can_grow() predicate which correctly handles the case of architectures without upward growing stacks. As part of this change, introduce VMA_SHADOW_STACK so we can correctly handle the case of the shadow stack not being defined. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-2-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
3a50308b39 |
mm: introduce vma_flags_can_grow() and vma_can_grow()
Patch series "mm: convert more vm_flags_t users to vma_flags_t", v2. This series makes further progress in converting usage of the deprecated vm_flags_t type to its replacement, vma_flags_t. It focuses on mm, though updates some users of mm APIs also. It updates: * The core do_mmap() code path for VMA mapping. * Unmapped area logic. * The usage of mm->def_vma_flags. * VMA page protection bit logic. * General usage of VMA flags in core mm code, mlock, mprotect, mremap. This patch (of 13): These test whether the VMA has stack semantics, i.e. is able to grow upwards or downwards depending on the architecture. In order to account for arches which do not support upward-growing stacks, introduce VMA_GROWSUP whose definition depends on the architecture supporting it, and use vma_flags_test_single_mask() in vma_flags_can_grow() to account for this. No functional change intended. Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-0-0fa2357d5431@kernel.org Link: https://lore.kernel.org/20260711-b4-vma-flags-mm-v2-1-0fa2357d5431@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Dave Airlie <airlied@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Mike Rapoport <rppt@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Zi Yan <ziy@nvidia.com> Cc: Jani Nikula <jani.nikula@intel.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
98bd3af0bb |
mm/huge_memory: separate out CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic
Rather than mixing the refcounted and non-refcounted
CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic, separate the two out cleanly so
it is clear what happens when this configuration option is set and what
happens when it is not.
Introduce HUGE_ZERO_UNSET_PFN to abstract the ~0UL assignment, only
introduce the refcount, lock and shrinker if
!CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, abstract initialisation and teardown,
abstract the huge zero folio allocation from refcounting.
Also change a BUG_ON() to WARN_ON_ONCE() while we're at it.
No functional change intended.
Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-2-c5d8a41b317f@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Fixes:
|
||
|
Lorenzo Stoakes (ARM)
|
33192a26cd |
mm/huge_memory: fix huge_zero_pfn race
Patch series "mm/huge_memory: fix huge_zero_pfn race", v2.
There is a subtle race in the reference-counted huge_zero_folio
implementation.
The fast path atomic logic fails to account for the fact that the shrinker
(which drops the final huge_zero_refcount pin) can overwrite huge_zero_pfn
with the ~0UL sentinel value in shrink_huge_zero_folio_scan() after a
racing get_huge_zero_folio() installed a valid value there.
This results in huge_zero_folio being correctly set but huge_zero_pfn
being set incorrectly and thus is_huge_zero_pfn() and consequently
is_huge_zero_pmd() will misidentify the huge zero folio as being an
ordinary THP folio.
This can result in the huge zero folio being split and otherwise treated
incorrectly.
The solution to this is very subtle as there is an atomic fast path, and
thus ordering in weakly ordered architectures has to be treated very
carefully.
The first commit fixes the issue by introducing a spinlock around
huge_zero_[pfn, folio, refcount] write, with careful consideration paid to
load/store ordering in the fast path. It is placed first and kept as
small as possible so that it can be backported on its own.
The second commit is a pure cleanup which reworks the
CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic to better separate the persistent
logic from the dynamically allocated one.
This patch (of 2):
If !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, the huge_zero_folio is refcounted
by huge_zero_refcount and returned by mm_get_huge_zero_folio().
When the caller is done with the huge zero page, its reference count is
decremented. Only a shrinker can set the reference count to zero.
A race can unfortunately occur between a shrinker decrementing the
reference count to zero and a concurrent page fault.
This is because shrink_huge_zero_folio_scan() might, if very unlucky, be
preempted between setting huge_zero_refcount to zero and writing an
invalid value.
During this time get_huge_zero_folio() could write to huge_zero_pfn before
shrink_huge_zero_folio_scan() resumes.
In this event the huge zero folio will be persistently misidentified
causing the THP code path to be entered inappropriately for the huge zero
folio:
CPU 0 CPU 1
=======================================|=================================
shrink_huge_zero_folio_scan() |
atomic_cmpxchg() sets refcount to 0 |
xchg() sets huge_zero_folio to NULL | get_huge_zero_folio()
| | atomic_inc_not_zero() -> zero
preempted for a long time | Allocate new huge zero folio
| | Write valid huge_zero_folio
v | Write valid huge_zero_pfn
Overwrite huge_zero_pfn with ~0UL <--- Invalid overwrite!
This results in is_huge_zero_pfn() and is_huge_zero_pmd() incorrectly
returning false for a huge zero page which could result in issues like the
huge zero folio being incorrectly split.
Note that the issue is with huge_zero_pfn not huge_zero_folio, as
get_huge_zero_folio() uses cmpxchg() gated on huge_zero_folio being NULL
with a retry loop and shrink_huge_zero_folio_scan() uses xchg() to set
huge_zero_folio.
Fix the issue by introducing a spinlock, huge_zero_lock, to prevent
concurrent write of huge_zero_folio, huge_zero_pfn and huge_zero_refcount.
There needs to be significant care taken here to ensure correctness:
The fast path in get_huge_zero_folio() uses atomic_inc_not_zero(), which
is outside of the critical section, and means huge zero allocation is
gated on zero huge_zero_refcount.
The fast path doesn't use huge_zero_lock, so the critical section is
irrelevant to it.
So invariants are required - huge_zero_refcount MUST:
* Only be set in the huge_zero_lock critical section to ensure
serialisation of huge_zero_pfn, huge_zero_folio and huge_zero_refcount
writes.
* Be set non-zero only AFTER huge_zero_[pfn, folio] are set to valid values
so installation of the huge zero folio on read page fault ensures
concurrent is_huge_zero_*() calls correctly identify the huge zero folio.
* Be set zero only BEFORE huge_zero_[pfn, folio] are set to NULL and ~0UL
respectively, and atomically.
Establish these by:
* Only setting huge_zero_refcount to zero or an absolute value in the
huge_zero_lock critical section in get_huge_zero_folio() and
shrink_huge_zero_folio_scan(), and always updating atomically there
and elsewhere.
* Using atomic_set_release(&huge_zero_refcount) in get_huge_zero_folio()
after huge_zero_[pfn, folio] are set. This is paired with
atomic_inc_not_zero() to ensure atomic_inc_not_zero() only observes a
non-zero value if huge_zero_[pfn, folio] are set.
* Using atomic_cmpxchg() in shrink_huge_zero_folio_scan() (as before) to
ensure that it is set zero only when equal to 1 and set atomically.
* atomic_cmpxchg() being fully ordered ensures this is done prior to
huge_zero_[folio, pfn] being set to NULL and ~0UL respectively.
Eliminate the retry loop in get_huge_zero_folio() as the atomic_cmpxchg()
in shrink_huge_zero_folio_scan() is now performed under the lock, and
replace with an equally locked atomic_inc() to set the reference count
should the caller be raced on huge zero folio installation.
folio_put() naturally implies a full memory barrier so its ordering is
maintained correctly.
The huge zero folio also cannot be released except when the shrinker does
so as it is non-LRU and non-rmappable.
Note that only the huge zero shrinker (via shrink_huge_zero_folio_scan())
can actually set huge_zero_refcount to zero, which is the count of mm's
which have at least one huge zero folio installed plus one shrinker pin.
Additionally convert a BUG_ON() to a VM_WARN_ON_ONCE().
Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-0-c5d8a41b317f@kernel.org
Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-1-c5d8a41b317f@kernel.org
Fixes:
|
||
|
Lorenzo Stoakes (ARM)
|
9d3277b2c0 |
arm64: remove redundant concurrent ptdump UAF mitigation
This partially reverts commit |
||
|
Lorenzo Stoakes (ARM)
|
27c32e5538 |
mm/ptdump: always stabilise against page table freeing using init_mm
Previous commits have established the invariant that kernel page table freeing is performed while an mmap read lock on init_mm is held, which fixes races between ptdump and kernel page table freeing over init_mm. However, x86 and arm64 can perform a ptdump over an mm other than init_mm via ptdump_walk_pgd() and since kernel memory ranges are shared across non-kernel mm's, this means that the race still exists for these cases. Fix this by acquiring a nested mmap write lock for init_mm in ptdump_walk_pgd(). This is safe as we take this after mmap write locking the mm, and nothing acquires the init_mm lock first before locking an arbitrary mm, so no deadlock is possible. Also update walk_page_range_debug() to assert that init_mm is write locked, add a comment explaining why and remove some redundant code, and eliminate the unnecessary and confusing invocation of walk_kernel_page_table_range(). We can safely remove the non-NULL check for walk.mm, as the mmap lock asserts would NULL pointer deref if it was (and of course no callers do this). The first point at which ptdump can race kernel page table freeing is commit |
||
|
Lorenzo Stoakes (ARM)
|
26444eb714 |
mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF
Patch series "mm: fix UAF caused by race between ptdump and vmap pgtable freeing", v6. Kernel page table walkers fall into two broad categories - those ranges where no exclusion is required via walk_kernel_page_table_range_lockless() and those where exclusion is required via walk_kernel_page_table_range() or walk_page_range_debug(). The former category is used only by arm64 arch code operating on ranges it both wholly owns and does not concurrently write. The latter category consists of kernel page table walkers operating on ranges that are wholly owned (but which need exclusion against concurrent writers). The lock used for exclusion is the mmap lock, and for kernel ranges this is the mmap lock on init_mm. ptdump is a special case being both the only user of walk_page_range_debug(), and the only case in which it walks ranges it does not own. This presents a problem, as page tables may be freed under ptdump. And indeed there is a use-after-free bug in the kernel as a result, which this series addresses. vmap promotes page tables to huge leaf entries where possible, freeing the lower page table when it does. It does this with no meaningful locks held against concurrent ptdump walks. As a result, use-after-free can currently occur. This series addresses the issue by having the vmap huge promotion logic acquire the mmap read lock while both setting the huge page table entry and freeing the prior leaf page table. The ptdump code already acquires the mmap write lock, so by doing so we ensure that the ptdump walker only ever observes either the huge page table entry or the existing page table entry, and nothing is freed underneath it. A mitigation for this issue was already applied for arm64 in commit |
||
|
Lorenzo Stoakes
|
f1966d954b |
tools/testing/vma: output compared expression on ASSERT_[EQ, NE]()
Update the macros to output the compared values at hex for easier debugging when test asserts fail. We have to be careful not to re-evaluate expressions as they may have side-effects. So update the code to take local copies and use these for both the test and the debug output. Also remove unused IS_SET() macro. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-33-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
fe95ccb1ba |
tools/testing/vma: default VMA, mm flag bits to 64-bit
With all of the sanitisers turned on, setting the VMA and mm flag bits depth to 128 by default results in overly long build times. Reduce this to 64 - we can always manipulate these later for testing of larger bitmaps as needed. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-32-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
9e98b6dd46 |
mm/vma: use guard clauses in can_vma_merge_[before, after]()
Rather than combining a bunch of conditionals in a single expression, simplify by inverting the mergeability requirements into guard clauses. that is - instead of checking what must be true for the conditions to be met, instead check the inverse of the requirements and return false if any are true, defaulting to true. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-31-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
dd7d7d00c0 |
mm/vma: correct incorrect vma.h inclusion
The only files which should be including vma.h are the implementation files for the core VMA logic - vma.c, vma_init.c, and vma_exec.c. This is in order to allow for userland testing of core VMA logic. In this cases, vma_internal.h and vma.h are included, providing both the dependencies upon which the core VMA logic requires and its declarations. Userland testable VMA logic is achieved by having separate vma_internal.h implementations for userland and kernel. Callers other than the core VMA implementation should include internal.h instead. This header does not need to include vma_internal.h as it only contains the vma.h declarations, for which the includes already present suffice. Update code to reflect this, update comments to reflect the fact there are 3 VMA implementation files and document things more clearly. While we're here, slightly improve the language of the comment describing vma_exec.c. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-30-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
613562c29f |
mm/vma: introduce and use vma_set_pgoff()
In order to lay the foundation for work that permits us to track the virtual page offset of MAP_PRIVATE file-backed mappings, we abstract the assignment of vma->vm_pgoff to vma_set_pgoff(). We additionally add a lock check here using the newly introduced vma_assert_can_modify(). This asserts the VMA write lock if the VMA is attached. We also assert that, if this is an anonymous VMA and unfaulted, that its (virtual) page offset is equal to the page offset of the VMA's address. We must be careful about MAP_PRIVATE-/dev/zero which violates fundamental assumptions about anonymous memory, so we check for !vma->vm_file after using vma_is_anonymous() which these mappings satisfy. Additionally, we only perform the assert if CONFIG_MMU is defined, as nommu does not set vma->vm_pgoff = addr >> PAGE_SHIFT. This isn't really relevant to rmap as it has no anon rmap (nor needs it), but we must avoid it asserting falsely. All of this logic is kept in assert_sane_pgoff() to keep things clear. In order to maintain correctness given this assert, we also update __install_special_mapping() to invoke vma_set_range() after it's set vma->vm_ops (which determine whether the VMA is anonymous or not). We do not use vma_set_pgoff() in vm_area_init_from(), as at the point of forking, we don't necessarily have correct locking state. Updating vma_set_range() covers most cases, but in addition to this we also update insert_vm_struct(), compat_set_vma_from_desc() and nommu callers. We also update vma_add_pgoff() and vma_sub_pgoff() to use vma_set_pgoff(). While we're here, we drop a BUG_ON() and update insert_vm_struct()'s comment to reflect the fact anonymous mappings can be added here. Finally, we update the CONFIG_MMU, CONFIG_PER_VMA_LOCK defines in the VMA userland tests so IS_ENABLED() will work correctly with them. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-29-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
bcb52fc56e |
mm/vma: slightly rework the anonymous check in __mmap_new_vma()
Determine if the VMA is anonymous early on, so we separate the logic which invokes callbacks from the rest of the logic. This is in order that a subsequent commit which asserts correct virtual page offset for anonymous mappings correctly asserts this for new anonymous mappings. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-28-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
ce4ec991da |
mm/vma: update vmg_adjust_set_range() to offset pgoff instead
We are calculating the pgoff as an offset, since we have vma_add_pgoff() and vma_sub_pgoff() available, just offset this value directly and use __vma_set_range() for vma->vm_[start, end] values. We take care to update the range before offsetting the page offset, so the adjusted VMA's vm_start and vm_pgoff are mutually consistent at the point the page offset helpers operate - this matters once vma_set_pgoff() comes to assert invariants which relate the two. Doing so lays the foundation for future work which allows for use of virtual page offsets for MAP_PRIVATE-file backed mappings. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-27-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
6387563e6e |
mm/vma: update vma_shrink() to not pass start, pgoff parameters
vma_shrink() is only used by relocate_vma_down() to shrink the tail of a VMA. Therefore neither the start nor the pgoff parameters make any sense. It seemed we were passing the pgoff parameter solely to satisfy vma_set_range()'s requirement for pgoff being specified. Since vma_set_range() is now isolated to vma.c, we can simply introduce __vma_set_range() which sets only vma->vm_[start, end], and invoke this instead, removing start and pgoff from vma_shrink() altogether. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-26-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
775659f8fc |
mm/vma: make vma_set_range() static, drop insert_vm_struct() decl
With __install_special_mapping() moved to vma.c, vma_set_range() can be made into a static function there and is now completely isolated from the rest of mm. While we're here, we can also remove the insert_vm_struct() declaration from mm.h - the function is implemented in vma.c and already declared in vma.h, and has no users outside of mm. Also update the VMA userland tests to reflect this change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-25-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
7578fc092a |
mm/vma: move __install_special_mapping() to vma.c
This function is operating on VMAs and rightly belongs in vma.c, where it can be subject to VMA userland testing and allows us to isolate it from the rest of mm. The _install_special_mapping() function will remain in mmap.c as a wrapper, since this is used by architecture-specific code. Doing so allows us to isolate more functions in vma.c for the same reasons. This forms part of work to allow for tracking MAP_PRIVATE file-backed mappings by their anonymous virtual page offset, as doing so allows us to isolate and keep code that interacts with this together. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-24-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
4ea24922ae |
mm/vma: add and use vma_[add/sub]_pgoff()
Add helpers for adding or subtracting to a VMA's page offset, exposed internally for VMA users within mm in mm/vma.h. This is to lay the foundations for tracking anonymous page offset for MAP_PRIVATE file-backed mappings, where adding and subtracting from this value must be reflected in both the file and anonymous offsets. These are used on VMA split and downward stack expansion. No functional change intended. [akpm@linux-foundation.org: use linear_page_delta() in __split_vma(), per Vlastimal & Lorenzo] Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-23-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
e65f4dfac1 |
mm/vma: introduce vma_assert_can_modify()
vma_assert_write_locked() and vma_assert_attached() are useful for their own purposes, however VMA code absolutely does allow the modification of non-write locked VMAs if they are at that point detached (i.e. unreachable from anywhere). It's therefore useful to be able to assert that a VMA is either detached (modification doesn't matter) or write locked (you're explicitly locked for modification). Therefore introduce vma_assert_can_modify() for this purpose. While we're here, make vma_is_attached() available generally - if !CONFIG_PER_VMA_LOCK, then there's no sense in which a VMA is detached (vma_mark_detached() is a noop), so have this default to true in this case. Also update VMA userland tests to reflect this change, correcting the previously open-coded vma_assert_[attached,detached]() there. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-22-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
93d23dff8e |
mm: use linear_page_[index, delta]() consistently
There are a number of places where we open code what linear_page_index() and linear_page_delta() calculate. Replace this code with the appropriate functions for consistency. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-21-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Thomas Zimmermann <tzimmermann@suse.de> [DRM] Acked-by: Kai Huang <kai.huang@intel.com> # for sgx Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Pedro Falcato <pfalcato@suse.de> # for mm Reviewed-by: Ackerley Tng <ackerleytng@google.com> [guest_memfd] Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
ff8fbc9e08 |
mm/vma: remove duplicative vma_pgoff_offset() helper
This is doing what linear_page_index() does, so eliminate it and replace it with linear_page_index(). Update the VMA userland tests to reflect this change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-20-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
4276358f11 |
mm: prefer vma_[start,end]_pgoff() to vma->vm_pgoff in kernel/
Be consistent in using vma_start_pgoff() and vma_end_pgoff(), which clearly indicates which part of the VMA the page offset refers to and aids greppability. This is part of a broader series laying the ground to provide a virtual page offset for MAP_PRIVATE-file backed anon folios. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-19-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Marek Szyprowski <m.szyprowski@samsung.com> # for kernel/dma Reviewed-by: Gregory Price <gourry@gourry.net> Acked-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
5872168de5 |
mm/vma: use vma_start_pgoff(), linear_page_index() in mm code
There are many instances in which linear_page_index() (as well as linear_page_delta()) is open-coded, which is confusing and inconsistent. Additionally, vma->vm_pgoff doesn't necessarily make it clear that this is the page offset of the start of the VMA range. Doing so also aids greppability. So use vma_start_pgoff() in favour of directly accessing vma->vm_pgoff, and linear_page_index() where we can. This also lays the ground for future changes which will add an anonymous page offset in order to be able to index MAP_PRIVATE-file backed anon folios in terms of their virtual page offset. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-18-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: SJ Park <sj@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
1268e1572b |
mm: introduce and use linear_page_delta()
It's often useful to obtain the number of pages a given address lies at within a VMA. Add linear_page_delta() to determine this and update linear_page_index() to make use of it. Add comments to describe both linear_page_delta() and linear_page_index(). We refer the reader to the comment for vma_start_pgoff() so we don't duplicate information about the meaning of a VMA's page offset, which is also less likely to result in bitrot. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-17-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Ackerley Tng <ackerleytng@google.com> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
68d307442b |
mm/vma: minor cleanup of expand_[upwards, downwards]()
Adjust the stack expansion functions expand_upwards() and expand_downwards() such that they are expressed in terms of named constant values, and make use of vma_start_pgoff(). This clearly documents that we are referencing the page offset of the start of the VMA. Additionally this cleans up the overflow check in expand_upwards(). No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-16-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
a30574d073 |
mm/vma: refactor vmg_adjust_set_range() for clarity
Add comments with ASCII diagrams to describe what we're doing, avoid dubious use of PHYS_PFN(), and use vma_start_pgoff(). The most complicated scenario represented here is vmg->__adjust_next_start - when this is set, vmg->[start, end] actually indicate the range to be retained, so take special care to describe this accurately. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-15-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
dfe2b9abf6 |
mm/vma: clean up anon_vma_compatible()
Break up the existing very large conditional, add comments and use vma_[start/end]_pgoff() to make clearer what we're doing here. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-14-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
8802c4069e |
mm/vma: introduce and use vmg_pages(), vmg_[start, end]_pgoff()
In the VMA logic we often need to determine the number of pages in the specified merge range, as well as the start and end page offsets of that range. Introduce and use helpers for these purposes. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-13-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
271f11c7a5 |
MAINTAINERS: move mm/interval_tree.c to rmap section
This file implements code for the interval trees used by the file and anon rmap implementation, so belongs in the rmap section. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-12-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Pedro Falcato <pfalcato@suse.de> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Acked-by: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Gregory Price <gourry@gourry.net> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
b1e71c05fa |
mm/rmap: rename anon_vma_interval_tree_*() to anon_rmap_tree_*()
To be consistent with the newly renamed mapping_rmap_tree_*(), rename the anon_vma_interval_tree_*() helpers to anon_rmap_tree_*(). No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-11-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
0b099cc229 |
mm/rmap: rename anon_vma_interval_tree_*() params and use pgoff_t
Rename parameters used by anon_vma_interval_tree_*() functions: 'node' to 'avc', 'start/first' to 'pgoff_start', and 'last' to 'pgoff_last' to make clear what is being passed. Also, express page offsets in terms of pgoff_t to be consistent. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-10-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
68364437ec |
mm/rmap: parameterise anon_vma_interval_tree_*() by anon_vma
Similar to what we did with mapping_rmap_tree*(), let's declare anon_vma_interval_tree*() in terms of anon_vma rather than rb_root_cached. In each case the rb tree referenced is &anon_vma->rb_root, so just pass anon_vma and the functions can figure this out themselves. Also update the VMA userland tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-9-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
4b7a90a48b |
mm/rmap: rename vma_interval_tree_*() to mapping_rmap_tree_*()
The family of vma_interval_tree_() functions manipulate the address_space (which, of course, is generally referred to as 'mapping') reverse mapping, but are named the 'VMA' interval tree. VMAs may be mapped by an anon_vma, an address_space, or both. Therefore calling the mapping interval tree a 'VMA' interval tree is rather confusing. This is also inconsistent with the anon_vma_interval_tree_*() functions which explicitly reference the rmap object to which they pertain. Rename the vma_interval_tree_*() functions to mapping_rmap_tree_*() to correct this. We will rename the anon rmap functions similarly in a subsequent patch. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-8-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
5ec02309d7 |
mm/rmap: elide unnecessary static inline's in interval_tree.c
It's not necessary to declare these functions static inline as they are contained within a single compilation unit. This makes the anonymous interval tree code consistent with the newly updated file-backed interval tree code. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-7-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
7a67b96af0 |
mm/rmap: parameterise vma_interval_tree_*() by address_space
The file-backed mapping interval tree functions vma_interval_tree_*() accept a raw rb_root_cached pointer to determine the tree in which they are operating. However, in each case, this is always associated with an address_space data type. So simply pass a pointer to that instead to simplify the code, and more clearly differentiate between these operations and those concerning anonymous mappings. While we're here, make the generated interval tree functions static as they do not need to be used externally (any previously existing external users have now been removed). We also rename VMA parameters from 'node' to 'vma' as calling this a node is simply confusing, update the input index types to pgoff_t since they reference page offsets and rename the parameters to pgoff_start and pgoff_last. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-6-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
dbde17041c |
mm/rmap: update mm/interval_tree.c comments
Update the file comment to clarify that both file-backed and anonymous interval trees are provided, referencing the relevant data types for clarity. Also add comments to indicate which parts of the file apply to each. While we're here, convert the VM_BUG_ON_VMA() to VM_WARN_ON_ONCE_VMA(). Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-5-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
1219731c49 |
mm: introduce and use vma_end_pgoff()
We already have vma_last_pgoff() which retrieves the last page offset within a VMA. However, code often wishes to span a page offset range, which requires the exclusive end of this range. So provide this in vma_end_pgoff() and update vma_last_pgoff() to use this function. Also update the VMA userland tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-4-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Cc: Liam R. Howlett (Oracle) <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
3542bc07f4 |
tools/testing/vma: use vma_start_pgoff() in merge tests
Now we have the vma_start_pgoff() helper, update the merge tests to make use of it for consistency. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-3-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
f9711438df |
mm: add kdoc comments for vma_start/last_pgoff()
Describe what vma_start_pgoff() and vma_last_pgoff() actually provide in detail. This is in order that we can differentiate this between functions that will be added in a subsequent patch which provide a different page offset. We go to lengths to describe the edge cases that can be run into here. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-2-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Gregory Price <gourry@gourry.net> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
995f0eed5e |
mm: move vma_start_pgoff() into mm.h and clean up
Patch series "mm: make VMA page offset handling more consistent", v2. This series performs a series of cleanups and improvements around how the vma->vm_pgoff field is used. Folios belonging to file-backed mappings are simply indexed by the page offset within the file they map (excepting pfnmap and driver mappings which sometimes do things different). However, anonymous folios belonging to pure anonymous mappings are indexed by their "virtual" page offset, which is equal to addr >> PAGE_SHIFT at the time at which the VMA was first faulted in. The page offset of a VMA is stored in vma->vm_pgoff and indicates the page offset of the start of the VMA range, whether it be file-backed or anonymous. The work here both cleans up how we reference this field, as well as laying the foundations for a future series which addresses the inconsistency of CoW'd folios in MAP_PRIVATE-file backed mappings, which are indexed as if they were file-backed but behave as if they were anonymous. This future series will make it such that all anonymous folios are indexed by virtual page offset whether belonging to VMAs who satisfy vma_is_anonymous() or MAP_PRIVATE-mapped file-backed mappings. This series: * Exposes vma_start_pgoff() and updates the kernel to use it consistently. * Adds and uses the useful vma_end_pgoff() helper. * Parameterises the file-backed mapping helpers vma_interval_tree_*() by address_space rather than rb_root_cached. * Renames: the misleadingly-named vma_interval_tree_*() helpers to mapping_rmap_tree_*() to be consistent with the renamed anon_rmap_tree_*(). * Parameterises anon_rmap_tree_*() by anon_vma. * Moves mm/interval_tree.c to the rmap section. * Adds vmg_*() helpers for page offset. * Clarifies the confusing vmg_adjust_set_range() function. * Introduces linear_page_delta() to provide relative pgoff within a VMA. * Replaces open-coded versions of linear_page_delta() and linear_page_index() with invocations of these functions. * Introduces and uses vma_assert_can_modify() to account for whether a VMA can be modified (detached or write locked). * Adds and uses vma_[add,sub]_pgoff() to adjust VMA page offset. * Moves __install_special_mapping() to vma.c. * Makes vma_set_range() static and internal to vma.c. * Introduces and makes use of vma_set_pgoff(). * Fixes incorrect vma.h header inclusion. * Defaults VMA userland tests to 64-bit vma flags size. * Updates VMA userland tests to give better output on failure. * Various smaller cleanups. * Updated the vma_set_pgoff() assert to account for MAP_PRIVATE /dev/zero correctly. This patch (of 33): vma_last_pgoff() already lives there, so it's a bit odd to keep vma_start_pgoff() in mm/interval_tree.c. Move them together. These each return unsigned long, which pgoff_t is typedef'd to. Make this consistent and have these functions return pgoff_t instead. Additionally, express vma_last_pgoff() in terms of vma_start_pgoff(), since we wrap the vma->vm_pgoff access, we may as well use it here. Also while we're here, const-ify the VMA and cleanup a bit. Also update the VMA userland tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-0-2a5aa403d977@kernel.org Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-1-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Gregory Price <gourry@gourry.net> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Cc: Ackerley Tng <ackerleytng@google.com> Cc: Kai Huang <kai.huang@intel.com> Cc: Marek Szyprowski <m.szyprowski@samsung.com> Cc: SJ Park <sj@kernel.org> Cc: Thomas Zimmermann <tzimmermann@suse.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
3c77682d80 |
mm: move alloc tag to mm
The alloc tagging work is really mm-specific, so move alloc_tag.c to mm/ and additionally update the MAINTAINERS entry to place it within memory management and port over the Kconfig and Makefile code to mm. Link: https://lore.kernel.org/20260625184857.2193482-3-surenb@google.com Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Signed-off-by: Suren Baghdasaryan <surenb@google.com> Reviewed-by: SeongJae Park <sj@kernel.org> Tested-by: Hao Ge <hao.ge@linux.dev> Acked-by: Hao Ge <hao.ge@linux.dev> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Harry Yoo (Oracle) <harry@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Kent Overstreet <kent.overstreet@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes (ARM)
|
ab11176bd3 |
x86/elf: Correct comment for STACK_RND_MASK()
Commit
|
||
|
Lorenzo Stoakes
|
d699a5da10 |
docs: link AI coding assistants and tool-generated content pages
Both of these guidelines are complimentary to one another - one specifying guidelines on AI coding assistants, and the other specifying those for tool-generated content. Since they are complimentary to one another, provide a link to each in the other. Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Acked-by: SJ Park <sj@kernel.org> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Jeff Layton <jlayton@kernel.org> Signed-off-by: Jonathan Corbet <corbet@lwn.net> Message-ID: <20260702164901.121128-1-ljs@kernel.org> |
||
|
Lorenzo Stoakes
|
1cb6cf6f2b |
MAINTAINERS: add Lance as an rmap reviewer
Lance has been doing excellent work reviewing rmap series and has proven himself to be a great member of the community in general, so add him as an rmap reviewer. Link: https://lore.kernel.org/20260622155913.280355-1-ljs@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: SeongJae Park <sj@kernel.org> Acked-by: Harry Yoo (Oracle) <harry@kernel.org> Acked-by: Dev Jain <dev.jain@arm.com> Acked-by: Barry Song <baohua@kernel.org> Acked-by: Lance Yang <lance.yang@linux.dev> Cc: Jann Horn <jannh@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Rik van Riel <riel@surriel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
13a1e1a618 |
Revert "mm: limit filemap_fault readahead to VMA boundaries"
This reverts commit |
||
|
Lorenzo Stoakes
|
4f5b875926 |
mm/vma: eliminate mmap_action->error_hook, introduce error_override
Rather than providing a hook, simplify things by providing the ability to override mmap action errors. This allows us to more carefully validate the value provided and thus ensure only a valid error code is specified, and simplifies the interface. This way, we eliminate all hooks but mmap_prepare and allow only mmap actions to be specified (which core mm controls). This significantly improves robustness and eliminates any unnecessary code duplication in driver mmap hooks. We also update the /dev/mem logic (the only user) to use mmap_action->error_override instead. Link: https://lore.kernel.org/55d13f7d016b827c459946d46a56105635be111c.1780397980.git.ljs@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Jann Horn <jannh@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
8876dc0780 |
mm/vma: remove mmap_action->success_hook
This hook was introduced to work around code that seemed to absolutely require access to a VMA pointer upon mmap(). However, providing this hook leaves a backdoor to drivers getting access to the very thing mmap_prepare eliminates - a pointer to the VMA. Let's solve this contradiction by removing it. The key intended user was hugetlb, however it seems that the best course now is to avoid allowing all drivers the ability to work around mmap_prepare, and find a different solution there. Link: https://lore.kernel.org/f79434e6d30af6d92999be6b76e197f1847105fa.1780397980.git.ljs@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Jann Horn <jannh@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
17986198a7 |
drivers/char/mem: eliminate unnecessary use of success_hook
Patch series "remove mmap_action success, error hooks", v3. The mmap_action->success_hook was a strange beast added to enable code which appeared to absolutely require access to a VMA pointer to work correctly. Primarily this was for hugetlb, however a different approach will be taken there, as clearly more work is required to figure out a sensible way of converting hugetlb to use mmap_prepare. The other user was the memory char driver, specifically /dev/zero which has the unusual property of explicitly setting file-backed VMAs anonymous. Providing the success hook was always foolish, as it allowed drivers a way to workaround the restriction that they should not access a pointer to a not-yet-correctly-initialised VMA - which defeats the purpose of the mmap_prepare work. We can achieve the same thing in memory char driver without needing the success hook, so this series removes that, then removes the success hook altogether. The error hook is also unnecessary - the motivation for this was for functions which need to override the error code when performing an mmap action in order to avoid breaking userspace. We can achieve this by just providing a field for the error code. Doing this means we don't have to worry about the hook doing anything odd. We also add a check to ensure the error code is in fact valid. Again the memory char driver is the only current user of this, so this series updates it to use that. After this change mmap_action has no custom hooks at all, which seems rather more cromulent than before. This patch (of 3): /dev/zero, uniquely, marks memory mapped there as anonymous. This is currently achieved using the mmap_action->success_hook. However this hook circumvents the abstraction of VMA initialisation so it's preferable to do things a different way. To achieve this, this patch firstly defaults the VMA descriptor's vm_ops field to the dummy VMA operations, which is what file-backed VMAs default this field to. That way, we can detect whether a driver sets this field to NULL in order to mark it anonymous. We then introduce vma_desc_set_anonymous() to do this explicitly, and invoke it in mmap_zero_prepare(). This way, any driver which does not explicitly set desc->vm_ops, retains the dummy vm_ops as they would previously. We also update set_vma_user_defined_fields() to make clear that we are either setting vma->vm_ops to what is provided by the driver (or defaulting to dummy_vm_ops if not set), or setting the VMA anonymous. This lays the groundwork for removing the success hook. Link: https://lore.kernel.org/cover.1780397980.git.ljs@kernel.org Link: https://lore.kernel.org/010579cca6787cf7bb057ab1f7228978b10601c8.1780397980.git.ljs@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Jann Horn <jannh@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
43e7f18976 |
mm/huge_memory: use correct flags for device private PMD entry
Commit |
||
|
Lorenzo Stoakes
|
b4aea43cd3 |
mm/hugetlb: avoid false positive lockdep assertion
Commit |
||
|
Lorenzo Stoakes
|
83f9efcce9 |
Revert "mm/hugetlbfs: update hugetlbfs to use mmap_prepare"
This reverts commit |
||
|
Lorenzo Stoakes
|
619eab23e1 |
mm/vma: do not try to unmap a VMA if mmap_prepare() invoked from mmap()
The mmap_prepare hook functionality includes the ability to invoke
mmap_prepare() from the mmap() hook of existing 'stacked' drivers, that is
ones which are capable of calling the mmap hooks of other drivers/file
systems (e.g. overlayfs, shm).
As part of the mmap_prepare action functionality, we deal with errors by
unmapping the VMA should one arise. This works in the usual mmap_prepare
case, as we invoke this action at the last moment, when the VMA is
established in the maple tree.
However, the mmap() hook passes a not-fully-established VMA pointer to the
caller (which is the motivation behind the mmap_prepare() work), which is
detached.
So attempting to unmap a VMA in this state will be problematic, with the
most obvious symptom being a warning in vma_mark_detached(), because the
VMA is already detached.
It's also unncessary - the mmap() handler will clean up the VMA on error.
So to fix this issue, this patch propagates whether or not an mmap action
is being completed via the compatibility layer or directly.
If the former, then we do not attempt VMA cleanup, if the latter, then we
do.
This patch also updates the userland VMA tests to reflect the change.
Link: https://lore.kernel.org/20260421102150.189982-1-ljs@kernel.org
Fixes:
|
||
|
Lorenzo Stoakes (Oracle)
|
84f4928446 |
tools/testing/selftests: add merge test for partial msealed range
Commit
|
||
|
Lorenzo Stoakes (Oracle)
|
34c45804ae |
MAINTAINERS: update MGLRU entry to reflect current status
We are moving to a far more proactive model of maintainership within mm and thus put a great deal of emphasis on sub-maintainers being active within the community both in terms of code contributions and review. The MGLRU has not had much activity since being added to the kernel and the current maintainers who kindly stepped up have unfortunately not been able to contribute a great deal to it for over a year, nor engage all that heavily in review. As a result, and within no negative connotations implied whatsoever, it seems appropriate to downgrade the current maintainers to reviewers. At this time nobody is quite exercising the maintainer role in this area of the kernel, but there is encouraging activity from a number of people who are trusted elsewhere in the kernel, and who have contributed relevant work or review. Therefore add further reviewers, and at this stage - to reflect the reality on the ground - we will not have any sub-maintainers listed at all. Each of the files listed are shared with other sections in MAINTAINERS, so this doesn't reduce sub-maintainer coverage. Link: https://lore.kernel.org/20260326185629.355476-1-ljs@kernel.org Signed-off-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Acked-by: Axel Rasmussen <axelrasmussen@google.com> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Barry Song <baohua@kernel.org> Acked-by: SeongJae Park <sj@kernel.org> Acked-by: Kairui Song <kasong@tencent.com> Acked-by: Qi Zheng <qi.zheng@linux.dev> Acked-by: Yuanchu Xie <yuanchu@google.com> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Wei Xu <weixugc@google.com> Cc: Kalesh Singh <kaleshsingh@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Lorenzo Stoakes
|
3d3544a6c9 |
mm/vma: remove __vma_check_mmap_hook()
Commit |
||
|
Lorenzo Stoakes (Oracle)
|
5a62019807 |
mm/khugepaged: fix issue with tracking lock
We are incorrectly treating lock_dropped to track both whether the lock is currently held and whether or not the lock was ever dropped. Update this change to account for this. Link: https://lkml.kernel.org/r/7760c811-e100-4d40-9217-0813c28314be@lucifer.local Fixes: 330f3758a3bc ("mm/khugepaged: unify khugepaged and madv_collapse with collapse_single_pmd()") Signed-off-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Nico Pache <npache@redhat.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: David Hildenbrand <david@kernel.org> Cc: David Rientjes <rientjes@google.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Gregory Price <gourry@gourry.net> Cc: "Huang, Ying" <ying.huang@linux.alibaba.com> Cc: Hugh Dickins <hughd@google.com> Cc: Jan Kara <jack@suse.cz> Cc: Jann Horn <jannh@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Kefeng Wang <wangkefeng.wang@huawei.com> Cc: Liam Howlett <liam.howlett@oracle.com> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nanyong Sun <sunnanyong@huawei.com> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Peter Xu <peterx@redhat.com> Cc: Rafael Aquini <raquini@redhat.com> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Randy Dunlap <rdunlap@infradead.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Shivank Garg <shivankg@amd.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Takashi Iwai (SUSE) <tiwai@suse.de> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com> Cc: Usama Arif <usamaarif642@gmail.com> Cc: Vishal Moola (Oracle) <vishal.moola@gmail.com> Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Wei Yang <richard.weiyang@gmail.com> Cc: Will Deacon <will@kernel.org> Cc: Yang Shi <yang@os.amperecomputing.com> Cc: Zach O'Keefe <zokeefe@google.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |