mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-09-18 23:09:29 +02:00
Gregory Price
2a6f2aef11
There is no atomic mechanism to offline and remove an entire
multi-block DAX kmem device. This is presently done in two steps:
1. offline all
2. remove all
This creates a race condition where another entity operates directly on
the memory blocks and can cause hot-unplug to fail / unbind to deadlock.
Add a new 'state' sysfs attribute that enables an atomic whole-device
hotplug operation across its entire memory region.
daxX.Y/state mirrors the per-block memoryX/state ABI:
- [offline, online, online_kernel, online_movable]
- "unplugged" - is added specifically for dax0.0/state
The valid writable states include:
- "unplugged": memory blocks are not present
- "online": memory is online, zone chosen by the kernel
- "online_kernel": memory is online in ZONE_NORMAL
- "online_movable": memory is online in ZONE_MOVABLE
Valid transitions:
- unplugged -> online[_kernel|_movable]
- online[_kernel|_movable] -> unplugged
- offline -> unplugged
A device can only be onlined from "unplugged", so it must be returned
there before being onlined into a different state.
For backwards compatibility the memory blocks are always created at probe
- existing tools expect them to be present after kmem binds.
"offline" is therefore a reportable state but is not writable: it only
arises from the legacy auto_online_blocks=offline policy. Onlining such a
device through this attribute requires unplugging it first in an effort to
get drivers creating DAX devices to set a default.
Unplug is atomic across the whole device: dax_kmem_do_hotremove() collects
every added range and offlines/removes them in one operation. Either the
operation succeeds or is entirely rolled back.
Unbind Note:
An offline dax device memory is removed on unbind as before.
If online at unbind, the resources are leaked (as before), but now
we prevent deadlock if a memory region is impossible to hotremove.
Link: https://lore.kernel.org/20260712154505.3564379-10-gourry@gourry.net
Signed-off-by: Gregory Price <gourry@gourry.net>
Suggested-by: Hannes Reinecke <hare@suse.de>
Suggested-by: David Hildenbrand <david@kernel.org>
Reviewed-by: Dan Williams <djbw@kernel.org>
Cc: Alison Schofield <alison.schofield@intel.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Pankaj Gupta <pankaj.gupta@amd.com>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
178 lines
6.6 KiB
Plaintext
178 lines
6.6 KiB
Plaintext
What: /sys/bus/dax/devices/daxX.Y/align
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RW) Provides a way to specify an alignment for a dax device.
|
|
Values allowed are constrained by the physical address ranges
|
|
that back the dax device, and also by arch requirements.
|
|
|
|
What: /sys/bus/dax/devices/daxX.Y/mapping
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(WO) Provides a way to allocate a mapping range under a dax
|
|
device. Specified in the format <start>-<end>.
|
|
|
|
What: /sys/bus/dax/devices/daxX.Y/mapping[0..N]/start
|
|
What: /sys/bus/dax/devices/daxX.Y/mapping[0..N]/end
|
|
What: /sys/bus/dax/devices/daxX.Y/mapping[0..N]/page_offset
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) A dax device may have multiple constituent discontiguous
|
|
address ranges. These are represented by the different
|
|
'mappingX' subdirectories. The 'start' attribute indicates the
|
|
start physical address for the given range. The 'end' attribute
|
|
indicates the end physical address for the given range. The
|
|
'page_offset' attribute indicates the offset of the current
|
|
range in the dax device.
|
|
|
|
What: /sys/bus/dax/devices/daxX.Y/resource
|
|
Date: June, 2019
|
|
KernelVersion: v5.3
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) The resource attribute indicates the starting physical
|
|
address of a dax device. In case of a device with multiple
|
|
constituent ranges, it indicates the starting address of the
|
|
first range.
|
|
|
|
What: /sys/bus/dax/devices/daxX.Y/size
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RW) The size attribute indicates the total size of a dax
|
|
device. For creating subdivided dax devices, or for resizing
|
|
an existing device, the new size can be written to this as
|
|
part of the reconfiguration process.
|
|
|
|
What: /sys/bus/dax/devices/daxX.Y/numa_node
|
|
Date: November, 2019
|
|
KernelVersion: v5.5
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) If NUMA is enabled and the platform has affinitized the
|
|
backing device for this dax device, emit the CPU node
|
|
affinity for this device.
|
|
|
|
What: /sys/bus/dax/devices/daxX.Y/target_node
|
|
Date: February, 2019
|
|
KernelVersion: v5.1
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) The target-node attribute is the Linux numa-node that a
|
|
device-dax instance may create when it is online. Prior to
|
|
being online the device's 'numa_node' property reflects the
|
|
closest online cpu node which is the typical expectation of a
|
|
device 'numa_node'. Once it is online it becomes its own
|
|
distinct numa node.
|
|
|
|
What: $(readlink -f /sys/bus/dax/devices/daxX.Y)/../dax_region/available_size
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) The available_size attribute tracks available dax region
|
|
capacity. This only applies to volatile hmem devices, not pmem
|
|
devices, since pmem devices are defined by nvdimm namespace
|
|
boundaries.
|
|
|
|
What: $(readlink -f /sys/bus/dax/devices/daxX.Y)/../dax_region/size
|
|
Date: July, 2017
|
|
KernelVersion: v5.1
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) The size attribute indicates the size of a given dax region
|
|
in bytes.
|
|
|
|
What: $(readlink -f /sys/bus/dax/devices/daxX.Y)/../dax_region/align
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) The align attribute indicates alignment of the dax region.
|
|
Changes on align may not always be valid, when say certain
|
|
mappings were created with 2M and then we switch to 1G. This
|
|
validates all ranges against the new value being attempted, post
|
|
resizing.
|
|
|
|
What: $(readlink -f /sys/bus/dax/devices/daxX.Y)/../dax_region/seed
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) The seed device is a concept for dynamic dax regions to be
|
|
able to split the region amongst multiple sub-instances. The
|
|
seed device, similar to libnvdimm seed devices, is a device
|
|
that starts with zero capacity allocated and unbound to a
|
|
driver.
|
|
|
|
What: $(readlink -f /sys/bus/dax/devices/daxX.Y)/../dax_region/create
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RW) The create interface to the dax region provides a way to
|
|
create a new unconfigured dax device under the given region, which
|
|
can then be configured (with a size etc.) and then probed.
|
|
|
|
What: $(readlink -f /sys/bus/dax/devices/daxX.Y)/../dax_region/delete
|
|
Date: October, 2020
|
|
KernelVersion: v5.10
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(WO) The delete interface for a dax region provides for deletion
|
|
of any 0-sized and idle dax devices.
|
|
|
|
What: $(readlink -f /sys/bus/dax/devices/daxX.Y)/../dax_region/id
|
|
Date: July, 2017
|
|
KernelVersion: v5.1
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RO) The id attribute indicates the region id of a dax region.
|
|
|
|
What: /sys/bus/dax/devices/daxX.Y/memmap_on_memory
|
|
Date: January, 2024
|
|
KernelVersion: v6.8
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RW) Control the memmap_on_memory setting if the dax device
|
|
were to be hotplugged as system memory. This determines whether
|
|
the 'altmap' for the hotplugged memory will be placed on the
|
|
device being hotplugged (memmap_on_memory=1) or if it will be
|
|
placed on regular memory (memmap_on_memory=0). This attribute
|
|
must be set before the device is handed over to the 'kmem'
|
|
driver (i.e. hotplugged into system-ram). Additionally, this
|
|
depends on CONFIG_MHP_MEMMAP_ON_MEMORY, and a globally enabled
|
|
memmap_on_memory parameter for memory_hotplug. This is
|
|
typically set on the kernel command line -
|
|
memory_hotplug.memmap_on_memory set to 'true' or 'force'."
|
|
|
|
What: /sys/bus/dax/devices/daxX.Y/state
|
|
Contact: nvdimm@lists.linux.dev
|
|
Description:
|
|
(RW) Controls the state of the memory region.
|
|
Applies to all memory blocks associated with the device.
|
|
Only applies to dax_kmem devices.
|
|
|
|
Reading returns the current state; the writable states mirror
|
|
the per-block /sys/devices/system/memory/memoryX/state ABI::
|
|
|
|
"unplugged": memory blocks are not present
|
|
"online": memory is online, zone chosen by the kernel
|
|
"online_kernel": memory is online in ZONE_NORMAL
|
|
"online_movable": memory is online in ZONE_MOVABLE
|
|
|
|
"offline" (memory blocks are present but offline) may also be
|
|
reported - this happens when the device is bound while the
|
|
auto_online_blocks policy is "offline". It cannot be written,
|
|
as it's not useful and creates device destruction races.
|
|
|
|
A device can only be onlined from the "unplugged" state, so a
|
|
device must be returned to "unplugged" before it can be onlined
|
|
into a different state.
|