Sandro/linux
1
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-09-18 22:09:30 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux

Browse Source 
Pull block updates from Jens Axboe:

 - NVMe updates via Keith:
     - Enable Clang context analysis for the nvme host driver, adding
       context annotations across core, fabrics, rdma, tcp and pci
     - nvmet reservation state exposed through a new namespace-level
       debugfs directory, plus ABI documentation for the host sysfs and
       target configfs interfaces
     - nvme-tcp host memory disclosure fixes on the read path: reject a
       read that transferred too few bytes, don't accept C2HData based
       on blk_rq_payload_bytes() alone, and fix the R2T case for a read
       command
     - Parallelize nvme-rdma I/O queue allocation and startup (Surabhi)
     - Apple nvme fixes and quirks: page aligned admin queue buffers,
       destroy the admin queue on removal, and various DMA/NVMMU
       correctness fixes
     - A large pile of nvmet and host fixes for out-of-bounds reads,
       refcount/resource leaks, and NULL derefs across auth, zns,
       passthru, pci-epf, rdma and configfs
     - Various other fixes and cleanups

 - MD updates via Yu Kuai:
     - llbitmap reshape support, the large series wiring exact bitmap
       mapping and reshape lifecycle through raid5 and raid10, growing
       the page cache in place, and remapping checkpointed bits as
       reshape progresses
     - raid5 fixes for lockless max_nr_stripes and recovery_offset
       accesses, a reshape deadlock with more failed devices than max
       degraded, and bitmap batch counter consistency
     - Atomic write handling for raid1/raid10, and removal of the
       REQ_NOWAIT support from raid1/10/456
     - raid5-ppl use-after-free fix in ppl_do_flush()
     - A batch of smaller fixes across md core and the bitmap code

 - s390/dasd ESE full-track write support and the surrounding
   infrastructure, plus enabling CONTEXT_ANALYSIS for s390/block

 - RWF_DONTCACHE support for block devices, built on new task-context
   bio completion infrastructure, and wiring it up for the iomap and
   buffer dropbehind writeback paths

 - Async io_uring zone reset all, plus zone management command cleanups
   allowing REQ_NOWAIT and tightening conventional zone rejection

 - Block integrity refactoring: lift BIP_CHECK_FLAGS to the shared
   header, handle nogenerate/noverify properly in fs-integrity, and drop
   the blk-integrity.h include from bdev.c

 - Split out a new blk_plug.h header

 - ublk improvements: add UBLK_F_IO_DESC_SIZE, split request validation
   from io_desc init, reject non-power-of-2 zone sizes in SET_PARAMS,
   and a series of hardening fixes around map/unmap and auto buf reg

 - null_blk cleanups and configfs serialization fixes

 - nbd queue freeze removal on the setup paths, and a new
   pre_defined_connections module parameter for pre-created devices

 - blk-cgroup fixes for the race between policy activation and blkg
   destruction, and accounting per-cpu stats over possible CPUs across
   blk-stat, iolatency, iocost and kyber

 - Various dio fixes: leak on metadata mapping error, validate user
   space vectors during extraction, and set dma_alignment from the
   backing file for loop and zloop direct I/O

 - bio cleanups

 - Various other fixes and cleanups all over

* tag 'for-7.3/block-20260819' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (241 commits)
  nbd: add pre_defined_connections module parameter for pre-created devices
  nbd: remove queue freeze for newly created nbd from netlink path
  nbd: factor out a nbd_genl_foreach_sock
  nbd: skip queue freeze when setting size at device startup
  nbd: remove queue freeze in nbd_add_socket
  nbd: clear queue limits on disconnect
  nbd: disallow NBD_SET_SOCK on an active device
  nbd: simplify find_fallback() by removing redundant logic
  blk-mq: add missing call to srcu_barrier() in blk_mq_free_tag_set()
  block: mtip32xx: synchronize ioctls with device removal
  ublk: avoid teardown retry loop on xarray allocation failure
  null_blk: fix UBSAN shift-out-of-bounds when zone_size is 0 or overflows
  block: don't include blk-integrity.h in bdev.c
  xfs: avoid double deferrals for RWF_DONTCACHE writes
  loop: Fix recently introduced lock inversion
  block: set QUEUE_FLAG_DYING unconditionally in blk_mark_disk_dead()
  swim3: Add missing MODULE_DESCRIPTION
  selftests: ublk: add SET_PARAMS validation test
  selftests: ublk: add helper for SET_PARAMS
  ublk: reject non-power-of-2 zone sizes in SET_PARAMS
  ...
This commit is contained in:
Linus Torvalds
2026-08-20 13:55:16 -07:00
parent f5437ff729 326d49039c
commit 55ab7e1422
136 changed files with 6022 additions and 1598 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/ABI/stable/configfs-nvmet
+352
View File
@@ -0,0 +1,352 @@
What: /config/nvmet/ports/N/addr_adrfam
What: /config/nvmet/ports/N/addr_portid
What: /config/nvmet/ports/N/addr_traddr
What: /config/nvmet/ports/N/addr_trsvcid
What: /config/nvmet/ports/N/addr_trtype
What: /config/nvmet/ports/N/addr_treq
Date: June 2016
KernelVersion: 4.8
Contact: Christoph Hellwig <hch@lst.de>
Description:
Address attributes for an NVMe-oF target port.
addr_adrfam: Shows or sets the address family. Accepted
values: "pcie", "ipv4", "ipv6", "ib", "fc", "pci", "loop".
addr_portid: Shows or sets the port identifier (u16).
addr_traddr: Shows or sets the transport address string.
addr_trsvcid: Shows or sets the transport service identifier.
addr_trtype: Shows or sets the transport type. Accepted
values: "rdma", "fc", "tcp", "pci", "loop". Also
initializes default TSAS values.
addr_treq: Shows or sets the transport security requirements.
Accepted values: "not specified", "required",
"not required". For TCP with TLS1.3, "not specified" is
rejected.
All attributes require the port to be disabled before
modification.
What: /config/nvmet/ports/N/referrals/NAME/addr_adrfam
What: /config/nvmet/ports/N/referrals/NAME/addr_portid
What: /config/nvmet/ports/N/referrals/NAME/addr_traddr
What: /config/nvmet/ports/N/referrals/NAME/addr_trsvcid
What: /config/nvmet/ports/N/referrals/NAME/addr_trtype
What: /config/nvmet/ports/N/referrals/NAME/addr_treq
What: /config/nvmet/ports/N/referrals/NAME/enable
Date: June 2016
KernelVersion: 4.8
Contact: Christoph Hellwig <hch@lst.de>
Description:
Address attributes and enable control for a referral entry
under a port. The addr_* attributes have the same semantics
as the corresponding port-level attributes. The enable
attribute shows or sets whether this referral is enabled
(boolean).
What: /config/nvmet/ports/N/param_inline_data_size
Date: June 2018
KernelVersion: 4.19
Contact: Steve Wise <swise@opengridcomputing.com>
Description:
Shows or sets the inline data size for this port. Default
is -1 which lets the transport choose. The port must be
disabled before modification.
What: /config/nvmet/ports/N/ana_groups/ID/ana_state
Date: June 2018
KernelVersion: 4.19
Contact: Christoph Hellwig <hch@lst.de>
Description:
Shows or sets the ANA (Asymmetric Namespace Access) state
for this group on this port. Accepted values: "optimized",
"non-optimized", "inaccessible", "persistent-loss",
"change". Changes trigger an ANA change event.
What: /config/nvmet/ports/N/param_pi_enable
Date: May 2020
KernelVersion: 5.8
Contact: Israel Rukshin <israelr@mellanox.com>
Description:
Shows or sets whether protection information (PI) is
enabled/supported for this port. Accepts boolean value.
Only available when CONFIG_BLK_DEV_INTEGRITY is enabled.
The port must be disabled before modification.
What: /config/nvmet/ports/N/addr_tsas
Date: August 2023
KernelVersion: 6.7
Contact: Hannes Reinecke <hare@suse.de>
Description:
Shows or sets the transport-specific address subtype. For
TCP transport, accepted values: "none", "tls1.3" (requires
CONFIG_NVME_TARGET_TCP_TLS). For RDMA transport, shows the
QP type: "connected" or "datagram". The port must be
disabled before modification.
What: /config/nvmet/ports/N/param_max_queue_size
Date: January 2024
KernelVersion: 6.9
Contact: Max Gurtovoy <mgurtovoy@nvidia.com>
Description:
Shows or sets the maximum queue size for this port. Default
is -1 which lets the transport choose. The port must be
disabled before modification.
What: /config/nvmet/ports/N/param_mdts
Date: April 2026
KernelVersion: 7.1
Contact: Aurelien Aptel <aaptel@nvidia.com>
Description:
Shows or sets the maximum data transfer size for this port.
Default is -1 which lets the transport choose. The port
must be disabled before modification.
What: /config/nvmet/subsystems/NAME/attr_allow_any_host
Date: June 2016
KernelVersion: 4.8
Contact: Christoph Hellwig <hch@lst.de>
Description:
Shows or sets whether any host is allowed to connect.
Accepts boolean value. Cannot be set to 1 if explicit
hosts are linked in the allowed_hosts/ directory.
What: /config/nvmet/subsystems/NAME/namespaces/NSID/device_path
What: /config/nvmet/subsystems/NAME/namespaces/NSID/device_nguid
What: /config/nvmet/subsystems/NAME/namespaces/NSID/enable
Date: June 2016
KernelVersion: 4.8
Contact: Christoph Hellwig <hch@lst.de>
Description:
Namespace attributes added with the initial NVMe target.
device_path: Shows or sets the backend block device path.
The namespace must be disabled before modification.
device_nguid: Shows or sets the NGUID (128-bit identifier).
Accepts 32 hex digits with optional "-" or ":" separators.
The namespace must be disabled before modification.
enable: Shows or sets whether this namespace is enabled
(boolean).
What: /config/nvmet/subsystems/NAME/namespaces/NSID/device_uuid
Date: June 2017
KernelVersion: 4.13
Contact: Johannes Thumshirn <jthumshirn@suse.de>
Description:
Shows or sets the UUID for this namespace. The namespace
must be disabled before modification.
What: /config/nvmet/subsystems/NAME/attr_version
What: /config/nvmet/subsystems/NAME/attr_serial
Date: July 2017
KernelVersion: 4.13
Contact: Johannes Thumshirn <jthumshirn@suse.de>
Description:
attr_version: Shows or sets the NVMe version reported by
this subsystem. Format: "major.minor" or
"major.minor.tertiary". Cannot be changed after the
subsystem has been discovered.
attr_serial: Shows or sets the serial number. Must be a
1-20 byte ASCII string (characters 0x20-0x7e). Cannot be
changed after the subsystem has been discovered.
What: /config/nvmet/subsystems/NAME/namespaces/NSID/ana_grpid
Date: June 2018
KernelVersion: 4.19
Contact: Christoph Hellwig <hch@lst.de>
Description:
Shows or sets the ANA (Asymmetric Namespace Access) Group
ID for this namespace. Must be between 1 and 128. Changing
triggers an ANA event notification.
What: /config/nvmet/subsystems/NAME/namespaces/NSID/buffered_io
Date: June 2018
KernelVersion: 4.19
Contact: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Description:
Shows or sets whether buffered I/O is used for this
namespace. Accepts boolean value. The namespace must be
disabled before modification.
What: /config/nvmet/subsystems/NAME/namespaces/NSID/p2pmem
Date: October 2018
KernelVersion: 4.20
Contact: Logan Gunthorpe <logang@deltatee.com>
Description:
Shows or sets the P2P DMA memory device for this namespace.
Accepts a PCI device BDF, "auto", or "none". The namespace
must be disabled before modification. Only available when
CONFIG_PCI_P2PDMA is enabled.
What: /config/nvmet/subsystems/NAME/attr_cntlid_min
What: /config/nvmet/subsystems/NAME/attr_cntlid_max
Date: January 2020
KernelVersion: 5.7
Contact: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Description:
attr_cntlid_min: Shows or sets the minimum controller ID
(u16). Must be nonzero and not greater than attr_cntlid_max.
attr_cntlid_max: Shows or sets the maximum controller ID
(u16). Must be nonzero and not less than attr_cntlid_min.
What: /config/nvmet/subsystems/NAME/attr_model
Date: January 2020
KernelVersion: 5.7
Contact: Mark Ruijter <MRuijter@onestopsystems.com>
Description:
Shows or sets the model number for this subsystem. Must
be a 1-40 byte ASCII string (characters 0x20-0x7e).
Cannot be changed after the subsystem has been discovered.
What: /config/nvmet/subsystems/NAME/attr_pi_enable
Date: May 2020
KernelVersion: 5.8
Contact: Israel Rukshin <israelr@mellanox.com>
Description:
Shows or sets whether protection information (PI) is
enabled/supported for this subsystem. Accepts boolean
value. Only available when CONFIG_BLK_DEV_INTEGRITY is
enabled.
What: /config/nvmet/subsystems/NAME/namespaces/NSID/revalidate_size
Date: May 2020
KernelVersion: 5.8
Contact: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Description:
Write-only. Writing 1 triggers namespace size revalidation.
If the size has changed, a namespace changed AEN is sent.
The namespace must be enabled.
What: /config/nvmet/subsystems/NAME/passthru/device_path
What: /config/nvmet/subsystems/NAME/passthru/enable
Date: July 2020
KernelVersion: 5.9
Contact: Logan Gunthorpe <logang@deltatee.com>
Description:
Passthrough mode attributes.
device_path: Shows or sets the NVMe controller character
device path (e.g., /dev/nvme0). Cannot be changed while
the passthrough controller is active.
enable: Shows or sets whether passthrough mode is enabled
(boolean).
Only available when CONFIG_NVME_TARGET_PASSTHRU is enabled.
What: /config/nvmet/subsystems/NAME/passthru/admin_timeout
What: /config/nvmet/subsystems/NAME/passthru/io_timeout
Date: November 2020
KernelVersion: 5.11
Contact: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Description:
admin_timeout: Shows or sets the admin command timeout for
passthrough mode, in jiffies.
io_timeout: Shows or sets the I/O command timeout for
passthrough mode, in jiffies.
Only available when CONFIG_NVME_TARGET_PASSTHRU is enabled.
What: /config/nvmet/subsystems/NAME/passthru/clear_ids
Date: June 2022
KernelVersion: 5.19
Contact: Alan Adamson <alan.adamson@oracle.com>
Description:
Shows or sets whether to clear identify data IDs in
passthrough mode. Only available when
CONFIG_NVME_TARGET_PASSTHRU is enabled.
What: /config/nvmet/subsystems/NAME/attr_qid_max
Date: August 2022
KernelVersion: 6.1
Contact: Daniel Wagner <dwagner@suse.de>
Description:
Shows or sets the maximum queue ID (number of I/O queues,
u16). Must be between 1 and 128. Changing this value
forces reconnection of all connected controllers.
What: /config/nvmet/subsystems/NAME/attr_ieee_oui
Date: November 2022
KernelVersion: 6.2
Contact: Aleksandr Miloserdov <a.miloserdov@yadro.com>
Description:
Shows or sets the IEEE OUI for this subsystem. Displayed
in "0x%06x" format. Must be a 24-bit value. Cannot be
changed after the subsystem has been discovered.
What: /config/nvmet/subsystems/NAME/attr_firmware
Date: November 2022
KernelVersion: 6.2
Contact: Aleksandr Miloserdov <a.miloserdov@yadro.com>
Description:
Shows or sets the firmware revision string for this
subsystem. Must be a 1-8 byte ASCII string (characters
0x20-0x7e). Cannot be changed after the subsystem has
been discovered.
What: /config/nvmet/subsystems/NAME/namespaces/NSID/resv_enable
Date: November 2024
KernelVersion: 6.13
Contact: Guixin Liu <kanie@linux.alibaba.com>
Description:
Shows or sets whether persistent reservation support is
enabled for this namespace. Accepts boolean value. The
namespace must be disabled before modification.
What: /config/nvmet/subsystems/NAME/attr_vendor_id
What: /config/nvmet/subsystems/NAME/attr_subsys_vendor_id
Date: January 2025
KernelVersion: 6.14
Contact: Damien Le Moal <dlemoal@kernel.org>
Description:
attr_vendor_id: Shows or sets the PCI vendor ID reported
by this subsystem. Displayed in "0x%x" format.
attr_subsys_vendor_id: Shows or sets the PCI subsystem
vendor ID. Displayed in "0x%x" format.
What: /config/nvmet/hosts/HOSTNQN/dhchap_key
What: /config/nvmet/hosts/HOSTNQN/dhchap_ctrl_key
What: /config/nvmet/hosts/HOSTNQN/dhchap_hash
What: /config/nvmet/hosts/HOSTNQN/dhchap_dhgroup
Date: June 2022
KernelVersion: 6.0
Contact: Hannes Reinecke <hare@suse.de>
Description:
DH-HMAC-CHAP authentication attributes.
dhchap_key: Shows or sets the host secret key. Accepts a
key string in "DHHC-1:" format.
dhchap_ctrl_key: Shows or sets the controller secret key
for bidirectional authentication. Same format as dhchap_key.
dhchap_hash: Shows or sets the HMAC hash algorithm.
Accepted values: "hmac(sha256)", "hmac(sha384)",
"hmac(sha512)".
dhchap_dhgroup: Shows or sets the Diffie-Hellman group for
DH-HMAC-CHAP key exchange. Accepted values: "null",
"ffdhe2048", "ffdhe3072", "ffdhe4096", "ffdhe6144".
Non-null groups require the corresponding KPP crypto
algorithm to be available.
Only available when CONFIG_NVME_TARGET_AUTH is enabled.
What: /config/nvmet/discovery_nqn
Date: April 2024
KernelVersion: 6.9
Contact: Hannes Reinecke <hare@kernel.org>
Description:
Shows or sets the NQN of the discovery subsystem. The
value must be unique and not duplicate any existing
subsystem name.
Documentation/ABI/stable/sysfs-nvme
+453
View File
@@ -0,0 +1,453 @@
What: /sys/class/nvme/nvmeX/model
What: /sys/class/nvme/nvmeX/serial
What: /sys/class/nvme/nvmeX/firmware_rev
Date: January 2016
KernelVersion: 4.5
Contact: Keith Busch <kbusch@kernel.org>
Description:
Shows the model, serial number, or firmware revision string
of the NVMe controller, as reported in the Identify
Controller data structure.
What: /sys/class/nvme/nvmeX/cntlid
Date: February 2016
KernelVersion: 4.6
Contact: Ming Lin <ming.l@ssi.samsung.com>
Description:
Shows the controller identifier assigned by the NVMe
subsystem.
What: /sys/class/nvme/nvmeX/cntrltype
What: /sys/class/nvme/nvmeX/dctype
Date: February 2022
KernelVersion: 5.18
Contact: Martin Belanger <martin.belanger@dell.com>
Description:
cntrltype: Shows the controller type. Possible values: "io",
"discovery", "admin", "reserved".
dctype: Shows the discovery controller type. Possible values:
"none", "ddc", "cdc", "reserved".
What: /sys/class/nvme/nvmeX/reset_controller
Date: November 2015
KernelVersion: 4.5
Contact: Christoph Hellwig <hch@lst.de>
Description:
Write-only. Writing any value triggers a synchronous
controller reset.
What: /sys/class/nvme/nvmeX/rescan_controller
Date: April 2016
KernelVersion: 4.7
Contact: Keith Busch <kbusch@kernel.org>
Description:
Write-only. Writing any value triggers a namespace rescan
on this controller.
What: /sys/class/nvme/nvmeX/transport
What: /sys/class/nvme/nvmeX/subsysnqn
What: /sys/class/nvme/nvmeX/address
What: /sys/class/nvme/nvmeX/delete_controller
What: /sys/class/nvme/nvmeX/reconnect_delay
What: /sys/class/nvme/nvmeX/ctrl_loss_tmo
Date: June 2016
KernelVersion: 4.8
Contact: Ming Lin <ming.l@ssi.samsung.com>
Description:
Fabrics controller attributes added with NVMe-oF support.
transport: Shows the transport type string. Possible values:
"pcie", "tcp", "rdma", "fc", "loop".
subsysnqn: Shows the NVMe Qualified Name (NQN) of the
subsystem this controller belongs to.
address: Shows the transport-specific address string. Only
available for fabrics controllers.
delete_controller: Write-only. Triggers deletion of this
fabrics controller.
reconnect_delay: Shows or sets the reconnect delay in
seconds. Reading returns the delay value, or "off" if
disabled.
ctrl_loss_tmo: Shows or sets the controller loss timeout in
seconds. Reading returns the timeout value, or "off" if
infinite reconnects are allowed. Writing a negative value
disables the timeout.
What: /sys/class/nvme/nvmeX/hostnqn
What: /sys/class/nvme/nvmeX/hostid
Date: February 2020
KernelVersion: 5.7
Contact: Sagi Grimberg <sagi@grimberg.me>
Description:
hostnqn: Shows the host NQN used by this fabrics controller.
hostid: Shows the host identifier (UUID format) used by this
fabrics controller.
Only available for fabrics controllers.
What: /sys/class/nvme/nvmeX/fast_io_fail_tmo
Date: November 2020
KernelVersion: 5.11
Contact: Victor Gladkov <Victor.Gladkov@kioxia.com>
Description:
Shows or sets the fast I/O fail timeout in seconds. Reading
returns the timeout value, or "off" if disabled. Writing a
negative value disables the fast I/O fail. Only available
for fabrics controllers.
What: /sys/class/nvme/nvmeX/kato
Date: April 2021
KernelVersion: 5.13
Contact: Hannes Reinecke <hare@suse.de>
Description:
Shows the Keep Alive Timeout value in milliseconds for
this controller.
What: /sys/class/nvme/nvmeX/cmb
Date: October 2016
KernelVersion: 4.9
Contact: Stephen Bates <sbates@raithlin.com>
Description:
Shows the Controller Memory Buffer (CMB) register values
in format "cmbloc : 0x%08x\ncmbsz : 0x%08x\n". Only
visible when the controller has a CMB (cmbsz != 0).
PCI transport only.
What: /sys/class/nvme/nvmeX/cmbloc
What: /sys/class/nvme/nvmeX/cmbsz
What: /sys/class/nvme/nvmeX/hmb
Date: July 2021
KernelVersion: 5.15
Contact: Keith Busch <kbusch@kernel.org>
Description:
cmbloc: Shows the CMBLOC register value.
cmbsz: Shows the CMBSZ register value.
cmbloc and cmbsz are only visible when the controller has
a CMB. PCI transport only.
hmb: Shows or sets whether the Host Memory Buffer (HMB) is
enabled. Reading returns 1 (enabled) or 0 (disabled).
Writing 1 enables HMB; writing 0 disables it. Only
visible when the controller supports HMB (hmpre != 0).
PCI transport only.
What: /sys/class/nvme/nvmeX/state
Date: November 2016
KernelVersion: 4.11
Contact: Sagi Grimberg <sagi@grimberg.me>
Description:
Shows the current state of the controller. Possible values:
"new", "live", "resetting", "connecting", "deleting",
"deleting (no IO)", "dead".
What: /sys/class/nvme/nvmeX/numa_node
Date: November 2018
KernelVersion: 5.0
Contact: Hannes Reinecke <hare@suse.de>
Description:
Shows the NUMA node the controller is attached to.
What: /sys/class/nvme/nvmeX/queue_count
What: /sys/class/nvme/nvmeX/sqsize
Date: September 2019
KernelVersion: 5.4
Contact: James Smart <jsmart2021@gmail.com>
Description:
queue_count: Shows the total number of queues (admin + I/O)
for this controller.
sqsize: Shows the submission queue size for this controller.
What: /sys/class/nvme/nvmeX/dhchap_secret
What: /sys/class/nvme/nvmeX/dhchap_ctrl_secret
Date: June 2022
KernelVersion: 6.0
Contact: Hannes Reinecke <hare@suse.de>
Description:
dhchap_secret: Shows or sets the host DH-HMAC-CHAP secret
for this controller. Reading returns "none" if not set.
Writing must use the "DHHC-1:" key format and triggers
re-authentication.
dhchap_ctrl_secret: Shows or sets the controller
DH-HMAC-CHAP secret for bidirectional authentication.
Same format as dhchap_secret.
Only available when CONFIG_NVME_HOST_AUTH is enabled and
for fabrics controllers.
What: /sys/class/nvme/nvmeX/tls_key
Date: August 2023
KernelVersion: 6.7
Contact: Hannes Reinecke <hare@suse.de>
Description:
Shows the serial of the currently active TLS PSK as hex.
Returns empty if no TLS key is active. Only available for
TCP controllers with TLS or secure concatenation enabled
(CONFIG_NVME_TCP_TLS).
What: /sys/class/nvme/nvmeX/tls_configured_key
Date: July 2024
KernelVersion: 6.12
Contact: Hannes Reinecke <hare@kernel.org>
Description:
Shows the serial of the configured TLS key. Writing 0
triggers a PSK reauthentication (REPLACETLSPSK) with
the target. After reauthentication the returned serial
will be the new key. Only available for TCP controllers
with secure concatenation enabled (CONFIG_NVME_TCP_TLS).
What: /sys/class/nvme/nvmeX/tls_keyring
Date: July 2024
KernelVersion: 6.12
Contact: Hannes Reinecke <hare@kernel.org>
Description:
Shows the TLS keyring description. Only available for TCP
controllers with a keyring configured (CONFIG_NVME_TCP_TLS).
What: /sys/class/nvme/nvmeX/tls_mode
Date: April 2026
KernelVersion: 7.1
Contact: Daniel Wagner <wagi@kernel.org>
Description:
Shows the TLS mode: "tls" for direct TLS or "concat" for
secure concatenation. Only available for TCP controllers
with TLS or secure concatenation enabled
(CONFIG_NVME_TCP_TLS).
What: /sys/class/nvme/nvmeX/passthru_err_log_enabled
Date: January 2024
KernelVersion: 6.8
Contact: Alan Adamson <alan.adamson@oracle.com>
Description:
Shows or sets whether admin passthrough error logging is
enabled for this controller. Reading returns "on" or "off".
Writing accepts a boolean value.
What: /sys/class/nvme/nvmeX/quirks
Date: November 2025
KernelVersion: 7.0
Contact: Maurizio Lombardi <mlombard@redhat.com>
Description:
Shows the active quirk names for this controller, one per
line. Shows "none" if no quirks are active.
What: /sys/class/nvme/nvmeX/admin_timeout
What: /sys/class/nvme/nvmeX/io_timeout
Date: May 2026
KernelVersion: 7.2
Contact: Maurizio Lombardi <mlombard@redhat.com>
Description:
admin_timeout: Shows or sets the admin command timeout in
milliseconds.
io_timeout: Shows or sets the I/O command timeout in
milliseconds. Changes are propagated to all namespace
request queues.
The value must be nonzero. Only writable after the
controller has been started at least once.
What: /sys/block/nvmeXnY/uuid
What: /sys/block/nvmeXnY/eui
What: /sys/block/nvmeXnY/nsid
Date: December 2015
KernelVersion: 4.5
Contact: Keith Busch <kbusch@kernel.org>
Description:
Namespace identification attributes.
uuid: Shows the UUID for this namespace. Falls back to
showing the NGUID for backward compatibility. Hidden if
both are all zeros.
eui: Shows the IEEE Extended Unique Identifier (EUI-64).
Hidden if all zeros.
nsid: Shows the namespace identifier (NSID).
What: /sys/block/nvmeXnY/wwid
Date: February 2016
KernelVersion: 4.6
Contact: Keith Busch <kbusch@kernel.org>
Description:
Shows the World Wide Identifier for this namespace. The
format depends on available identifiers (in priority
order): "uuid.{UUID}", "eui.{NGUID}", "eui.{EUI64}", or
"nvme.{VID}-{SERIAL}-{MODEL}-{NSID}".
What: /sys/block/nvmeXnY/nguid
Date: June 2017
KernelVersion: 4.13
Contact: Johannes Thumshirn <jthumshirn@suse.de>
Description:
Shows the Namespace Globally Unique Identifier (NGUID).
Hidden if the NGUID is all zeros.
What: /sys/block/nvmeXcYnZ/ana_grpid
What: /sys/block/nvmeXcYnZ/ana_state
Date: May 2018
KernelVersion: 4.19
Contact: Christoph Hellwig <hch@lst.de>
Description:
ana_grpid: Shows the ANA Group ID for this namespace
path device.
ana_state: Shows the ANA state. Possible values:
"optimized", "non-optimized", "inaccessible",
"persistent-loss", "change".
Only visible when the controller supports ANA.
Requires CONFIG_NVME_MULTIPATH.
What: /sys/block/nvmeXcYnZ/queue_depth
Date: June 2024
KernelVersion: 6.11
Contact: Thomas Song <tsong@purestorage.com>
Description:
Shows the current active I/O count on this path's
controller. Returns empty if iopolicy is not "queue-depth".
Requires CONFIG_NVME_MULTIPATH.
What: /sys/block/nvmeXcYnZ/numa_nodes
Date: January 2025
KernelVersion: 6.15
Contact: Nilay Shroff <nilay@linux.ibm.com>
Description:
Shows the NUMA node mask for which this path is the
currently selected path. Returns empty if iopolicy is not
"numa". Requires CONFIG_NVME_MULTIPATH.
What: /sys/block/nvmeXnY/delayed_removal_secs
Date: May 2025
KernelVersion: 6.16
Contact: Nilay Shroff <nilay@linux.ibm.com>
Description:
Shows or sets the delayed removal timeout in seconds for
the multipath head device. When nonzero, I/O is queued
instead of failed when all paths are gone, and head removal
is deferred. Only visible on multipath head devices.
Requires CONFIG_NVME_MULTIPATH.
What: /sys/block/nvmeXnY/csi
What: /sys/block/nvmeXnY/metadata_bytes
What: /sys/block/nvmeXnY/nuse
Date: December 2023
KernelVersion: 6.8
Contact: Daniel Wagner <dwagner@suse.de>
Description:
csi: Shows the Command Set Identifier for this namespace.
metadata_bytes: Shows the metadata size in bytes.
nuse: Shows the Namespace Utilization (NUSE) value. Reading
triggers an Identify Namespace command to refresh the
value (rate-limited to avoid excessive commands).
What: /sys/block/nvmeXnY/passthru_err_log_enabled
Date: January 2024
KernelVersion: 6.8
Contact: Alan Adamson <alan.adamson@oracle.com>
Description:
Shows or sets whether I/O passthrough error logging is
enabled for this namespace. Reading returns "on" or "off".
Writing accepts a boolean value.
What: /sys/class/nvme/nvmeX/diag/command_error_count
What: /sys/class/nvme/nvmeX/diag/reset_count
What: /sys/class/nvme/nvmeX/diag/reconnect_count
Date: May 2026
KernelVersion: 7.2
Contact: Nilay Shroff <nilay@linux.ibm.com>
Description:
Controller diagnostic counters.
command_error_count: Admin command error counter.
reset_count: Controller reset counter.
reconnect_count: Accumulated reconnect counter. Only
available for fabrics controllers.
All counters can be reset by writing a value.
What: /sys/block/nvmeXnY/diag/command_retries_count
What: /sys/block/nvmeXnY/diag/command_error_count
Date: May 2026
KernelVersion: 7.2
Contact: Nilay Shroff <nilay@linux.ibm.com>
Description:
Namespace diagnostic counters for non-multipath
configurations (when CONFIG_NVME_MULTIPATH is not
configured).
command_retries_count: I/O command retry counter.
command_error_count: I/O command error counter.
All counters can be reset by writing any value.
What: /sys/block/nvmeXcYnZ/diag/command_retries_count
What: /sys/block/nvmeXcYnZ/diag/command_error_count
What: /sys/block/nvmeXcYnZ/diag/multipath_failover_count
What: /sys/block/nvmeXnY/diag/io_requeue_no_usable_path_count
What: /sys/block/nvmeXnY/diag/io_fail_no_available_path_count
Date: May 2026
KernelVersion: 7.2
Contact: Nilay Shroff <nilay@linux.ibm.com>
Description:
Namespace diagnostic counters for multipath
configurations (when CONFIG_NVME_MULTIPATH is
configured).
command_retries_count: I/O command retry counter.
command_error_count: I/O command error counter.
multipath_failover_count: Multipath failover counter.
io_requeue_no_usable_path_count: Counter of I/Os
requeued because no usable path was available.
io_fail_no_available_path_count: Counter of I/Os
failed because no available path existed.
All counters can be reset by writing any value.
What: /sys/class/nvme-subsystem/nvme-subsysX/model
What: /sys/class/nvme-subsystem/nvme-subsysX/serial
What: /sys/class/nvme-subsystem/nvme-subsysX/firmware_rev
What: /sys/class/nvme-subsystem/nvme-subsysX/subsysnqn
Date: November 2017
KernelVersion: 4.15
Contact: Hannes Reinecke <hare@suse.de>
Description:
Shows the model, serial number, firmware revision, or NQN
of the NVMe subsystem.
What: /sys/class/nvme-subsystem/nvme-subsysX/iopolicy
Date: February 2019
KernelVersion: 5.1
Contact: Hannes Reinecke <hare@suse.de>
Description:
Shows or sets the multipath I/O path selection policy for
this subsystem. Accepted values: "numa", "round-robin",
"queue-depth". Changing the policy clears all current path
selections. Only available when CONFIG_NVME_MULTIPATH is
enabled.
What: /sys/class/nvme-subsystem/nvme-subsysX/subsystype
Date: September 2021
KernelVersion: 5.16
Contact: Hannes Reinecke <hare@suse.de>
Description:
Shows the subsystem type. Possible values: "discovery",
"nvm", "reserved".
Documentation/ABI/testing/sysfs-nvme
-13
View File
@@ -1,13 +0,0 @@
What: /sys/devices/virtual/nvme-fabrics/ctl/.../tls_configured_key
Date: November 2025
KernelVersion: 6.19
Contact: Linux NVMe mailing list <linux-nvme@lists.infradead.org>
Description:
The file is avaliable when using a secure concatanation
connection to a NVMe target. Reading the file will return
the serial of the currently negotiated key.
Writing 0 to the file will trigger a PSK reauthentication
(REPLACETLSPSK) with the target. After a reauthentication
the value returned by tls_configured_key will be the new
serial.
Documentation/admin-guide/blockdev/zoned_loop.rst
+8 -7
View File
@@ -30,11 +30,10 @@ indicates the position of the write pointer of the zone.
When resetting a sequential zone, its backing file size is truncated to zero.
Conversely, for a zone finish operation, the backing file is truncated to the
zone size. With this, the maximum capacity of a zloop zoned block device created
can be larger configured to be larger than the storage space available on the
backing file system. Of course, for such configuration, writing more data than
the storage space available on the backing file system will result in write
errors.
zone capacity. With this, a zloop zoned block device can be configured with a
larger capacity than the storage space available on the backing file system. Of
course, for such configuration, writing more data than the storage space
available on the backing file system will result in write errors.
The zoned loop block device driver implements a complete zone transition state
machine. That is, zones can be empty, implicitly opened, explicitly opened,
@@ -71,8 +70,10 @@ follows.
=================== =========================================================
id Device number (the X in /dev/zloopX).
Default: automatically assigned.
capacity_mb Device total capacity in MiB. This is always rounded up
to the nearest higher multiple of the zone size.
capacity_mb Device total capacity in MiB. A smaller last zone is not
supported, so a capacity value that is not a multiple of
the zone size is rounded down to the closest multiple of
the zone size.
Default: 16384 MiB (16 GiB).
zone_size_mb Device zone size in MiB. Default: 256 MiB.
zone_capacity_mb Device zone capacity (must always be equal to or lower
Documentation/block/biovecs.rst
+4 -4
View File
@@ -16,16 +16,16 @@ bv_len by the number of bytes completed in that biovec.
In the new scheme of things, everything that must be mutated in order to
partially complete a bio is segregated into struct bvec_iter: bi_sector,
bi_size and bi_idx have been moved there; and instead of modifying bv_offset
and bv_len, struct bvec_iter has bi_bvec_done, which represents the number of
and bv_len, struct bvec_iter has bi_offset, which represents the number of
bytes completed in the current bvec.
There are a bunch of new helper macros for hiding the gory details - in
particular, presenting the illusion of partially completed biovecs so that
normal code doesn't have to deal with bi_bvec_done.
normal code doesn't have to deal with bi_offset.
* Driver code should no longer refer to biovecs directly; we now have
bio_iovec() and bio_iter_iovec() macros that return literal struct biovecs,
constructed from the raw biovecs but taking into account bi_bvec_done and
constructed from the raw biovecs but taking into account bi_offset and
bi_size.
bio_for_each_segment() has been updated to take a bvec_iter argument
@@ -101,7 +101,7 @@ Other implications:
I.e. instead of using bio_iovec_idx() (or bio->bi_iovec[bio->bi_idx]), you
now use bio_iter_iovec(), which takes a bvec_iter and returns a
literal struct bio_vec - constructed on the fly from the raw biovec but
taking into account bi_bvec_done (and bi_size).
taking into account bi_offset (and bi_size).
* bi_vcnt can't be trusted or relied upon by driver code - i.e. anything that
doesn't actually own the bio. The reason is twofold: firstly, it's not
MAINTAINERS
+4
View File
@@ -19426,6 +19426,9 @@ L: linux-nvme@lists.infradead.org
S: Supported
W: http://git.infradead.org/nvme.git
T: git git://git.infradead.org/nvme.git
F: Documentation/ABI/stable/sysfs-nvme
F: Documentation/admin-guide/nvme-multipath.rst
F: Documentation/fault-injection/nvme-fault-injection.rst
F: Documentation/nvme/
F: drivers/nvme/common/
F: drivers/nvme/host/
@@ -19468,6 +19471,7 @@ L: linux-nvme@lists.infradead.org
S: Supported
W: http://git.infradead.org/nvme.git
T: git git://git.infradead.org/nvme.git
F: Documentation/ABI/stable/configfs-nvmet
F: drivers/nvme/target/
NVMEM FRAMEWORK
block/bdev.c
-1
View File
@@ -12,7 +12,6 @@
#include <linux/major.h>
#include <linux/device_cgroup.h>
#include <linux/blkdev.h>
#include <linux/blk-integrity.h>
#include <linux/backing-dev.h>
#include <linux/module.h>
#include <linux/blkpg.h>
block/bfq-cgroup.c
+2 -2
View File
@@ -610,7 +610,7 @@ struct bfq_group *bfq_bio_bfqg(struct bfq_data *bfqd, struct bio *bio)
struct bfq_group *bfqg;
while (blkg) {
if (!blkg->online) {
if (!data_race(blkg->online)) {
blkg = blkg->parent;
continue;
}
@@ -1168,7 +1168,7 @@ static u64 bfqg_prfill_stat_recursive(struct seq_file *sf,
struct blkg_policy_data *pd;
struct bfq_stat *stat;
if (!pos_blkg->online)
if (!data_race(pos_blkg->online))
continue;
pd = blkg_to_pd(pos_blkg, &blkcg_policy_bfq);
block/bio-integrity-auto.c
+2 -8
View File
@@ -44,12 +44,6 @@ static void bio_integrity_verify_fn(struct work_struct *work)
bio_endio(bio);
}
#define BIP_CHECK_FLAGS (BIP_CHECK_GUARD | BIP_CHECK_REFTAG | BIP_CHECK_APPTAG)
static bool bip_should_check(struct bio_integrity_payload *bip)
{
return bip->bip_flags & BIP_CHECK_FLAGS;
}
/**
* __bio_integrity_endio - Integrity I/O completion function
* @bio: Protected bio
@@ -66,7 +60,7 @@ bool __bio_integrity_endio(struct bio *bio)
container_of(bip, struct bio_integrity_data, bip);
if (bio_op(bio) == REQ_OP_READ && !bio->bi_status &&
bip_should_check(bip)) {
(bip->bip_flags & BIP_CHECK_FLAGS)) {
INIT_WORK(&bid->work, bio_integrity_verify_fn);
queue_work(kintegrityd_wq, &bid->work);
return false;
@@ -99,7 +93,7 @@ void bio_integrity_prep(struct bio *bio, unsigned int action)
bio_integrity_setup_default(bio);
/* Auto-generate integrity metadata if this is a write */
if (bio_data_dir(bio) == WRITE && bip_should_check(&bid->bip))
if (bio_data_dir(bio) == WRITE && (bid->bip.bip_flags & BIP_CHECK_FLAGS))
bio_integrity_generate(bio);
else
bid->saved_bio_iter = bio->bi_iter;
block/bio-integrity-fs.c
+5 -1
View File
@@ -46,7 +46,8 @@ void fs_bio_integrity_free(struct bio *bio)
void fs_bio_integrity_generate(struct bio *bio)
{
if (fs_bio_integrity_alloc(bio))
if (fs_bio_integrity_alloc(bio) &&
(bio_integrity(bio)->bip_flags & BIP_CHECK_FLAGS))
bio_integrity_generate(bio);
}
EXPORT_SYMBOL_GPL(fs_bio_integrity_generate);
@@ -60,6 +61,9 @@ int fs_bio_integrity_verify(struct bio *bio, sector_t sector, unsigned int size)
.bi_size = size,
};
if (!bip || !(bip->bip_flags & BIP_CHECK_FLAGS))
return 0;
/*
* Reinitialize bip->bip_iter.
*
block/bio.c
+194 -12
View File
@@ -229,7 +229,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,
bio->bi_iter.bi_sector = 0;
bio->bi_iter.bi_size = 0;
bio->bi_iter.bi_idx = 0;
bio->bi_iter.bi_bvec_done = 0;
bio->bi_iter.bi_offset = 0;
bio->bi_end_io = NULL;
bio->bi_private = NULL;
#ifdef CONFIG_BLK_CGROUP
@@ -860,6 +860,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
bio->bi_write_hint = bio_src->bi_write_hint;
bio->bi_write_stream = bio_src->bi_write_stream;
bio->bi_iter = bio_src->bi_iter;
bio->bi_io_vec = bio_src->bi_io_vec;
if (bio->bi_bdev) {
if (bio->bi_bdev == bio_src->bi_bdev &&
@@ -902,8 +903,6 @@ struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src,
bio_put(bio);
return NULL;
}
bio->bi_io_vec = bio_src->bi_io_vec;
return bio;
}
EXPORT_SYMBOL(bio_alloc_clone);
@@ -923,7 +922,7 @@ int bio_init_clone(struct block_device *bdev, struct bio *bio,
{
int ret;
bio_init(bio, bdev, bio_src->bi_io_vec, 0, bio_src->bi_opf);
bio_init(bio, bdev, NULL, 0, bio_src->bi_opf);
ret = __bio_clone(bio, bio_src, gfp);
if (ret)
bio_uninit(bio);
@@ -1182,15 +1181,19 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty)
}
EXPORT_SYMBOL_GPL(__bio_release_pages);
void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter)
bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter)
{
if (!iov_iter_is_bvec(iter))
return false;
WARN_ON_ONCE(bio->bi_max_vecs);
bio->bi_io_vec = (struct bio_vec *)iter->bvec;
bio->bi_iter.bi_idx = 0;
bio->bi_iter.bi_bvec_done = iter->iov_offset;
bio->bi_iter.bi_offset = iter->iov_offset;
bio->bi_iter.bi_size = iov_iter_count(iter);
bio_set_flag(bio, BIO_CLONED);
return true;
}
/*
@@ -1221,10 +1224,45 @@ static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,
return 0;
}
#ifdef CONFIG_DEBUG_KERNEL
static inline bool bio_iov_bvec_aligned(const struct bio *bio,
unsigned mem_align_mask)
{
struct bvec_iter iter;
struct bio_vec bv;
/*
* Correct callers never break the alignment requirements, so this
* exhaustive check is only paid for in debug builds.
*/
for_each_mp_bvec(bv, bio->bi_io_vec, iter, bio->bi_iter)
if ((bv.bv_offset | bv.bv_len) & mem_align_mask)
return false;
return true;
}
#else
static inline bool bio_iov_bvec_aligned(const struct bio *bio,
unsigned mem_align_mask)
{
/*
* We forward the bio_vec as-is, so ITER_BVEC callers must provide
* segments already aligned to the device's DMA alignment. The only
* unchecked user-controllable offset that reaches here is an io_uring
* registered buffer where just the first segment can be unaligned
* (the rest is virtually contiguous), so checking only that one is
* sufficient to know if the entire vector is valid.
*/
return !(mp_bvec_iter_offset(bio->bi_io_vec, bio->bi_iter) &
mem_align_mask);
}
#endif
/**
* bio_iov_iter_get_pages - add user or kernel pages to a bio
* @bio: bio to add pages to
* @iter: iov iterator describing the region to be added
* @mem_align_mask: the mask the source address and length must be aligned to,
* 0 for no requirement
* @len_align_mask: the mask to align the total size to, 0 for any length
*
* This takes either an iterator pointing to user memory, or one pointing to
@@ -1243,15 +1281,18 @@ static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,
* is returned only if 0 pages could be pinned.
*/
int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
unsigned len_align_mask)
unsigned mem_align_mask, unsigned len_align_mask)
{
iov_iter_extraction_t flags = 0;
if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))
return -EIO;
if (iov_iter_is_bvec(iter)) {
bio_iov_bvec_set(bio, iter);
if (bio_iov_iter_set(bio, iter)) {
if (iov_iter_is_bvec(iter) &&
!bio_iov_bvec_aligned(bio, mem_align_mask))
return -EINVAL;
iov_iter_advance(iter, bio->bi_iter.bi_size);
return 0;
}
@@ -1266,8 +1307,19 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
ret = iov_iter_extract_bvecs(iter, bio->bi_io_vec,
BIO_MAX_SIZE - bio->bi_iter.bi_size,
&bio->bi_vcnt, bio->bi_max_vecs, flags);
&bio->bi_vcnt, bio->bi_max_vecs,
mem_align_mask, flags);
if (ret <= 0) {
/*
* A misaligned vector fails the whole I/O. Release any
* pages pinned by earlier iterations before returning
* since this bio won't be submitted to release them.
*/
if (ret == -EINVAL) {
bio_release_pages(bio, false);
bio_clear_flag(bio, BIO_PAGE_PINNED);
bio->bi_vcnt = 0;
}
if (!bio->bi_vcnt)
return ret;
break;
@@ -1380,7 +1432,7 @@ static int bio_iov_iter_bounce_read(struct bio *bio, struct iov_iter *iter,
do {
ret = iov_iter_extract_bvecs(iter, bio->bi_io_vec + 1, len,
&bio->bi_vcnt, bio->bi_max_vecs - 1, 0);
&bio->bi_vcnt, bio->bi_max_vecs - 1, 0, 0);
if (ret <= 0) {
if (!bio->bi_vcnt)
goto out_folio_put;
@@ -1741,6 +1793,61 @@ defer:
schedule_work(&bio_dirty_work);
}
/*
* Infrastructure for deferring bio completions to task-context via a per-CPU
* workqueue. Triggered either by the BIO_COMPLETE_IN_TASK bio flag (static
* decision at submit time) or by calling bio_complete_in_task() from
* bi_end_io() (dynamic decision at completion time).
*/
struct bio_complete_batch {
struct bio_list list;
struct work_struct work;
int cpu;
};
static DEFINE_PER_CPU(struct bio_complete_batch, bio_complete_batch);
static struct workqueue_struct *bio_complete_wq;
static void bio_complete_work_fn(struct work_struct *w)
{
struct bio_complete_batch *batch =
container_of(w, struct bio_complete_batch, work);
while (1) {
struct bio_list list;
struct bio *bio;
local_irq_disable();
list = batch->list;
bio_list_init(&batch->list);
local_irq_enable();
if (bio_list_empty(&list))
break;
while ((bio = bio_list_pop(&list)))
bio->bi_end_io(bio);
}
}
void __bio_complete_in_task(struct bio *bio)
{
struct bio_complete_batch *batch;
unsigned long flags;
bool was_empty;
local_irq_save(flags);
batch = this_cpu_ptr(&bio_complete_batch);
was_empty = bio_list_empty(&batch->list);
bio_list_add(&batch->list, bio);
local_irq_restore(flags);
if (was_empty)
queue_work_on(batch->cpu, bio_complete_wq, &batch->work);
}
EXPORT_SYMBOL_GPL(__bio_complete_in_task);
static inline bool bio_remaining_done(struct bio *bio)
{
/*
@@ -1815,7 +1922,9 @@ again:
}
#endif
if (bio->bi_end_io)
if (bio_flagged(bio, BIO_COMPLETE_IN_TASK) && bio_in_atomic())
__bio_complete_in_task(bio);
else if (bio->bi_end_io)
bio->bi_end_io(bio);
}
EXPORT_SYMBOL(bio_endio);
@@ -2001,6 +2110,55 @@ bad:
}
EXPORT_SYMBOL(bioset_init);
static int bio_complete_batch_cpu_online(unsigned int cpu)
{
struct bio_complete_batch *batch = &per_cpu(bio_complete_batch, cpu);
enable_work(&batch->work);
if (!bio_list_empty(&batch->list))
queue_work_on(cpu, bio_complete_wq, &batch->work);
return 0;
}
/*
* Disable this CPU's work item so that it cannot run on an unbound worker
* after the CPU is offlined.
*/
static int bio_complete_batch_cpu_down_prep(unsigned int cpu)
{
disable_work_sync(&per_cpu(bio_complete_batch, cpu).work);
return 0;
}
/*
* Drain a dead CPU's deferred bio completions. The CPU is dead and the worker
* is canceled so no locking is needed.
*/
static int bio_complete_batch_cpu_dead(unsigned int cpu)
{
struct bio_complete_batch *batch =
per_cpu_ptr(&bio_complete_batch, cpu);
struct bio *bio;
while ((bio = bio_list_pop(&batch->list)))
bio->bi_end_io(bio);
return 0;
}
static void __init bio_complete_batch_init(int cpu)
{
struct bio_complete_batch *batch =
per_cpu_ptr(&bio_complete_batch, cpu);
bio_list_init(&batch->list);
INIT_WORK(&batch->work, bio_complete_work_fn);
batch->cpu = cpu;
if (!cpu_online(cpu))
disable_work_sync(&batch->work);
}
static int __init init_bio(void)
{
int i;
@@ -2015,6 +2173,30 @@ static int __init init_bio(void)
SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL);
}
for_each_possible_cpu(i)
bio_complete_batch_init(i);
bio_complete_wq = alloc_workqueue("bio_complete",
WQ_MEM_RECLAIM | WQ_PERCPU, 0);
if (!bio_complete_wq)
panic("bio: can't allocate bio_complete workqueue\n");
/*
* bio task-context completion draining on hot-unplugged CPUs:
*
* 1. Stop the per-CPU work item while the CPU is still online, so
* that it cannot run on an unbound worker later.
* 2. Drain leftover bios added between worker disabling and CPU
* offlining.
*/
cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
"block/bio:complete:online",
bio_complete_batch_cpu_online,
bio_complete_batch_cpu_down_prep);
cpuhp_setup_state_nocalls(CPUHP_BP_PREPARE_DYN,
"block/bio:complete:dead",
NULL, bio_complete_batch_cpu_dead);
cpuhp_setup_state_multi(CPUHP_BIO_DEAD, "block/bio:dead", NULL,
bio_cpu_dead);
block/blk-cgroup-rwstat.c
+2 -3
View File
@@ -88,8 +88,7 @@ EXPORT_SYMBOL_GPL(blkg_prfill_rwstat);
* @sum: blkg_rwstat_sample structure containing the results
*
* Collect the blkg_rwstat specified by @blkg, @pol and @off and all its
* online descendants and their aux counts. The caller must be holding the
* queue lock for online tests.
* online descendants and their aux counts.
*
* If @pol is NULL, blkg_rwstat is at @off bytes into @blkg; otherwise, it
* is at @off bytes into @blkg's blkg_policy_data of the policy.
@@ -107,7 +106,7 @@ void blkg_rwstat_recursive_sum(struct blkcg_gq *blkg, struct blkcg_policy *pol,
blkg_for_each_descendant_pre(pos_blkg, pos_css, blkg) {
struct blkg_rwstat *rwstat;
if (!pos_blkg->online)
if (!data_race(pos_blkg->online))
continue;
if (pol) {
block/blk-cgroup.c
+49 -31
View File
@@ -53,6 +53,9 @@ EXPORT_SYMBOL_GPL(blkcg_root);
struct cgroup_subsys_state * const blkcg_root_css = &blkcg_root.css;
EXPORT_SYMBOL_GPL(blkcg_root_css);
/* number of blkcgs with a non-zero congestion_count */
atomic_t blkcg_nr_congested __read_mostly = ATOMIC_INIT(0);
static struct blkcg_policy *blkcg_policy[BLKCG_MAX_POLS];
static LIST_HEAD(all_blkcgs); /* protected by blkcg_pol_mutex */
@@ -569,6 +572,7 @@ static void blkg_destroy_all(struct gendisk *disk)
int i;
restart:
mutex_lock(&q->blkcg_mutex);
spin_lock_irq(&q->queue_lock);
list_for_each_entry(blkg, &q->blkg_list, q_node) {
struct blkcg *blkcg = blkg->blkcg;
@@ -587,6 +591,7 @@ restart:
if (!(--count)) {
count = BLKG_DESTROY_BATCH_SIZE;
spin_unlock_irq(&q->queue_lock);
mutex_unlock(&q->blkcg_mutex);
cond_resched();
goto restart;
}
@@ -606,6 +611,7 @@ restart:
q->root_blkg = NULL;
spin_unlock_irq(&q->queue_lock);
mutex_unlock(&q->blkcg_mutex);
wake_up_var(&q->root_blkg);
}
@@ -1347,6 +1353,16 @@ static void blkcg_css_free(struct cgroup_subsys_state *css)
struct blkcg *blkcg = css_to_blkcg(css);
int i;
/*
* Every blkg holds a reference on this css and drops any delay it
* still has from pd_free_fn(), so this is expected to be zero. Should
* a policy ever leave one behind, drop it here rather than let it pin
* blkcg_nr_congested and disable the fast path for the rest of the
* boot. Nothing can race with us at this point.
*/
if (WARN_ON_ONCE(atomic_xchg(&blkcg->congestion_count, 0) > 0))
atomic_dec(&blkcg_nr_congested);
mutex_lock(&blkcg_pol_mutex);
list_del(&blkcg->all_blkcgs_node);
@@ -1526,6 +1542,31 @@ struct cgroup_subsys io_cgrp_subsys = {
};
EXPORT_SYMBOL_GPL(io_cgrp_subsys);
/*
* Tear down per-blkg policy data for @pol on @q.
*/
static void blkcg_policy_teardown_pds(struct request_queue *q,
const struct blkcg_policy *pol)
{
struct blkcg_gq *blkg;
list_for_each_entry(blkg, &q->blkg_list, q_node) {
struct blkcg *blkcg = blkg->blkcg;
struct blkg_policy_data *pd;
spin_lock(&blkcg->lock);
pd = blkg->pd[pol->plid];
if (pd) {
if (pd->online && pol->pd_offline_fn)
pol->pd_offline_fn(pd);
pd->online = false;
pol->pd_free_fn(pd);
WRITE_ONCE(blkg->pd[pol->plid], NULL);
}
spin_unlock(&blkcg->lock);
}
}
/**
* blkcg_activate_policy - activate a blkcg policy on a gendisk
* @disk: gendisk of interest
@@ -1563,6 +1604,8 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
if (queue_is_mq(q))
memflags = blk_mq_freeze_queue(q);
mutex_lock(&q->blkcg_mutex);
retry:
spin_lock_irq(&q->queue_lock);
@@ -1572,6 +1615,8 @@ retry:
if (blkg->pd[pol->plid])
continue;
if (hlist_unhashed(&blkg->blkcg_node))
continue;
/* If prealloc matches, use it; otherwise try GFP_NOWAIT */
if (blkg == pinned_blkg) {
@@ -1625,6 +1670,7 @@ retry:
spin_unlock_irq(&q->queue_lock);
out:
mutex_unlock(&q->blkcg_mutex);
if (queue_is_mq(q))
blk_mq_unfreeze_queue(q, memflags);
if (pinned_blkg)
@@ -1636,21 +1682,7 @@ out:
enomem:
/* alloc failed, take down everything */
spin_lock_irq(&q->queue_lock);
list_for_each_entry(blkg, &q->blkg_list, q_node) {
struct blkcg *blkcg = blkg->blkcg;
struct blkg_policy_data *pd;
spin_lock(&blkcg->lock);
pd = blkg->pd[pol->plid];
if (pd) {
if (pd->online && pol->pd_offline_fn)
pol->pd_offline_fn(pd);
pd->online = false;
pol->pd_free_fn(pd);
WRITE_ONCE(blkg->pd[pol->plid], NULL);
}
spin_unlock(&blkcg->lock);
}
blkcg_policy_teardown_pds(q, pol);
spin_unlock_irq(&q->queue_lock);
ret = -ENOMEM;
goto out;
@@ -1669,7 +1701,6 @@ void blkcg_deactivate_policy(struct gendisk *disk,
const struct blkcg_policy *pol)
{
struct request_queue *q = disk->queue;
struct blkcg_gq *blkg;
unsigned int memflags;
if (!blkcg_policy_enabled(q, pol))
@@ -1682,20 +1713,7 @@ void blkcg_deactivate_policy(struct gendisk *disk,
spin_lock_irq(&q->queue_lock);
__clear_bit(pol->plid, q->blkcg_pols);
list_for_each_entry(blkg, &q->blkg_list, q_node) {
struct blkcg *blkcg = blkg->blkcg;
spin_lock(&blkcg->lock);
if (blkg->pd[pol->plid]) {
if (blkg->pd[pol->plid]->online && pol->pd_offline_fn)
pol->pd_offline_fn(blkg->pd[pol->plid]);
pol->pd_free_fn(blkg->pd[pol->plid]);
blkg->pd[pol->plid] = NULL;
}
spin_unlock(&blkcg->lock);
}
blkcg_policy_teardown_pds(q, pol);
spin_unlock_irq(&q->queue_lock);
mutex_unlock(&q->blkcg_mutex);
@@ -2223,7 +2241,7 @@ void blk_cgroup_bio_start(struct bio *bio)
put_cpu();
}
bool blk_cgroup_congested(void)
bool __blk_cgroup_congested(void)
{
struct blkcg *blkcg;
bool ret = false;
block/blk-cgroup.h
+21 -4
View File
@@ -375,12 +375,29 @@ static inline void blkg_put(struct blkcg_gq *blkg)
if (((d_blkg) = blkg_lookup(css_to_blkcg(pos_css), \
(p_blkg)->q)))
/*
* blkcg_nr_congested gates the hierarchy walk in blk_cgroup_congested().
* These two helpers keep it in step with each blkcg's congestion_count in
* normal operation; blkcg_css_free() drops a residual count as a backstop.
*/
static inline void blkcg_inc_congestion_count(struct blkcg *blkcg)
{
if (atomic_inc_return(&blkcg->congestion_count) == 1)
atomic_inc(&blkcg_nr_congested);
}
static inline void blkcg_dec_congestion_count(struct blkcg *blkcg)
{
if (atomic_dec_and_test(&blkcg->congestion_count))
atomic_dec(&blkcg_nr_congested);
}
static inline void blkcg_use_delay(struct blkcg_gq *blkg)
{
if (WARN_ON_ONCE(atomic_read(&blkg->use_delay) < 0))
return;
if (atomic_add_return(1, &blkg->use_delay) == 1)
atomic_inc(&blkg->blkcg->congestion_count);
blkcg_inc_congestion_count(blkg->blkcg);
}
static inline int blkcg_unuse_delay(struct blkcg_gq *blkg)
@@ -405,7 +422,7 @@ static inline int blkcg_unuse_delay(struct blkcg_gq *blkg)
if (old == 0)
return 0;
if (old == 1)
atomic_dec(&blkg->blkcg->congestion_count);
blkcg_dec_congestion_count(blkg->blkcg);
return 1;
}
@@ -424,7 +441,7 @@ static inline void blkcg_set_delay(struct blkcg_gq *blkg, u64 delay)
/* We only want 1 person setting the congestion count for this blkg. */
if (!old && atomic_try_cmpxchg(&blkg->use_delay, &old, -1))
atomic_inc(&blkg->blkcg->congestion_count);
blkcg_inc_congestion_count(blkg->blkcg);
atomic64_set(&blkg->delay_nsec, delay);
}
@@ -441,7 +458,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq *blkg)
/* We only want 1 person clearing the congestion count for this blkg. */
if (old && atomic_try_cmpxchg(&blkg->use_delay, &old, 0))
atomic_dec(&blkg->blkcg->congestion_count);
blkcg_dec_congestion_count(blkg->blkcg);
}
/**
block/blk-core.c
+5 -1
View File
@@ -897,10 +897,14 @@ void submit_bio_noacct(struct bio *bio)
if (!q->limits.max_write_zeroes_sectors)
goto not_supported;
break;
case REQ_OP_ZONE_RESET:
case REQ_OP_ZONE_OPEN:
case REQ_OP_ZONE_CLOSE:
case REQ_OP_ZONE_RESET:
case REQ_OP_ZONE_FINISH:
/* Zone management operations require sequential zones. */
if (!bdev_zone_is_seq(bio->bi_bdev, bio->bi_iter.bi_sector))
goto end_io;
break;
case REQ_OP_ZONE_RESET_ALL:
if (!bdev_is_zoned(bio->bi_bdev))
goto not_supported;
block/blk-iocost.c
+14 -7
View File
@@ -1592,7 +1592,7 @@ static void ioc_lat_stat(struct ioc *ioc, u32 *missed_ppm_ar, u32 *rq_wait_pct_p
u64 rq_wait_ns = 0;
int cpu, rw;
for_each_online_cpu(cpu) {
for_each_possible_cpu(cpu) {
struct ioc_pcpu_stat *stat = per_cpu_ptr(ioc->pcpu_stat, cpu);
u64 this_rq_wait_ns;
@@ -3063,6 +3063,7 @@ static void iocg_release(struct rcu_head *rcu)
static void ioc_pd_free(struct blkg_policy_data *pd)
{
struct ioc_gq *iocg = pd_to_iocg(pd);
struct blkcg_gq *blkg = pd_to_blkg(pd);
struct ioc *ioc = iocg->ioc;
unsigned long flags;
@@ -3085,6 +3086,12 @@ static void ioc_pd_free(struct blkg_policy_data *pd)
hrtimer_cancel(&iocg->waitq_timer);
}
/* off ->active_iocgs and timer gone, so nothing can re-arm the delay */
iocg->delay = 0;
iocg->indelay_since = 0;
if (blkg)
blkcg_clear_delay(blkg);
call_rcu(&pd->rcu_head, iocg_release);
}
@@ -3093,23 +3100,23 @@ static void ioc_pd_stat(struct blkg_policy_data *pd, struct seq_file *s)
struct ioc_gq *iocg = pd_to_iocg(pd);
struct ioc *ioc = iocg->ioc;
if (!ioc->enabled)
if (!data_race(ioc->enabled))
return;
if (iocg->level == 0) {
unsigned vp10k = DIV64_U64_ROUND_CLOSEST(
ioc->vtime_base_rate * 10000,
data_race(ioc->vtime_base_rate) * 10000,
VTIME_PER_USEC);
seq_printf(s, " cost.vrate=%u.%02u", vp10k / 100, vp10k % 100);
}
seq_printf(s, " cost.usage=%llu", iocg->last_stat.usage_us);
seq_printf(s, " cost.usage=%llu", data_race(iocg->last_stat.usage_us));
if (blkcg_debug_stats)
seq_printf(s, " cost.wait=%llu cost.indebt=%llu cost.indelay=%llu",
iocg->last_stat.wait_us,
iocg->last_stat.indebt_us,
iocg->last_stat.indelay_us);
data_race(iocg->last_stat.wait_us),
data_race(iocg->last_stat.indebt_us),
data_race(iocg->last_stat.indelay_us));
}
static u64 ioc_weight_prfill(struct seq_file *sf, struct blkg_policy_data *pd,
block/blk-iolatency.c
+11 -2
View File
@@ -523,7 +523,7 @@ static void iolatency_check_latencies(struct iolatency_grp *iolat, u64 now)
latency_stat_init(iolat, &stat);
preempt_disable();
for_each_online_cpu(cpu) {
for_each_possible_cpu(cpu) {
struct latency_stat *s;
s = per_cpu_ptr(iolat->stats, cpu);
latency_stat_sum(iolat, &stat, s);
@@ -925,7 +925,7 @@ static void iolatency_ssd_stat(struct iolatency_grp *iolat, struct seq_file *s)
latency_stat_init(iolat, &stat);
preempt_disable();
for_each_online_cpu(cpu) {
for_each_possible_cpu(cpu) {
struct latency_stat *s;
s = per_cpu_ptr(iolat->stats, cpu);
latency_stat_sum(iolat, &stat, s);
@@ -1043,6 +1043,15 @@ static void iolat_release(struct rcu_head *rcu)
static void iolatency_pd_free(struct blkg_policy_data *pd)
{
struct blkcg_gq *blkg = pd_to_blkg(pd);
/*
* Groups throttled as collateral have min_lat_nsec == 0, so
* iolatency_pd_offline() leaves their delay set. Drop it here, where
* no in-flight bio can re-arm it via check_scale_change().
*/
if (blkg)
blkcg_clear_delay(blkg);
call_rcu(&pd->rcu_head, iolat_release);
}
block/blk-map.c
+2 -2
View File
@@ -274,7 +274,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter,
* No alignment requirements on our part to support arbitrary
* passthrough commands.
*/
ret = bio_iov_iter_get_pages(bio, iter, 0);
ret = bio_iov_iter_get_pages(bio, iter, 0, 0);
if (ret)
goto out_put;
ret = blk_rq_append_bio(rq, bio);
@@ -473,7 +473,7 @@ static int blk_rq_map_user_bvec(struct request *rq, const struct iov_iter *iter)
bio = blk_rq_map_bio_alloc(rq, 0, GFP_KERNEL);
if (!bio)
return -ENOMEM;
bio_iov_bvec_set(bio, iter);
bio_iov_iter_set(bio, iter);
ret = blk_rq_append_bio(rq, bio);
if (ret)
block/blk-merge.c
+4 -4
View File
@@ -33,7 +33,7 @@ static inline void bio_get_last_bvec(struct bio *bio, struct bio_vec *bv)
bio_advance_iter(bio, &iter, iter.bi_size);
if (!iter.bi_bvec_done)
if (!iter.bi_offset)
idx = iter.bi_idx - 1;
else /* in the middle of bvec */
idx = iter.bi_idx;
@@ -41,11 +41,11 @@ static inline void bio_get_last_bvec(struct bio *bio, struct bio_vec *bv)
*bv = bio->bi_io_vec[idx];
/*
* iter.bi_bvec_done records actual length of the last bvec
* iter.bi_offset records actual length of the last bvec
* if this bio ends in the middle of one io vector
*/
if (iter.bi_bvec_done)
bv->bv_len = iter.bi_bvec_done;
if (iter.bi_offset)
bv->bv_len = iter.bi_offset;
}
static inline bool bio_will_gap(struct request_queue *q,
block/blk-mq-dma.c
+1 -1
View File
@@ -44,7 +44,7 @@ static bool blk_map_iter_next(struct request *req, struct blk_map_iter *iter,
* one could be merged into it. This typically happens when moving to
* the next bio, but some callers also don't pack bvecs tight.
*/
while (!iter->iter.bi_size || !iter->iter.bi_bvec_done) {
while (!iter->iter.bi_size || !iter->iter.bi_offset) {
struct bio_vec next;
if (!__blk_map_iter_next(iter))
block/blk-mq.c
+1
View File
@@ -4975,6 +4975,7 @@ void blk_mq_free_tag_set(struct blk_mq_tag_set *set)
srcu_barrier(&set->tags_srcu);
cleanup_srcu_struct(&set->tags_srcu);
if (set->flags & BLK_MQ_F_BLOCKING) {
srcu_barrier(set->srcu);
cleanup_srcu_struct(set->srcu);
kfree(set->srcu);
}
block/blk-stat.c
+1 -1
View File
@@ -83,7 +83,7 @@ static void blk_stat_timer_fn(struct timer_list *t)
for (bucket = 0; bucket < cb->buckets; bucket++)
blk_rq_stat_init(&cb->stat[bucket]);
for_each_online_cpu(cpu) {
for_each_possible_cpu(cpu) {
struct blk_rq_stat *cpu_stat;
cpu_stat = per_cpu_ptr(cb->cpu_stat, cpu);
block/blk-throttle.h
-2
View File
@@ -120,8 +120,6 @@ struct throtl_grp {
int64_t bytes_disp[2];
int io_disp[2];
unsigned long last_check_time;
/* When did we start a new slice */
unsigned long slice_start[2];
unsigned long slice_end[2];
block/blk-zoned.c
-30
View File
@@ -1574,30 +1574,6 @@ static void blk_zone_wplug_handle_native_zone_append(struct bio *bio)
disk_put_zone_wplug(zwplug);
}
static bool blk_zone_wplug_handle_zone_mgmt(struct bio *bio)
{
if (bio_op(bio) != REQ_OP_ZONE_RESET_ALL &&
!bdev_zone_is_seq(bio->bi_bdev, bio->bi_iter.bi_sector)) {
/*
* Zone reset and zone finish operations do not apply to
* conventional zones.
*/
bio_io_error(bio);
return true;
}
/*
* No-wait zone management BIOs do not make much sense as the callers
* issue these as blocking operations in most cases. To avoid issues
* with the BIO execution potentially failing with BLK_STS_AGAIN, warn
* about REQ_NOWAIT being set and ignore that flag.
*/
if (WARN_ON_ONCE(bio->bi_opf & REQ_NOWAIT))
bio->bi_opf &= ~REQ_NOWAIT;
return false;
}
/**
* blk_zone_plug_bio - Handle a zone write BIO with zone write plugging
* @bio: The BIO being submitted
@@ -1644,15 +1620,9 @@ bool blk_zone_plug_bio(struct bio *bio, unsigned int nr_segs)
case REQ_OP_WRITE:
case REQ_OP_WRITE_ZEROES:
return blk_zone_wplug_handle_write(bio, nr_segs);
case REQ_OP_ZONE_RESET:
case REQ_OP_ZONE_FINISH:
case REQ_OP_ZONE_RESET_ALL:
return blk_zone_wplug_handle_zone_mgmt(bio);
default:
return false;
}
return false;
}
EXPORT_SYMBOL_GPL(blk_zone_plug_bio);
block/blk.h
+1 -1
View File
@@ -406,7 +406,7 @@ static inline bool bio_may_need_split(struct bio *bio,
return true;
bv = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter);
if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_bvec_done)
if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_offset)
return true;
if ((bv->bv_offset | bv->bv_len) & lim->dma_alignment)
return true;
block/fops.c
+14 -14
View File
@@ -46,7 +46,7 @@ static bool blkdev_dio_invalid(struct block_device *bdev, struct kiocb *iocb,
static inline int blkdev_iov_iter_get_pages(struct bio *bio,
struct iov_iter *iter, struct block_device *bdev)
{
return bio_iov_iter_get_pages(bio, iter,
return bio_iov_iter_get_pages(bio, iter, bdev_dma_alignment(bdev),
bdev_logical_block_size(bdev) - 1);
}
@@ -218,7 +218,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
ret = blkdev_iov_iter_get_pages(bio, iter, bdev);
if (unlikely(ret)) {
bio_endio_status(bio, BLK_STS_IOERR);
bio_endio_status(bio, errno_to_blk_status(ret));
break;
}
if (iocb->ki_flags & IOCB_NOWAIT) {
@@ -238,8 +238,10 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
}
if (iocb->ki_flags & IOCB_HAS_METADATA) {
ret = bio_integrity_map_iter(bio, iocb->private);
if (unlikely(ret))
goto fail;
if (unlikely(ret)) {
bio_endio_status(bio, errno_to_blk_status(ret));
break;
}
}
if (is_read) {
@@ -340,15 +342,13 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
bio->bi_end_io = blkdev_bio_end_io_async;
bio->bi_ioprio = iocb->ki_ioprio;
if (iov_iter_is_bvec(iter)) {
/*
* Users don't rely on the iterator being in any particular
* state for async I/O returning -EIOCBQUEUED, hence we can
* avoid expensive iov_iter_advance(). Bypass
* bio_iov_iter_get_pages() and set the bvec directly.
*/
bio_iov_bvec_set(bio, iter);
} else {
/*
* Users don't rely on the iterator being in any particular
* state for async I/O returning -EIOCBQUEUED, hence we can
* avoid expensive iov_iter_advance(). Bypass
* bio_iov_iter_get_pages() and set the bvec directly.
*/
if (!bio_iov_iter_set(bio, iter)) {
ret = blkdev_iov_iter_get_pages(bio, iter, bdev);
if (unlikely(ret))
goto out_bio_put;
@@ -943,7 +943,7 @@ const struct file_operations def_blk_fops = {
.splice_write = iter_file_splice_write,
.fallocate = blkdev_fallocate,
.uring_cmd = blkdev_uring_cmd,
.fop_flags = FOP_BUFFER_RASYNC,
.fop_flags = FOP_BUFFER_RASYNC | FOP_DONTCACHE,
};
static __init int blkdev_init(void)
block/genhd.c
+1
View File
@@ -681,6 +681,7 @@ static bool __blk_mark_disk_dead(struct gendisk *disk)
*/
void blk_mark_disk_dead(struct gendisk *disk)
{
blk_queue_flag_set(QUEUE_FLAG_DYING, disk->queue);
__blk_mark_disk_dead(disk);
blk_report_disk_dead(disk, true);
}
block/ioctl.c
+47 -13
View File
@@ -14,6 +14,7 @@
#include <linux/pagemap.h>
#include <linux/io_uring/cmd.h>
#include <linux/blk-integrity.h>
#include <trace/events/block.h>
#include <uapi/linux/blkdev.h>
#include "blk.h"
#include "blk-crypto-internal.h"
@@ -887,14 +888,13 @@ static void bio_cmd_bio_end_io(struct bio *bio)
bio_put(bio);
}
static int blkdev_cmd_discard(struct io_uring_cmd *cmd,
struct block_device *bdev,
uint64_t start, uint64_t len, bool nowait)
static int blkdev_cmd_discard(struct io_uring_cmd *cmd)
{
struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd);
gfp_t gfp = nowait ? GFP_NOWAIT : GFP_KERNEL;
sector_t sector = start >> SECTOR_SHIFT;
sector_t nr_sects = len >> SECTOR_SHIFT;
struct block_device *bdev = I_BDEV(cmd->file->f_mapping->host);
gfp_t gfp = bic->nowait ? GFP_NOWAIT : GFP_KERNEL;
sector_t sector = bic->start >> SECTOR_SHIFT;
sector_t nr_sects = bic->len >> SECTOR_SHIFT;
struct bio *prev = NULL, *bio;
int err;
@@ -904,12 +904,12 @@ static int blkdev_cmd_discard(struct io_uring_cmd *cmd,
return -EBADF;
if (bdev_read_only(bdev))
return -EPERM;
err = blk_validate_byte_range(bdev, start, len);
err = blk_validate_byte_range(bdev, bic->start, bic->len);
if (err)
return err;
err = filemap_invalidate_pages(bdev->bd_mapping, start,
start + len - 1, nowait);
err = filemap_invalidate_pages(bdev->bd_mapping, bic->start,
bic->start + bic->len - 1, bic->nowait);
if (err)
return err;
@@ -917,7 +917,7 @@ static int blkdev_cmd_discard(struct io_uring_cmd *cmd,
bio = blk_alloc_discard_bio(bdev, &sector, &nr_sects, gfp);
if (!bio)
break;
if (nowait) {
if (bic->nowait) {
/*
* Don't allow multi-bio non-blocking submissions as
* subsequent bios may fail but we won't get a direct
@@ -944,9 +944,42 @@ static int blkdev_cmd_discard(struct io_uring_cmd *cmd,
return -EIOCBQUEUED;
}
static int blkdev_cmd_zone_reset_all(struct io_uring_cmd *cmd)
{
struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd);
struct block_device *bdev = I_BDEV(cmd->file->f_mapping->host);
struct bio *bio;
int err;
if (!(file_to_blk_mode(cmd->file) & BLK_OPEN_WRITE))
return -EBADF;
if (bdev_read_only(bdev))
return -EPERM;
if (!bdev_is_zoned(bdev))
return -EOPNOTSUPP;
if (bic->start || bic->len)
return -EINVAL;
err = filemap_invalidate_pages(bdev->bd_mapping, 0,
bdev_nr_bytes(bdev) - 1, bic->nowait);
if (err)
return err;
bio = bio_alloc(bdev, 0, REQ_OP_ZONE_RESET_ALL,
bic->nowait ? GFP_NOWAIT : GFP_KERNEL);
if (!bio)
return -EAGAIN;
if (bic->nowait)
bio->bi_opf |= REQ_NOWAIT;
trace_blkdev_zone_mgmt(bio, 0);
bio->bi_private = cmd;
bio->bi_end_io = bio_cmd_bio_end_io;
submit_bio(bio);
return -EIOCBQUEUED;
}
int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
{
struct block_device *bdev = I_BDEV(cmd->file->f_mapping->host);
struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd);
u32 cmd_op = cmd->cmd_op;
@@ -967,8 +1000,9 @@ int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags)
switch (cmd_op) {
case BLOCK_URING_CMD_DISCARD:
return blkdev_cmd_discard(cmd, bdev, bic->start, bic->len,
bic->nowait);
return blkdev_cmd_discard(cmd);
case BLOCK_URING_CMD_ZONE_RESET_ALL:
return blkdev_cmd_zone_reset_all(cmd);
}
return -EINVAL;
}
block/kyber-iosched.c
+1 -1
View File
@@ -275,7 +275,7 @@ static void kyber_timer_fn(struct timer_list *t)
bool bad = false;
/* Sum all of the per-cpu latency histograms. */
for_each_online_cpu(cpu) {
for_each_possible_cpu(cpu) {
struct kyber_cpu_latency *cpu_latency;
cpu_latency = per_cpu_ptr(kqd->cpu_latency, cpu);
block/mq-deadline.c
+11 -14
View File
@@ -233,9 +233,8 @@ static void dd_merged_requests(struct request_queue *q, struct request *req,
/*
* move an entry to dispatch queue
*/
static void
deadline_move_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
struct request *rq)
static void deadline_move_request(struct dd_per_prio *per_prio,
struct request *rq)
{
/*
* take it off the sort and fifo list
@@ -269,9 +268,8 @@ static inline bool deadline_check_fifo(struct dd_per_prio *per_prio,
* For the specified data direction, return the next request to
* dispatch using arrival ordered lists.
*/
static struct request *
deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
enum dd_data_dir data_dir)
static struct request *deadline_fifo_request(struct dd_per_prio *per_prio,
enum dd_data_dir data_dir)
{
if (list_empty(&per_prio->fifo_list[data_dir]))
return NULL;
@@ -283,9 +281,8 @@ deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
* For the specified data direction, return the next request to
* dispatch using sector position sorted lists.
*/
static struct request *
deadline_next_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
enum dd_data_dir data_dir)
static struct request *deadline_next_request(struct dd_per_prio *per_prio,
enum dd_data_dir data_dir)
{
return deadline_from_pos(per_prio, data_dir,
per_prio->latest_pos[data_dir]);
@@ -334,7 +331,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd,
/*
* batches are currently reads XOR writes
*/
rq = deadline_next_request(dd, per_prio, dd->last_dir);
rq = deadline_next_request(per_prio, dd->last_dir);
if (rq && dd->batching < dd->fifo_batch) {
/* we have a next request and are still entitled to batch */
data_dir = rq_data_dir(rq);
@@ -349,7 +346,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd,
if (!list_empty(&per_prio->fifo_list[DD_READ])) {
BUG_ON(RB_EMPTY_ROOT(&per_prio->sort_list[DD_READ]));
if (deadline_fifo_request(dd, per_prio, DD_WRITE) &&
if (deadline_fifo_request(per_prio, DD_WRITE) &&
(dd->starved++ >= dd->writes_starved))
goto dispatch_writes;
@@ -379,14 +376,14 @@ dispatch_find_request:
/*
* we are not running a batch, find best request for selected data_dir
*/
next_rq = deadline_next_request(dd, per_prio, data_dir);
next_rq = deadline_next_request(per_prio, data_dir);
if (deadline_check_fifo(per_prio, data_dir) || !next_rq) {
/*
* A deadline has expired, the last request was in the other
* direction, or we have run out of higher-sectored requests.
* Start again from the request with the earliest expiry time.
*/
rq = deadline_fifo_request(dd, per_prio, data_dir);
rq = deadline_fifo_request(per_prio, data_dir);
} else {
/*
* The last req was the same dir and we have a next request in
@@ -409,7 +406,7 @@ dispatch_request:
* rq is the selected appropriate request.
*/
dd->batching++;
deadline_move_request(dd, per_prio, rq);
deadline_move_request(per_prio, rq);
return dd_start_request(dd, data_dir, rq);
}
drivers/block/loop.c
+39 -9
View File
@@ -54,6 +54,7 @@ struct loop_device {
struct file *lo_backing_file;
unsigned int lo_min_dio_size;
unsigned int lo_dio_mem_align;
struct block_device *lo_device;
gfp_t old_gfp_mask;
@@ -378,7 +379,7 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd,
iov_iter_bvec(&iter, rw,
__bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter),
nr_bvec, blk_rq_bytes(rq));
iter.iov_offset = rq->bio->bi_iter.bi_bvec_done;
iter.iov_offset = rq->bio->bi_iter.bi_offset;
}
atomic_set(&cmd->ref, 2);
@@ -447,26 +448,37 @@ static void loop_reread_partitions(struct loop_device *lo)
__func__, lo->lo_number, lo->lo_file_name, rc);
}
static unsigned int loop_query_min_dio_size(struct loop_device *lo)
static void loop_update_dio_alignment(struct loop_device *lo)
{
struct file *file = lo->lo_backing_file;
struct block_device *sb_bdev = file->f_mapping->host->i_sb->s_bdev;
struct kstat st;
/*
* Use the minimal dio alignment of the file system if provided.
* Use the dio alignment of the file system if provided. The incomoing
* request's bio_vec is forwarded to the backing file unchanged, so its
* required memory alignment becomes the device's dma_alignment when
* used for direct-io.
*/
if (!vfs_getattr(&file->f_path, &st, STATX_DIOALIGN, 0) &&
(st.result_mask & STATX_DIOALIGN))
return st.dio_offset_align;
(st.result_mask & STATX_DIOALIGN)) {
lo->lo_min_dio_size = st.dio_offset_align;
lo->lo_dio_mem_align = st.dio_mem_align - 1;
return;
}
/*
* In a perfect world this wouldn't be needed, but as of Linux 6.13 only
* a handful of file systems support the STATX_DIOALIGN flag.
*/
if (sb_bdev)
return bdev_logical_block_size(sb_bdev);
return SECTOR_SIZE;
if (sb_bdev) {
lo->lo_min_dio_size = bdev_logical_block_size(sb_bdev);
lo->lo_dio_mem_align = bdev_dma_alignment(sb_bdev);
return;
}
lo->lo_min_dio_size = SECTOR_SIZE;
lo->lo_dio_mem_align = SECTOR_SIZE - 1;
}
static inline int is_loop_device(struct file *file)
@@ -509,7 +521,7 @@ static void loop_assign_backing_file(struct loop_device *lo, struct file *file)
lo->old_gfp_mask & ~(__GFP_IO | __GFP_FS));
if (lo->lo_backing_file->f_flags & O_DIRECT)
lo->lo_flags |= LO_FLAGS_DIRECT_IO;
lo->lo_min_dio_size = loop_query_min_dio_size(lo);
loop_update_dio_alignment(lo);
}
static int loop_check_backing_file(struct file *file)
@@ -940,6 +952,19 @@ static unsigned int loop_default_blocksize(struct loop_device *lo)
return SECTOR_SIZE;
}
static void loop_set_dma_limit(struct loop_device *lo, struct queue_limits *lim)
{
/*
* Direct I/O forwards the user pages to the backing file unchanged, so
* track the backing's DMA alignment requirement as the mode is toggled.
*/
if (lo->lo_flags & LO_FLAGS_DIRECT_IO)
lim->dma_alignment = max_t(unsigned int, lo->lo_dio_mem_align,
SECTOR_SIZE - 1);
else
lim->dma_alignment = SECTOR_SIZE - 1;
}
static void loop_update_limits(struct loop_device *lo, struct queue_limits *lim,
unsigned int bsize)
{
@@ -961,6 +986,7 @@ static void loop_update_limits(struct loop_device *lo, struct queue_limits *lim,
lim->logical_block_size = bsize;
lim->physical_block_size = bsize;
lim->io_min = bsize;
loop_set_dma_limit(lo, lim);
lim->features &= ~(BLK_FEAT_WRITE_CACHE | BLK_FEAT_ROTATIONAL);
if (file->f_op->fsync && !(lo->lo_flags & LO_FLAGS_READ_ONLY))
lim->features |= BLK_FEAT_WRITE_CACHE;
@@ -1412,6 +1438,7 @@ static int loop_set_dio(struct loop_device *lo, unsigned long arg)
{
bool use_dio = !!arg;
unsigned int memflags;
struct queue_limits lim;
if (lo->lo_state != Lo_bound)
return -ENXIO;
@@ -1425,11 +1452,14 @@ static int loop_set_dio(struct loop_device *lo, unsigned long arg)
vfs_fsync(lo->lo_backing_file, 0);
}
lim = queue_limits_start_update(lo->lo_queue);
memflags = blk_mq_freeze_queue(lo->lo_queue);
if (use_dio)
lo->lo_flags |= LO_FLAGS_DIRECT_IO;
else
lo->lo_flags &= ~LO_FLAGS_DIRECT_IO;
loop_set_dma_limit(lo, &lim);
queue_limits_commit_update(lo->lo_queue, &lim);
blk_mq_unfreeze_queue(lo->lo_queue, memflags);
return 0;
}
drivers/block/mtip32xx/mtip32xx.c
+7
View File
@@ -3048,6 +3048,8 @@ static int mtip_block_ioctl(struct block_device *dev,
if (!dd)
return -ENOTTY;
guard(mutex)(&dd->ioctl_mutex);
if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag)))
return -ENOTTY;
@@ -3086,6 +3088,8 @@ static int mtip_block_compat_ioctl(struct block_device *dev,
if (!dd)
return -ENOTTY;
guard(mutex)(&dd->ioctl_mutex);
if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag)))
return -ENOTTY;
@@ -3721,6 +3725,7 @@ static int mtip_pci_probe(struct pci_dev *pdev,
dd = kzalloc_node(sizeof(struct driver_data), GFP_KERNEL, my_node);
if (!dd)
return -ENOMEM;
mutex_init(&dd->ioctl_mutex);
/* Attach the private data to this PCI device. */
pci_set_drvdata(pdev, dd);
@@ -3887,6 +3892,7 @@ static void mtip_pci_remove(struct pci_dev *pdev)
}
set_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag);
mutex_lock(&dd->ioctl_mutex);
if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
del_gendisk(dd->disk);
@@ -3915,6 +3921,7 @@ static void mtip_pci_remove(struct pci_dev *pdev)
/* De-initialize the protocol layer. */
mtip_hw_exit(dd);
mutex_unlock(&dd->ioctl_mutex);
if (dd->isr_workq) {
destroy_workqueue(dd->isr_workq);
drivers/block/mtip32xx/mtip32xx.h
+2
View File
@@ -12,6 +12,7 @@
#define __MTIP32XX_H__
#include <linux/spinlock.h>
#include <linux/mutex.h>
#include <linux/rwsem.h>
#include <linux/ata.h>
#include <linux/interrupt.h>
@@ -432,6 +433,7 @@ struct driver_data {
struct request_queue *queue; /* Our request queue. */
struct blk_mq_tag_set tags; /* blk_mq tags */
struct mutex ioctl_mutex;
struct mtip_port *port; /* Pointer to the port data structure. */
drivers/block/nbd.c
+137 -120
View File
@@ -166,6 +166,7 @@ static struct dentry *nbd_dbg_dir;
static unsigned int nbds_max = 16;
static int max_part = 16;
static int pre_defined_connections = 1;
static int part_shift;
static int nbd_dev_dbg_init(struct nbd_device *nbd);
@@ -331,6 +332,26 @@ static void nbd_mark_nsock_dead(struct nbd_device *nbd, struct nbd_sock *nsock,
nsock->sent = 0;
}
static void nbd_apply_limits(struct queue_limits *lim, u32 flags)
{
lim->features &= ~(BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA | BLK_FEAT_ROTATIONAL);
lim->max_hw_discard_sectors = 0;
lim->max_write_zeroes_sectors = 0;
if (flags & NBD_FLAG_SEND_TRIM)
lim->max_hw_discard_sectors = UINT_MAX >> SECTOR_SHIFT;
if (flags & NBD_FLAG_SEND_FLUSH) {
lim->features |= BLK_FEAT_WRITE_CACHE;
if (flags & NBD_FLAG_SEND_FUA)
lim->features |= BLK_FEAT_FUA;
}
if (flags & NBD_FLAG_ROTATIONAL)
lim->features |= BLK_FEAT_ROTATIONAL;
if (flags & NBD_FLAG_SEND_WRITE_ZEROES)
lim->max_write_zeroes_sectors = UINT_MAX >> SECTOR_SHIFT;
}
static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize)
{
struct queue_limits lim;
@@ -352,26 +373,14 @@ static int nbd_set_size(struct nbd_device *nbd, loff_t bytesize, loff_t blksize)
return 0;
lim = queue_limits_start_update(nbd->disk->queue);
if (nbd->config->flags & NBD_FLAG_SEND_TRIM)
lim.max_hw_discard_sectors = UINT_MAX >> SECTOR_SHIFT;
else
lim.max_hw_discard_sectors = 0;
if (!(nbd->config->flags & NBD_FLAG_SEND_FLUSH)) {
lim.features &= ~(BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA);
} else if (nbd->config->flags & NBD_FLAG_SEND_FUA) {
lim.features |= BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA;
} else {
lim.features |= BLK_FEAT_WRITE_CACHE;
lim.features &= ~BLK_FEAT_FUA;
}
if (nbd->config->flags & NBD_FLAG_ROTATIONAL)
lim.features |= BLK_FEAT_ROTATIONAL;
if (nbd->config->flags & NBD_FLAG_SEND_WRITE_ZEROES)
lim.max_write_zeroes_sectors = UINT_MAX >> SECTOR_SHIFT;
nbd_apply_limits(&lim, nbd->config->flags);
lim.logical_block_size = blksize;
lim.physical_block_size = blksize;
error = queue_limits_commit_update_frozen(nbd->disk->queue, &lim);
/* No need freeze with 0 capacity and write cache disabled */
if (!get_capacity(nbd->disk) && !blk_queue_write_cache(nbd->disk->queue))
error = queue_limits_commit_update(nbd->disk->queue, &lim);
else
error = queue_limits_commit_update_frozen(nbd->disk->queue, &lim);
if (error)
return error;
@@ -1061,40 +1070,31 @@ static int find_fallback(struct nbd_device *nbd, int index)
int new_index = -1;
struct nbd_sock *nsock = config->socks[index];
int fallback = nsock->fallback_index;
int i;
if (test_bit(NBD_RT_DISCONNECTED, &config->runtime_flags))
return new_index;
if (config->num_connections <= 1) {
dev_err_ratelimited(disk_to_dev(nbd->disk),
"Dead connection, failed to find a fallback\n");
return new_index;
}
if (config->num_connections <= 1)
goto no_fallback;
if (fallback >= 0 && fallback < config->num_connections &&
!config->socks[fallback]->dead)
return fallback;
if (nsock->fallback_index < 0 ||
nsock->fallback_index >= config->num_connections ||
config->socks[nsock->fallback_index]->dead) {
int i;
for (i = 0; i < config->num_connections; i++) {
if (i == index)
continue;
if (!config->socks[i]->dead) {
new_index = i;
break;
}
}
nsock->fallback_index = new_index;
if (new_index < 0) {
dev_err_ratelimited(disk_to_dev(nbd->disk),
"Dead connection, failed to find a fallback\n");
return new_index;
for (i = 0; i < config->num_connections; i++) {
if (i != index && !config->socks[i]->dead) {
new_index = i;
break;
}
}
new_index = nsock->fallback_index;
nsock->fallback_index = new_index;
if (new_index >= 0)
return new_index;
no_fallback:
dev_err_ratelimited(disk_to_dev(nbd->disk),
"Dead connection, failed to find a fallback\n");
return new_index;
}
@@ -1281,23 +1281,23 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
struct socket *sock;
struct nbd_sock **socks;
struct nbd_sock *nsock;
unsigned int memflags;
int err;
/* Arg will be cast to int, check it to avoid overflow */
if (arg > INT_MAX)
return -EINVAL;
if (nbd->pid) {
dev_err(disk_to_dev(nbd->disk),
"Cannot add socket to a running device\n");
return -EBUSY;
}
sock = nbd_get_socket(nbd, arg, &err);
if (!sock)
return err;
nbd_reclassify_socket(sock);
/*
* We need to make sure we don't get any errant requests while we're
* reallocating the ->socks array.
*/
memflags = blk_mq_freeze_queue(nbd->disk->queue);
if (!netlink && !nbd->task_setup &&
!test_bit(NBD_RT_BOUND, &config->runtime_flags))
nbd->task_setup = current;
@@ -1337,17 +1337,15 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg,
INIT_WORK(&nsock->work, nbd_pending_cmd_work);
socks[config->num_connections++] = nsock;
atomic_inc(&config->live_connections);
blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
return 0;
put_socket:
blk_mq_unfreeze_queue(nbd->disk->queue, memflags);
sockfd_put(sock);
return err;
}
static int nbd_reconnect_socket(struct nbd_device *nbd, unsigned long arg)
static int nbd_genl_reconnect_sock_cb(struct nbd_device *nbd, unsigned long arg)
{
struct nbd_config *config = nbd->config;
struct socket *sock, *old;
@@ -1402,11 +1400,12 @@ static int nbd_reconnect_socket(struct nbd_device *nbd, unsigned long arg)
atomic_inc(&config->live_connections);
wake_up(&config->conn_wait);
dev_info(nbd_to_dev(nbd), "reconnected socket\n");
return 0;
}
sockfd_put(sock);
kfree(args);
return -ENOSPC;
return 1;
}
static void nbd_bdev_reset(struct nbd_device *nbd)
@@ -1471,8 +1470,13 @@ static void nbd_config_put(struct nbd_device *nbd)
if (refcount_dec_and_mutex_lock(&nbd->config_refs,
&nbd->config_lock)) {
struct nbd_config *config = nbd->config;
struct queue_limits lim;
nbd_dev_dbg_close(nbd);
invalidate_disk(nbd->disk);
/* reset queue limits to default */
lim = queue_limits_start_update(nbd->disk->queue);
nbd_apply_limits(&lim, 0);
queue_limits_commit_update(nbd->disk->queue, &lim);
if (nbd->config->bytesize)
kobject_uevent(&nbd_to_dev(nbd)->kobj, KOBJ_CHANGE);
if (test_and_clear_bit(NBD_RT_HAS_PID_FILE,
@@ -1941,7 +1945,8 @@ static const struct blk_mq_ops nbd_mq_ops = {
.timeout = nbd_xmit_timeout,
};
static struct nbd_device *nbd_dev_add(int index, unsigned int refs)
static struct nbd_device *nbd_dev_add(int index, unsigned int refs,
int nr_hw_queues)
{
struct queue_limits lim = {
.max_hw_sectors = 65536,
@@ -1958,7 +1963,7 @@ static struct nbd_device *nbd_dev_add(int index, unsigned int refs)
goto out;
nbd->tag_set.ops = &nbd_mq_ops;
nbd->tag_set.nr_hw_queues = 1;
nbd->tag_set.nr_hw_queues = nr_hw_queues;
nbd->tag_set.queue_depth = 128;
nbd->tag_set.numa_node = NUMA_NO_NODE;
nbd->tag_set.cmd_size = sizeof(struct nbd_cmd);
@@ -2107,6 +2112,58 @@ static int nbd_genl_size_set(struct genl_info *info, struct nbd_device *nbd)
return 0;
}
/*
* Walk the NBD_ATTR_SOCKETS nested list can call @cb for each socket fd.
*
* Return the number of fds walked, or a negative errno.
*/
static int nbd_genl_foreach_sock(struct genl_info *info,
int (*cb)(struct nbd_device *nbd, unsigned long fd),
struct nbd_device *nbd)
{
struct nlattr *attr;
int rem, count = 0;
if (!info->attrs[NBD_ATTR_SOCKETS])
return 0;
nla_for_each_nested(attr, info->attrs[NBD_ATTR_SOCKETS], rem) {
struct nlattr *socks[NBD_SOCK_MAX + 1];
int ret;
if (nla_type(attr) != NBD_SOCK_ITEM) {
pr_err("socks must be embedded in a SOCK_ITEM attr\n");
return -EINVAL;
}
if (nla_parse_nested_deprecated(socks, NBD_SOCK_MAX,
attr,
nbd_sock_policy,
info->extack)) {
pr_err("error processing sock list\n");
return -EINVAL;
}
if (!socks[NBD_SOCK_FD])
continue;
count++;
if (cb) {
ret = cb(nbd, (int)nla_get_u32(socks[NBD_SOCK_FD]));
if (ret > 0)
return count;
if (ret < 0)
return ret;
}
}
return count;
}
static int nbd_genl_connect_sock_cb(struct nbd_device *nbd, unsigned long fd)
{
return nbd_add_socket(nbd, fd, true);
}
static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
{
struct nbd_device *nbd;
@@ -2159,7 +2216,11 @@ again:
mutex_unlock(&nbd_index_mutex);
if (!nbd) {
nbd = nbd_dev_add(index, 2);
ret = nbd_genl_foreach_sock(info, NULL, NULL);
if (ret < 0)
return ret;
nbd = nbd_dev_add(index, 2, ret > 0 ? ret : 1);
if (IS_ERR(nbd)) {
pr_err("failed to add new device\n");
return PTR_ERR(nbd);
@@ -2226,36 +2287,9 @@ again:
}
}
if (info->attrs[NBD_ATTR_SOCKETS]) {
struct nlattr *attr;
int rem, fd;
nla_for_each_nested(attr, info->attrs[NBD_ATTR_SOCKETS],
rem) {
struct nlattr *socks[NBD_SOCK_MAX+1];
if (nla_type(attr) != NBD_SOCK_ITEM) {
pr_err("socks must be embedded in a SOCK_ITEM attr\n");
ret = -EINVAL;
goto out;
}
ret = nla_parse_nested_deprecated(socks, NBD_SOCK_MAX,
attr,
nbd_sock_policy,
info->extack);
if (ret != 0) {
pr_err("error processing sock list\n");
ret = -EINVAL;
goto out;
}
if (!socks[NBD_SOCK_FD])
continue;
fd = (int)nla_get_u32(socks[NBD_SOCK_FD]);
ret = nbd_add_socket(nbd, fd, true);
if (ret)
goto out;
}
}
ret = nbd_genl_foreach_sock(info, nbd_genl_connect_sock_cb, nbd);
if (ret < 0)
goto out;
if (info->attrs[NBD_ATTR_BACKEND_IDENTIFIER]) {
nbd->backend = nla_strdup(info->attrs[NBD_ATTR_BACKEND_IDENTIFIER],
@@ -2440,40 +2474,10 @@ static int nbd_genl_reconfigure(struct sk_buff *skb, struct genl_info *info)
}
}
if (info->attrs[NBD_ATTR_SOCKETS]) {
struct nlattr *attr;
int rem, fd;
nla_for_each_nested(attr, info->attrs[NBD_ATTR_SOCKETS],
rem) {
struct nlattr *socks[NBD_SOCK_MAX+1];
if (nla_type(attr) != NBD_SOCK_ITEM) {
pr_err("socks must be embedded in a SOCK_ITEM attr\n");
ret = -EINVAL;
goto out;
}
ret = nla_parse_nested_deprecated(socks, NBD_SOCK_MAX,
attr,
nbd_sock_policy,
info->extack);
if (ret != 0) {
pr_err("error processing sock list\n");
ret = -EINVAL;
goto out;
}
if (!socks[NBD_SOCK_FD])
continue;
fd = (int)nla_get_u32(socks[NBD_SOCK_FD]);
ret = nbd_reconnect_socket(nbd, fd);
if (ret) {
if (ret == -ENOSPC)
ret = 0;
goto out;
}
dev_info(nbd_to_dev(nbd), "reconnected socket\n");
}
}
ret = nbd_genl_foreach_sock(info, nbd_genl_reconnect_sock_cb, nbd);
/* foreach_sock returns a positive count on success; doit must return 0 */
if (ret >= 0)
ret = 0;
out:
mutex_unlock(&nbd->config_lock);
nbd_config_put(nbd);
@@ -2709,6 +2713,12 @@ static int __init nbd_init(void)
if (nbds_max > 1UL << (MINORBITS - part_shift))
return -EINVAL;
/* An excessively large value will be adjusted in blk_mq_alloc_tag_set */
if (pre_defined_connections < 1) {
pr_err("pre_defined_connections must be >= 1\n");
return -EINVAL;
}
if (register_blkdev(NBD_MAJOR, "nbd"))
return -EIO;
@@ -2725,8 +2735,12 @@ static int __init nbd_init(void)
}
nbd_dbg_init();
/*
* Set to the intended connection count so nbd_start_device() can skip
* the queue-freezing blk_mq_update_nr_hw_queues() call.
*/
for (i = 0; i < nbds_max; i++)
nbd_dev_add(i, 1);
nbd_dev_add(i, 1, pre_defined_connections);
return 0;
}
@@ -2787,3 +2801,6 @@ module_param(nbds_max, int, 0444);
MODULE_PARM_DESC(nbds_max, "number of network block devices to initialize (default: 16)");
module_param(max_part, int, 0444);
MODULE_PARM_DESC(max_part, "number of partitions per device (default: 16)");
module_param(pre_defined_connections, int, 0444);
MODULE_PARM_DESC(pre_defined_connections,
"number of connections for devices pre-created at module load (default: 1)");
drivers/block/null_blk/main.c
+54 -52
View File
@@ -66,7 +66,7 @@ struct nullb_page {
#define NULLB_PAGE_FREE (MAP_SZ - 2)
static LIST_HEAD(nullb_list);
static struct mutex lock;
static DEFINE_MUTEX(lock);
static int null_major;
static DEFINE_IDA(nullb_indexes);
static struct blk_mq_tag_set tag_set;
@@ -340,11 +340,20 @@ static ssize_t nullb_device_bool_attr_store(bool *val, const char *page,
return count;
}
/* The following macro should only be used with TYPE = {uint, ulong, bool}. */
/*
* The following macro should only be used with TYPE = {uint, ulong, bool}.
*
* The device configuration is modified under the global lock to serialize
* attribute changes against null_add_dev() and null_del_dev(): without this,
* an attribute could be changed while null_add_dev() is running, that is,
* before NULLB_DEV_FL_CONFIGURED is set, which would let null_add_dev()
* observe inconsistent values for the device configuration.
*/
#define NULLB_DEVICE_ATTR(NAME, TYPE, APPLY) \
static ssize_t \
nullb_device_##NAME##_show(struct config_item *item, char *page) \
{ \
guard(mutex)(&lock); \
return nullb_device_##TYPE##_attr_show( \
to_nullb_device(item)->NAME, page); \
} \
@@ -360,6 +369,7 @@ nullb_device_##NAME##_store(struct config_item *item, const char *page, \
ret = nullb_device_##TYPE##_attr_store(&new_value, page, count);\
if (ret < 0) \
return ret; \
guard(mutex)(&lock); \
if (apply_fn) \
ret = apply_fn(dev, new_value); \
else if (test_bit(NULLB_DEV_FL_CONFIGURED, &dev->flags)) \
@@ -379,9 +389,20 @@ static int nullb_update_nr_hw_queues(struct nullb_device *dev,
struct blk_mq_tag_set *set;
int ret, nr_hw_queues;
lockdep_assert_held(&lock);
if (!dev->nullb)
return 0;
/*
* A shared tag_set is mapped via the module-wide queue counts, so a
* per-device resize is meaningless. On shrink it would also leave
* mq_map[] pointing at NULLed hctx slots, causing a NULL deref in
* blk_mq_map_swqueue(). Reject it.
*/
if (dev->shared_tags)
return -EINVAL;
/*
* Make sure at least one submit queue exists.
*/
@@ -421,25 +442,13 @@ static int nullb_update_nr_hw_queues(struct nullb_device *dev,
static int nullb_apply_submit_queues(struct nullb_device *dev,
unsigned int submit_queues)
{
int ret;
mutex_lock(&lock);
ret = nullb_update_nr_hw_queues(dev, submit_queues, dev->poll_queues);
mutex_unlock(&lock);
return ret;
return nullb_update_nr_hw_queues(dev, submit_queues, dev->poll_queues);
}
static int nullb_apply_poll_queues(struct nullb_device *dev,
unsigned int poll_queues)
{
int ret;
mutex_lock(&lock);
ret = nullb_update_nr_hw_queues(dev, dev->submit_queues, poll_queues);
mutex_unlock(&lock);
return ret;
return nullb_update_nr_hw_queues(dev, dev->submit_queues, poll_queues);
}
NULLB_DEVICE_ATTR(size, ulong, NULL);
@@ -478,6 +487,7 @@ NULLB_DEVICE_ATTR(badblocks_partial_io, bool, NULL);
static ssize_t nullb_device_power_show(struct config_item *item, char *page)
{
guard(mutex)(&lock);
return nullb_device_bool_attr_show(to_nullb_device(item)->power, page);
}
@@ -493,15 +503,15 @@ static ssize_t nullb_device_power_store(struct config_item *item,
return ret;
ret = count;
mutex_lock(&lock);
guard(mutex)(&lock);
if (!dev->power && newp) {
if (test_and_set_bit(NULLB_DEV_FL_UP, &dev->flags))
goto out;
return ret;
ret = null_add_dev(dev);
if (ret) {
clear_bit(NULLB_DEV_FL_UP, &dev->flags);
goto out;
return ret;
}
set_bit(NULLB_DEV_FL_CONFIGURED, &dev->flags);
@@ -515,8 +525,6 @@ static ssize_t nullb_device_power_store(struct config_item *item,
clear_bit(NULLB_DEV_FL_CONFIGURED, &dev->flags);
}
out:
mutex_unlock(&lock);
return ret;
}
@@ -580,6 +588,7 @@ static ssize_t nullb_device_zone_readonly_store(struct config_item *item,
{
struct nullb_device *dev = to_nullb_device(item);
guard(mutex)(&lock);
return zone_cond_store(dev, page, count, BLK_ZONE_COND_READONLY);
}
CONFIGFS_ATTR_WO(nullb_device_, zone_readonly);
@@ -589,6 +598,7 @@ static ssize_t nullb_device_zone_offline_store(struct config_item *item,
{
struct nullb_device *dev = to_nullb_device(item);
guard(mutex)(&lock);
return zone_cond_store(dev, page, count, BLK_ZONE_COND_OFFLINE);
}
CONFIGFS_ATTR_WO(nullb_device_, zone_offline);
@@ -707,10 +717,9 @@ nullb_group_drop_item(struct config_group *group, struct config_item *item)
struct nullb_device *dev = to_nullb_device(item);
if (test_and_clear_bit(NULLB_DEV_FL_UP, &dev->flags)) {
mutex_lock(&lock);
guard(mutex)(&lock);
dev->power = false;
null_del_dev(dev->nullb);
mutex_unlock(&lock);
}
nullb_del_fault_config(dev);
config_item_put(item);
@@ -836,7 +845,6 @@ static void null_free_dev(struct nullb_device *dev)
if (!dev)
return;
null_free_zoned_dev(dev);
badblocks_exit(&dev->badblocks);
kfree(dev);
}
@@ -1770,18 +1778,19 @@ static void null_del_dev(struct nullb *nullb)
del_gendisk(nullb->disk);
if (test_bit(NULLB_DEV_FL_THROTTLED, &nullb->dev->flags)) {
if (test_bit(NULLB_DEV_FL_THROTTLED, &dev->flags)) {
hrtimer_cancel(&nullb->bw_timer);
atomic_long_set(&nullb->cur_bytes, LONG_MAX);
blk_mq_start_stopped_hw_queues(nullb->q, true);
}
put_disk(nullb->disk);
null_free_zoned_dev(dev);
if (nullb->tag_set == &nullb->__tag_set)
blk_mq_free_tag_set(nullb->tag_set);
kfree(nullb->queues);
if (null_cache_active(nullb))
null_free_device_storage(nullb->dev, true);
null_free_device_storage(dev, true);
kfree(nullb);
dev->nullb = NULL;
}
@@ -2081,14 +2090,13 @@ static struct nullb *null_find_dev_by_name(const char *name)
{
struct nullb *nullb = NULL, *nb;
mutex_lock(&lock);
guard(mutex)(&lock);
list_for_each_entry(nb, &nullb_list, list) {
if (strcmp(nb->disk_name, name) == 0) {
nullb = nb;
break;
}
}
mutex_unlock(&lock);
return nullb;
}
@@ -2102,9 +2110,9 @@ static int null_create_dev(void)
if (!dev)
return -ENOMEM;
mutex_lock(&lock);
ret = null_add_dev(dev);
mutex_unlock(&lock);
scoped_guard(mutex, &lock) {
ret = null_add_dev(dev);
}
if (ret) {
null_free_dev(dev);
return ret;
@@ -2162,17 +2170,9 @@ static int __init null_init(void)
config_group_init(&nullb_subsys.su_group);
mutex_init(&nullb_subsys.su_mutex);
ret = configfs_register_subsystem(&nullb_subsys);
if (ret)
return ret;
mutex_init(&lock);
null_major = register_blkdev(0, "nullb");
if (null_major < 0) {
ret = null_major;
goto err_conf;
}
if (null_major < 0)
return null_major;
for (i = 0; i < nr_devices; i++) {
ret = null_create_dev();
@@ -2180,6 +2180,10 @@ static int __init null_init(void)
goto err_dev;
}
ret = configfs_register_subsystem(&nullb_subsys);
if (ret)
goto err_dev;
pr_info("module loaded\n");
return 0;
@@ -2189,8 +2193,8 @@ err_dev:
null_destroy_dev(nullb);
}
unregister_blkdev(null_major, "nullb");
err_conf:
configfs_unregister_subsystem(&nullb_subsys);
if (tag_set.ops)
blk_mq_free_tag_set(&tag_set);
return ret;
}
@@ -2200,19 +2204,17 @@ static void __exit null_exit(void)
configfs_unregister_subsystem(&nullb_subsys);
unregister_blkdev(null_major, "nullb");
mutex_lock(&lock);
while (!list_empty(&nullb_list)) {
nullb = list_entry(nullb_list.next, struct nullb, list);
null_destroy_dev(nullb);
scoped_guard(mutex, &lock) {
while (!list_empty(&nullb_list)) {
nullb = list_entry(nullb_list.next, struct nullb, list);
null_destroy_dev(nullb);
}
}
mutex_unlock(&lock);
unregister_blkdev(null_major, "nullb");
if (tag_set.ops)
blk_mq_free_tag_set(&tag_set);
mutex_destroy(&lock);
}
module_init(null_init);
drivers/block/null_blk/zoned.c
+8 -2
View File
@@ -18,6 +18,8 @@ static inline sector_t mb_to_sects(unsigned long mb)
static inline unsigned int null_zone_no(struct nullb_device *dev, sector_t sect)
{
if (WARN_ON_ONCE(!dev->zone_size_sects))
return 0;
return sect >> ilog2(dev->zone_size_sects);
}
@@ -56,8 +58,8 @@ int null_init_zoned_dev(struct nullb_device *dev,
sector_t sector = 0;
unsigned int i;
if (!is_power_of_2(dev->zone_size)) {
pr_err("zone_size must be power-of-two\n");
if (!dev->zone_size || !is_power_of_2(dev->zone_size)) {
pr_err("zone_size must be non-zero power-of-two\n");
return -EINVAL;
}
if (dev->zone_size > dev->size) {
@@ -88,6 +90,10 @@ int null_init_zoned_dev(struct nullb_device *dev,
zone_capacity_sects = mb_to_sects(dev->zone_capacity);
dev_capacity_sects = mb_to_sects(dev->size);
dev->zone_size_sects = mb_to_sects(dev->zone_size);
if (!dev->zone_size_sects) {
pr_err("zone_size too large or too small, leads to zero sectors\n");
return -EINVAL;
}
dev->nr_zones = round_up(dev_capacity_sects, dev->zone_size_sects)
>> ilog2(dev->zone_size_sects);
drivers/block/swim3.c
+1
View File
@@ -1290,4 +1290,5 @@ module_init(swim3_init)
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Paul Mackerras");
MODULE_DESCRIPTION("SWIM3 floppy driver for PowerMacs");
MODULE_ALIAS_BLOCKDEV_MAJOR(FLOPPY_MAJOR);
drivers/block/ublk_drv.c
+196 -182
View File
@@ -89,7 +89,8 @@
| UBLK_F_SAFE_STOP_DEV \
| UBLK_F_BATCH_IO \
| UBLK_F_NO_AUTO_PART_SCAN \
| UBLK_F_SHMEM_ZC)
| UBLK_F_SHMEM_ZC \
| UBLK_F_IO_DESC_SIZE)
#define UBLK_F_ALL_RECOVERY_FLAGS (UBLK_F_USER_RECOVERY \
| UBLK_F_USER_RECOVERY_REISSUE \
@@ -107,6 +108,8 @@
UBLK_BATCH_F_HAS_BUF_ADDR | \
UBLK_BATCH_F_AUTO_BUF_REG_FALLBACK)
#define UBLK_MAX_IO_DESC_SIZE 256
/* ublk batch fetch uring_cmd */
struct ublk_batch_fetch_cmd {
struct list_head node;
@@ -237,8 +240,9 @@ struct ublk_io {
} ____cacheline_aligned_in_smp;
struct ublk_queue {
int q_id;
int q_depth;
u16 q_id;
u16 q_depth;
u16 io_desc_size;
unsigned long flags;
struct ublksrv_io_desc *io_cmd_buf;
@@ -248,7 +252,7 @@ struct ublk_queue {
bool fail_io; /* copy of dev->state == UBLK_S_DEV_FAIL_IO */
spinlock_t cancel_lock;
struct ublk_device *dev;
u32 nr_io_ready;
u16 nr_io_ready;
/*
* For supporting UBLK_F_BATCH_IO only.
@@ -327,7 +331,7 @@ struct ublk_device {
struct ublk_params params;
u32 nr_queue_ready;
u16 nr_queue_ready;
bool unprivileged_daemons;
struct mutex cancel_mutex;
bool canceling;
@@ -403,9 +407,9 @@ static inline void ublk_io_evts_deinit(struct ublk_queue *q)
}
static inline struct ublksrv_io_desc *
ublk_get_iod(const struct ublk_queue *ubq, unsigned tag)
ublk_get_iod(const struct ublk_queue *ubq, u16 tag)
{
return &ubq->io_cmd_buf[tag];
return (void *)ubq->io_cmd_buf + tag * (size_t)ubq->io_desc_size;
}
static inline bool ublk_support_zero_copy(const struct ublk_queue *ubq)
@@ -423,8 +427,7 @@ static inline bool ublk_support_shmem_zc(const struct ublk_queue *ubq)
return ubq->flags & UBLK_F_SHMEM_ZC;
}
static inline bool ublk_iod_is_shmem_zc(const struct ublk_queue *ubq,
unsigned int tag)
static inline bool ublk_iod_is_shmem_zc(const struct ublk_queue *ubq, u16 tag)
{
return ublk_get_iod(ubq, tag)->op_flags & UBLK_IO_F_SHMEM_ZC;
}
@@ -529,7 +532,6 @@ static void ublk_init_iod(struct ublk_queue *ubq, struct request *req,
struct ublk_zoned_report_desc {
__u64 sector;
__u32 operation;
__u32 nr_zones;
};
@@ -659,7 +661,6 @@ static int ublk_report_zones(struct gendisk *disk, sector_t sector,
goto out;
}
desc.operation = UBLK_IO_OP_REPORT_ZONES;
desc.sector = sector;
desc.nr_zones = zones_in_request;
ret = ublk_zoned_insert_report_desc(req, &desc);
@@ -703,8 +704,24 @@ out:
return ret;
}
static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq,
struct request *req)
static bool ublk_validate_req_zoned(const struct request *req)
{
switch (req_op(req)) {
case REQ_OP_ZONE_OPEN:
case REQ_OP_ZONE_CLOSE:
case REQ_OP_ZONE_FINISH:
case REQ_OP_ZONE_RESET:
case REQ_OP_ZONE_APPEND:
case REQ_OP_ZONE_RESET_ALL:
return true;
case REQ_OP_DRV_IN:
return !!ublk_zoned_get_report_desc(req);
default:
return false;
}
}
static void ublk_setup_iod_zoned(struct ublk_queue *ubq, struct request *req)
{
struct ublk_zoned_report_desc *desc;
u32 ublk_op;
@@ -730,26 +747,15 @@ static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq,
break;
case REQ_OP_DRV_IN:
desc = ublk_zoned_get_report_desc(req);
if (!desc)
return BLK_STS_IOERR;
ublk_op = desc->operation;
switch (ublk_op) {
case UBLK_IO_OP_REPORT_ZONES:
ublk_init_iod(ubq, req, ublk_op, desc->nr_zones,
desc->sector);
return BLK_STS_OK;
default:
return BLK_STS_IOERR;
}
case REQ_OP_DRV_OUT:
/* We do not support drv_out */
return BLK_STS_NOTSUPP;
ublk_init_iod(ubq, req, UBLK_IO_OP_REPORT_ZONES, desc->nr_zones,
desc->sector);
return;
default:
return BLK_STS_IOERR;
WARN_ON_ONCE(1);
return;
}
ublk_init_iod(ubq, req, ublk_op, blk_rq_sectors(req), blk_rq_pos(req));
return BLK_STS_OK;
}
#else
@@ -770,10 +776,14 @@ static int ublk_revalidate_disk_zones(struct ublk_device *ub)
return 0;
}
static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq,
struct request *req)
static bool ublk_validate_req_zoned(const struct request *req)
{
return BLK_STS_NOTSUPP;
return false;
}
static void ublk_setup_iod_zoned(struct ublk_queue *ubq, struct request *req)
{
WARN_ON_ONCE(1);
}
#endif
@@ -864,7 +874,7 @@ static unsigned int unprivileged_ublks_added; /* protected by ublk_ctl_mutex */
static struct miscdevice ublk_misc;
static inline unsigned ublk_pos_to_hwq(loff_t pos)
static inline u16 ublk_pos_to_hwq(loff_t pos)
{
return ((pos - UBLKSRV_IO_BUF_OFFSET) >> UBLK_QID_OFF) &
UBLK_QID_BITS_MASK;
@@ -875,7 +885,7 @@ static inline unsigned ublk_pos_to_buf_off(loff_t pos)
return (pos - UBLKSRV_IO_BUF_OFFSET) & UBLK_IO_BUF_BITS_MASK;
}
static inline unsigned ublk_pos_to_tag(loff_t pos)
static inline u16 ublk_pos_to_tag(loff_t pos)
{
return ((pos - UBLKSRV_IO_BUF_OFFSET) >> UBLK_TAG_OFF) &
UBLK_TAG_BITS_MASK;
@@ -970,7 +980,7 @@ static int ublk_validate_params(const struct ublk_device *ub)
if (p->max_sectors < PAGE_SECTORS)
return -EINVAL;
if (ublk_dev_is_zoned(ub) && !p->chunk_sectors)
if (ublk_dev_is_zoned(ub) && !is_power_of_2(p->chunk_sectors))
return -EINVAL;
} else
return -EINVAL;
@@ -1231,30 +1241,31 @@ static noinline void ublk_put_device(struct ublk_device *ub)
}
static inline struct ublk_queue *ublk_get_queue(struct ublk_device *dev,
int qid)
u16 qid)
{
return dev->queues[qid];
}
static inline struct ublksrv_io_desc *
ublk_queue_cmd_buf(struct ublk_device *ub, int q_id)
ublk_queue_cmd_buf(struct ublk_device *ub, u16 q_id)
{
return ublk_get_queue(ub, q_id)->io_cmd_buf;
}
static inline int __ublk_queue_cmd_buf_size(int depth)
static inline size_t __ublk_queue_cmd_buf_size(const struct ublk_device *ub,
u16 depth)
{
return round_up(depth * sizeof(struct ublksrv_io_desc), PAGE_SIZE);
return round_up(depth * (size_t)ub->dev_info.io_desc_size, PAGE_SIZE);
}
static inline int ublk_queue_cmd_buf_size(struct ublk_device *ub)
static inline size_t ublk_queue_cmd_buf_size(const struct ublk_device *ub)
{
return __ublk_queue_cmd_buf_size(ub->dev_info.queue_depth);
return __ublk_queue_cmd_buf_size(ub, ub->dev_info.queue_depth);
}
static int ublk_max_cmd_buf_size(void)
static size_t ublk_max_cmd_buf_size(const struct ublk_device *ub)
{
return __ublk_queue_cmd_buf_size(UBLK_MAX_QUEUE_DEPTH);
return __ublk_queue_cmd_buf_size(ub, UBLK_MAX_QUEUE_DEPTH);
}
/*
@@ -1457,52 +1468,47 @@ static inline bool ublk_need_unmap_req(const struct request *req)
(req_op(req) == REQ_OP_READ || req_op(req) == REQ_OP_DRV_IN);
}
static unsigned int ublk_map_io(const struct ublk_queue *ubq,
const struct request *req,
static unsigned int ublk_map_io(const struct request *req,
const struct ublk_io *io)
{
const unsigned int rq_bytes = blk_rq_bytes(req);
struct iov_iter iter;
const int dir = ITER_DEST;
if (!ublk_need_map_io(ubq))
return rq_bytes;
if (import_ubuf(dir, u64_to_user_ptr(io->buf.addr), blk_rq_bytes(req),
&iter) < 0)
return 0;
/*
* no zero copy, we delay copy WRITE request data into ublksrv
* context and the big benefit is that pinning pages in current
* context is pretty fast, see ublk_pin_user_pages
*/
if (ublk_need_map_req(req)) {
struct iov_iter iter;
const int dir = ITER_DEST;
import_ubuf(dir, u64_to_user_ptr(io->buf.addr), rq_bytes, &iter);
return ublk_copy_user_pages(req, 0, &iter, dir);
}
return rq_bytes;
return ublk_copy_user_pages(req, 0, &iter, dir);
}
static unsigned int ublk_unmap_io(bool need_map,
const struct request *req,
static unsigned int ublk_unmap_io(const struct request *req,
const struct ublk_io *io)
{
const unsigned int rq_bytes = blk_rq_bytes(req);
struct iov_iter iter;
const int dir = ITER_SOURCE;
if (!need_map)
return rq_bytes;
if (import_ubuf(dir, u64_to_user_ptr(io->buf.addr), io->res, &iter) < 0)
return 0;
if (ublk_need_unmap_req(req)) {
struct iov_iter iter;
const int dir = ITER_SOURCE;
WARN_ON_ONCE(io->res > rq_bytes);
import_ubuf(dir, u64_to_user_ptr(io->buf.addr), io->res, &iter);
return ublk_copy_user_pages(req, 0, &iter, dir);
}
return rq_bytes;
return ublk_copy_user_pages(req, 0, &iter, dir);
}
static blk_status_t ublk_setup_iod(struct ublk_queue *ubq, struct request *req)
static bool ublk_validate_req(const struct ublk_queue *ubq,
const struct request *req)
{
switch (req_op(req)) {
case REQ_OP_READ:
case REQ_OP_WRITE:
case REQ_OP_FLUSH:
case REQ_OP_DISCARD:
case REQ_OP_WRITE_ZEROES:
return true;
default:
return ublk_queue_is_zoned(ubq) && ublk_validate_req_zoned(req);
}
}
static void ublk_setup_iod(struct ublk_queue *ubq, struct request *req)
{
u32 ublk_op;
@@ -1523,13 +1529,11 @@ static blk_status_t ublk_setup_iod(struct ublk_queue *ubq, struct request *req)
ublk_op = UBLK_IO_OP_WRITE_ZEROES;
break;
default:
if (ublk_queue_is_zoned(ubq))
return ublk_setup_iod_zoned(ubq, req);
return BLK_STS_IOERR;
ublk_setup_iod_zoned(ubq, req);
return;
}
ublk_init_iod(ubq, req, ublk_op, blk_rq_sectors(req), blk_rq_pos(req));
return BLK_STS_OK;
}
static inline struct ublk_uring_cmd_pdu *ublk_get_uring_cmd_pdu(
@@ -1562,30 +1566,27 @@ static inline void __ublk_complete_rq(struct request *req, struct ublk_io *io,
goto exit;
}
/*
* FLUSH, DISCARD or WRITE_ZEROES usually won't return bytes returned, so end them
* directly.
*
* Both the two needn't unmap.
*/
if (req_op(req) != REQ_OP_READ && req_op(req) != REQ_OP_WRITE &&
req_op(req) != REQ_OP_DRV_IN)
goto exit;
/* shmem zero copy: no data to unmap, pages already shared */
if (ublk_iod_is_shmem_zc(req->mq_hctx->driver_data, req->tag))
if (!need_map || !ublk_need_unmap_req(req) ||
ublk_iod_is_shmem_zc(req->mq_hctx->driver_data, req->tag))
goto exit;
/* for READ request, writing data in iod->addr to rq buffers */
unmapped_bytes = ublk_unmap_io(need_map, req, io);
unmapped_bytes = ublk_unmap_io(req, io);
/*
* Extremely impossible since we got data filled in just before
*
* Re-read simply for this unlikely case.
*/
if (unlikely(unmapped_bytes < io->res))
if (unlikely(unmapped_bytes < io->res)) {
if (unlikely(!unmapped_bytes)) {
res = BLK_STS_IOERR;
goto exit;
}
io->res = unmapped_bytes;
}
/*
* Run bio->bi_end_io() with softirqs disabled. If the final fput
@@ -1657,7 +1658,7 @@ static inline void __ublk_abort_rq(struct ublk_queue *ubq,
}
static void
ublk_auto_buf_reg_fallback(const struct ublk_queue *ubq, unsigned tag)
ublk_auto_buf_reg_fallback(const struct ublk_queue *ubq, u16 tag)
{
struct ublksrv_io_desc *iod = ublk_get_iod(ubq, tag);
@@ -1737,10 +1738,11 @@ static bool ublk_start_io(const struct ublk_queue *ubq, struct request *req,
unsigned mapped_bytes;
/* shmem zero copy: skip data copy, pages already shared */
if (ublk_iod_is_shmem_zc(ubq, req->tag))
if (!ublk_need_map_io(ubq) || !ublk_need_map_req(req) ||
ublk_iod_is_shmem_zc(ubq, req->tag))
return true;
mapped_bytes = ublk_map_io(ubq, req, io);
mapped_bytes = ublk_map_io(req, io);
/* partially mapped, update io descriptor */
if (unlikely(mapped_bytes != blk_rq_bytes(req))) {
@@ -1768,9 +1770,10 @@ static bool ublk_start_io(const struct ublk_queue *ubq, struct request *req,
static void ublk_dispatch_req(struct ublk_queue *ubq, struct request *req)
{
unsigned int issue_flags = IO_URING_CMD_TASK_WORK_ISSUE_FLAGS;
int tag = req->tag;
u16 tag = req->tag;
struct ublk_io *io = &ubq->ios[tag];
ublk_setup_iod(ubq, req);
pr_devel("%s: complete: qid %d tag %d io_flags %x addr %llx\n",
__func__, ubq->q_id, req->tag, io->flags,
ublk_get_iod(ubq, req->tag)->addr);
@@ -1824,6 +1827,7 @@ static bool __ublk_batch_prep_dispatch(struct ublk_queue *ubq,
enum auto_buf_reg_res res = AUTO_BUF_REG_FALLBACK;
struct io_uring_cmd *cmd = data->cmd;
ublk_setup_iod(ubq, req);
if (!ublk_start_io(ubq, req, io))
return false;
@@ -2137,8 +2141,6 @@ static enum blk_eh_timer_return ublk_timeout(struct request *rq)
static blk_status_t ublk_prep_req(struct ublk_queue *ubq, struct request *rq,
bool check_cancel)
{
blk_status_t res;
if (unlikely(READ_ONCE(ubq->fail_io)))
return BLK_STS_TARGET;
@@ -2159,8 +2161,7 @@ static blk_status_t ublk_prep_req(struct ublk_queue *ubq, struct request *rq,
return BLK_STS_IOERR;
/* fill iod to slot in io cmd buffer */
res = ublk_setup_iod(ubq, rq);
if (unlikely(res != BLK_STS_OK))
if (unlikely(!ublk_validate_req(ubq, rq)))
return BLK_STS_IOERR;
blk_mq_start_request(rq);
@@ -2358,7 +2359,7 @@ static const struct blk_mq_ops ublk_batch_mq_ops = {
static void ublk_queue_reinit(struct ublk_device *ub, struct ublk_queue *ubq)
{
int i;
u16 i;
ubq->nr_io_ready = 0;
@@ -2403,7 +2404,7 @@ static int ublk_ch_open(struct inode *inode, struct file *filp)
static void ublk_reset_ch_dev(struct ublk_device *ub)
{
int i;
u16 i;
for (i = 0; i < ub->dev_info.nr_hw_queues; i++) {
struct ublk_queue *ubq = ublk_get_queue(ub, i);
@@ -2475,7 +2476,7 @@ out:
static void ublk_set_canceling(struct ublk_device *ub, bool canceling)
__must_hold(&ub->cancel_mutex)
{
int i;
u16 i;
ub->canceling = canceling;
for (i = 0; i < ub->dev_info.nr_hw_queues; i++)
@@ -2484,7 +2485,7 @@ static void ublk_set_canceling(struct ublk_device *ub, bool canceling)
static bool ublk_check_and_reset_active_ref(struct ublk_device *ub)
{
int i, j;
u16 i, j;
if (!ublk_dev_need_req_ref(ub))
return false;
@@ -2517,7 +2518,7 @@ static void ublk_ch_release_work_fn(struct work_struct *work)
struct ublk_device *ub =
container_of(work, struct ublk_device, exit_work.work);
struct gendisk *disk;
int i;
u16 i;
/*
* For zero-copy and auto buffer register modes, I/O references
@@ -2634,9 +2635,10 @@ static int ublk_ch_mmap(struct file *filp, struct vm_area_struct *vma)
{
struct ublk_device *ub = filp->private_data;
size_t sz = vma->vm_end - vma->vm_start;
unsigned max_sz = ublk_max_cmd_buf_size();
size_t max_sz = ublk_max_cmd_buf_size(ub);
unsigned long pfn, end, phys_off = vma->vm_pgoff << PAGE_SHIFT;
int q_id, ret = 0;
int ret = 0;
u16 q_id;
spin_lock(&ub->lock);
if (!ub->mm)
@@ -2709,7 +2711,7 @@ static void ublk_abort_batch_queue(struct ublk_device *ub,
*/
static void ublk_abort_queue(struct ublk_device *ub, struct ublk_queue *ubq)
{
int i;
u16 i;
for (i = 0; i < ubq->q_depth; i++) {
struct ublk_io *io = &ubq->ios[i];
@@ -2752,7 +2754,7 @@ out:
ublk_put_disk(disk);
}
static void ublk_cancel_cmd(struct ublk_queue *ubq, unsigned tag,
static void ublk_cancel_cmd(struct ublk_queue *ubq, u16 tag,
unsigned int issue_flags)
{
struct ublk_io *io = &ubq->ios[tag];
@@ -2903,7 +2905,7 @@ static inline bool ublk_dev_ready(const struct ublk_device *ub)
static void ublk_cancel_queue(struct ublk_queue *ubq)
{
int i;
u16 i;
if (ublk_support_batch_io(ubq)) {
ublk_batch_cancel_queue(ubq);
@@ -2917,7 +2919,7 @@ static void ublk_cancel_queue(struct ublk_queue *ubq)
/* Cancel all pending commands, must be called after del_gendisk() returns */
static void ublk_cancel_dev(struct ublk_device *ub)
{
int i;
u16 i;
for (i = 0; i < ub->dev_info.nr_hw_queues; i++)
ublk_cancel_queue(ublk_get_queue(ub, i));
@@ -2951,7 +2953,7 @@ static void ublk_wait_tagset_rqs_idle(struct ublk_device *ub)
static void ublk_force_abort_dev(struct ublk_device *ub)
{
int i;
u16 i;
pr_devel("%s: force abort ub: dev_id %d state %s\n",
__func__, ub->dev_info.dev_id,
@@ -3075,18 +3077,19 @@ static inline int ublk_check_cmd_op(u32 cmd_op)
return 0;
}
static inline int ublk_set_auto_buf_reg(struct ublk_io *io, struct io_uring_cmd *cmd)
/* Must run before ublk_fill_io_cmd() / __ublk_fetch(). */
static inline int ublk_validate_io_buf(const struct ublk_device *ub,
struct io_uring_cmd *cmd,
struct ublk_auto_buf_reg *buf)
{
struct ublk_auto_buf_reg buf;
if (!ublk_dev_support_auto_buf_reg(ub))
return 0;
buf = ublk_sqe_addr_to_auto_buf_reg(READ_ONCE(cmd->sqe->addr));
if (buf.reserved0 || buf.reserved1)
*buf = ublk_sqe_addr_to_auto_buf_reg(READ_ONCE(cmd->sqe->addr));
if (buf->reserved0 || buf->reserved1)
return -EINVAL;
if (buf.flags & ~UBLK_AUTO_BUF_REG_F_MASK)
if (buf->flags & ~UBLK_AUTO_BUF_REG_F_MASK)
return -EINVAL;
io->buf.auto_reg = buf;
return 0;
}
@@ -3107,17 +3110,25 @@ static void ublk_clear_auto_buf_reg(struct ublk_io *io,
* responsibility for unregistering the buffer, otherwise
* this ublk request gets stuck.
*/
if (io->buf_ctx_handle == io_uring_cmd_ctx_handle(cmd))
if (buf_idx &&
io->buf_ctx_handle == io_uring_cmd_ctx_handle(cmd))
*buf_idx = io->buf.auto_reg.index;
}
}
static int ublk_handle_auto_buf_reg(struct ublk_io *io,
struct io_uring_cmd *cmd,
u16 *buf_idx)
static inline void ublk_apply_io_buf(const struct ublk_device *ub,
struct ublk_io *io,
struct io_uring_cmd *cmd,
unsigned long buf_addr,
const struct ublk_auto_buf_reg *auto_buf,
u16 *buf_idx)
{
ublk_clear_auto_buf_reg(io, cmd, buf_idx);
return ublk_set_auto_buf_reg(io, cmd);
if (ublk_dev_support_auto_buf_reg(ub)) {
ublk_clear_auto_buf_reg(io, cmd, buf_idx);
io->buf.auto_reg = *auto_buf;
} else {
io->buf.addr = buf_addr;
}
}
/* Once we return, `io->req` can't be used any more */
@@ -3134,21 +3145,9 @@ ublk_fill_io_cmd(struct ublk_io *io, struct io_uring_cmd *cmd)
return req;
}
static inline int
ublk_config_io_buf(const struct ublk_device *ub, struct ublk_io *io,
struct io_uring_cmd *cmd, unsigned long buf_addr,
u16 *buf_idx)
{
if (ublk_dev_support_auto_buf_reg(ub))
return ublk_handle_auto_buf_reg(io, cmd, buf_idx);
io->buf.addr = buf_addr;
return 0;
}
static inline void ublk_prep_cancel(struct io_uring_cmd *cmd,
unsigned int issue_flags,
struct ublk_queue *ubq, unsigned int tag)
struct ublk_queue *ubq, u16 tag)
{
struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd);
@@ -3286,6 +3285,7 @@ static int __ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
struct ublk_io *io, __u64 buf_addr, u16 q_id)
{
struct ublk_auto_buf_reg auto_buf;
int ret;
/*
@@ -3294,11 +3294,13 @@ static int ublk_fetch(struct io_uring_cmd *cmd, struct ublk_device *ub,
* FETCH, so it is fine even for IO_URING_F_NONBLOCK.
*/
mutex_lock(&ub->mutex);
ret = __ublk_fetch(cmd, ub, io, q_id);
if (!ret)
ret = ublk_config_io_buf(ub, io, cmd, buf_addr, NULL);
ret = ublk_validate_io_buf(ub, cmd, &auto_buf);
if (!ret)
ret = __ublk_fetch(cmd, ub, io, q_id);
if (!ret) {
ublk_apply_io_buf(ub, io, cmd, buf_addr, &auto_buf, NULL);
ublk_mark_io_ready(ub, q_id, io);
}
mutex_unlock(&ub->mutex);
return ret;
}
@@ -3441,13 +3443,18 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
case UBLK_IO_REGISTER_IO_BUF:
return ublk_daemon_register_io_buf(cmd, ub, q_id, tag, io, addr,
issue_flags);
case UBLK_IO_COMMIT_AND_FETCH_REQ:
case UBLK_IO_COMMIT_AND_FETCH_REQ: {
struct ublk_auto_buf_reg auto_buf;
ret = ublk_check_commit_and_fetch(ub, io, addr);
if (ret)
goto out;
ret = ublk_validate_io_buf(ub, cmd, &auto_buf);
if (ret)
goto out;
io->res = result;
req = ublk_fill_io_cmd(io, cmd);
ret = ublk_config_io_buf(ub, io, cmd, addr, &buf_idx);
ublk_apply_io_buf(ub, io, cmd, addr, &auto_buf, &buf_idx);
if (buf_idx != UBLK_INVALID_BUF_IDX)
io_buffer_unregister_bvec(cmd, buf_idx, issue_flags);
compl = ublk_need_complete_req(ub, io);
@@ -3456,10 +3463,8 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
req->__sector = addr;
if (compl)
__ublk_complete_rq(req, io, ublk_dev_need_map_io(ub), NULL);
if (ret)
goto out;
break;
}
case UBLK_IO_NEED_GET_DATA:
/*
* ublk_get_data() may fail and fallback to requeue, so keep
@@ -3467,8 +3472,7 @@ static int ublk_ch_uring_cmd_local(struct io_uring_cmd *cmd,
* request
*/
req = ublk_fill_io_cmd(io, cmd);
ret = ublk_config_io_buf(ub, io, cmd, addr, NULL);
WARN_ON_ONCE(ret);
io->buf.addr = addr;
if (likely(ublk_get_data(ubq, io, req))) {
__ublk_prep_compl_io_cmd(io, req);
return UBLK_IO_RES_OK;
@@ -3780,11 +3784,11 @@ static int ublk_batch_commit_io(struct ublk_queue *ubq,
ret = ublk_batch_commit_io_check(ubq, io, &buf);
if (!ret) {
io->res = elem->result;
io->buf = buf;
req = ublk_fill_io_cmd(io, data->cmd);
if (auto_reg)
ublk_clear_auto_buf_reg(io, data->cmd, &buf_idx);
io->buf = buf;
compl = ublk_need_complete_req(data->ub, io);
}
ublk_io_unlock(io);
@@ -3952,8 +3956,8 @@ static int ublk_handle_non_batch_cmd(struct io_uring_cmd *cmd,
const struct ublksrv_io_cmd *ub_cmd = io_uring_sqe_cmd(cmd->sqe,
struct ublksrv_io_cmd);
struct ublk_device *ub = cmd->file->private_data;
unsigned tag = READ_ONCE(ub_cmd->tag);
unsigned q_id = READ_ONCE(ub_cmd->q_id);
u16 tag = READ_ONCE(ub_cmd->tag);
u16 q_id = READ_ONCE(ub_cmd->q_id);
unsigned index = READ_ONCE(ub_cmd->addr);
struct ublk_queue *ubq;
struct ublk_io *io;
@@ -4159,7 +4163,8 @@ static const struct file_operations ublk_ch_batch_io_fops = {
static void __ublk_deinit_queue(struct ublk_device *ub, struct ublk_queue *ubq)
{
int size, i;
size_t size;
u16 i;
size = ublk_queue_cmd_buf_size(ub);
@@ -4180,7 +4185,7 @@ static void __ublk_deinit_queue(struct ublk_device *ub, struct ublk_queue *ubq)
kvfree(ubq);
}
static void ublk_deinit_queue(struct ublk_device *ub, int q_id)
static void ublk_deinit_queue(struct ublk_device *ub, u16 q_id)
{
struct ublk_queue *ubq = ub->queues[q_id];
@@ -4191,7 +4196,7 @@ static void ublk_deinit_queue(struct ublk_device *ub, int q_id)
ub->queues[q_id] = NULL;
}
static int ublk_get_queue_numa_node(struct ublk_device *ub, int q_id)
static int ublk_get_queue_numa_node(struct ublk_device *ub, u16 q_id)
{
unsigned int cpu;
@@ -4204,14 +4209,16 @@ static int ublk_get_queue_numa_node(struct ublk_device *ub, int q_id)
return NUMA_NO_NODE;
}
static int ublk_init_queue(struct ublk_device *ub, int q_id)
static int ublk_init_queue(struct ublk_device *ub, u16 q_id)
{
int depth = ub->dev_info.queue_depth;
u16 depth = ub->dev_info.queue_depth;
gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO;
struct ublk_queue *ubq;
struct page *page;
int numa_node;
int size, i, ret;
size_t size;
int ret;
u16 i;
/* Determine NUMA node based on queue's CPU affinity */
numa_node = ublk_get_queue_numa_node(ub, q_id);
@@ -4235,6 +4242,7 @@ static int ublk_init_queue(struct ublk_device *ub, int q_id)
return -ENOMEM;
}
ubq->io_cmd_buf = page_address(page);
ubq->io_desc_size = ub->dev_info.io_desc_size;
for (i = 0; i < ubq->q_depth; i++)
spin_lock_init(&ubq->ios[i].lock);
@@ -4256,7 +4264,7 @@ fail:
static void ublk_deinit_queues(struct ublk_device *ub)
{
int i;
u16 i;
for (i = 0; i < ub->dev_info.nr_hw_queues; i++)
ublk_deinit_queue(ub, i);
@@ -4264,7 +4272,8 @@ static void ublk_deinit_queues(struct ublk_device *ub)
static int ublk_init_queues(struct ublk_device *ub)
{
int i, ret;
int ret;
u16 i;
for (i = 0; i < ub->dev_info.nr_hw_queues; i++) {
ret = ublk_init_queue(ub, i);
@@ -4718,6 +4727,15 @@ static int ublk_ctrl_add_dev(const struct ublksrv_ctrl_cmd *header)
if (info.flags & UBLK_F_INTEGRITY && !(info.flags & UBLK_F_USER_COPY))
return -EINVAL;
if (info.flags & UBLK_F_IO_DESC_SIZE) {
if (info.io_desc_size < sizeof(struct ublksrv_io_desc) ||
info.io_desc_size % _Alignof(struct ublksrv_io_desc) ||
info.io_desc_size > UBLK_MAX_IO_DESC_SIZE)
return -EINVAL;
} else {
info.io_desc_size = sizeof(struct ublksrv_io_desc);
}
/* the created device is always owned by current user */
ublk_store_owner_uid_gid(&info.owner_uid, &info.owner_gid);
@@ -5171,7 +5189,7 @@ out:
struct count_busy {
const struct ublk_queue *ubq;
unsigned int nr_busy;
u16 nr_busy;
};
static bool ublk_count_busy_req(struct request *rq, void *data)
@@ -5209,8 +5227,7 @@ static int ublk_wait_for_idle_io(struct ublk_device *ub,
return 0;
while (elapsed < timeout_ms && !signal_pending(current)) {
unsigned int queues_cancelable = 0;
int i;
u16 i, queues_cancelable = 0;
for (i = 0; i < ub->dev_info.nr_hw_queues; i++) {
struct ublk_queue *ubq = ublk_get_queue(ub, i);
@@ -5502,39 +5519,36 @@ static void ublk_unpin_range_pages(unsigned long base_pfn,
/*
* Inner loop: erase up to UBLK_REMOVE_BATCH matching ranges under
* mas_lock, collecting them into an xarray. Then drop the lock and
* unpin pages + free ranges outside spinlock context.
* mas_lock, collecting the page ranges in a fixed-size array. Then
* drop the lock and unpin pages + free ranges outside spinlock context.
*
* Returns true if the tree walk completed, false if more ranges remain.
* Xarray key is the base PFN, value encodes nr_pages via xa_mk_value().
*/
#define UBLK_REMOVE_BATCH 64
struct ublk_unpin_range {
unsigned long base_pfn;
unsigned long nr_pages;
};
static bool __ublk_shmem_remove_ranges(struct ublk_device *ub,
int buf_index, int *ret)
{
MA_STATE(mas, &ub->buf_tree, 0, ULONG_MAX);
struct ublk_buf_range *range;
struct xarray to_unpin;
unsigned long idx;
struct ublk_unpin_range to_unpin[UBLK_REMOVE_BATCH];
unsigned int count = 0;
unsigned int i;
bool done = false;
void *entry;
xa_init(&to_unpin);
mas_lock(&mas);
mas_for_each(&mas, range, ULONG_MAX) {
unsigned long nr;
if (buf_index >= 0 && range->buf_index != buf_index)
continue;
*ret = 0;
nr = mas.last - mas.index + 1;
if (xa_err(xa_store(&to_unpin, mas.index,
xa_mk_value(nr), GFP_ATOMIC)))
goto unlock;
to_unpin[count].base_pfn = mas.index;
to_unpin[count].nr_pages = mas.last - mas.index + 1;
mas_erase(&mas);
kfree(range);
if (++count >= UBLK_REMOVE_BATCH)
@@ -5544,9 +5558,9 @@ static bool __ublk_shmem_remove_ranges(struct ublk_device *ub,
unlock:
mas_unlock(&mas);
xa_for_each(&to_unpin, idx, entry)
ublk_unpin_range_pages(idx, xa_to_value(entry));
xa_destroy(&to_unpin);
for (i = 0; i < count; i++)
ublk_unpin_range_pages(to_unpin[i].base_pfn,
to_unpin[i].nr_pages);
return done;
}
drivers/block/zloop.c
+29 -13
View File
@@ -144,6 +144,7 @@ struct zloop_device {
unsigned int nr_conv_zones;
unsigned int max_open_zones;
unsigned int block_size;
unsigned int dio_mem_align;
spinlock_t open_zones_lock;
struct list_head open_zones_lru_list;
@@ -478,7 +479,8 @@ static int zloop_finish_zone(struct zloop_device *zlo, unsigned int zone_no)
zone->cond == BLK_ZONE_COND_FULL)
goto unlock;
if (vfs_truncate(&zone->file->f_path, zlo->zone_size << SECTOR_SHIFT)) {
if (vfs_truncate(&zone->file->f_path,
zlo->zone_capacity << SECTOR_SHIFT)) {
set_bit(ZLOOP_ZONE_SEQ_ERROR, &zone->flags);
ret = -EIO;
goto unlock;
@@ -553,7 +555,7 @@ static int zloop_do_rw(struct zloop_cmd *cmd)
iov_iter_bvec(&iter, rw,
__bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter),
nr_bvec, blk_rq_bytes(rq));
iter.iov_offset = rq->bio->bi_iter.bi_bvec_done;
iter.iov_offset = rq->bio->bi_iter.bi_offset;
}
cmd->iocb.ki_pos = (cmd->sector - zone->start) << SECTOR_SHIFT;
@@ -1037,20 +1039,30 @@ static int zloop_get_block_size(struct zloop_device *zlo,
struct kstat st;
/*
* If the FS block size is lower than or equal to 4K, use that as the
* device block size. Otherwise, fallback to the FS direct IO alignment
* constraint if that is provided, and to the FS underlying device
* physical block size if the direct IO alignment is unknown.
* Use the dio alignment of the file system if provided. The incoming
* request's bio_vec is forwarded to the backing file unchanged, so its
* required memory alignment becomes the device's dma_alignment when
* used for direct-io.
*/
if (!vfs_getattr(&zone->file->f_path, &st, STATX_DIOALIGN, 0) &&
(st.result_mask & STATX_DIOALIGN)) {
zlo->block_size = st.dio_offset_align;
zlo->dio_mem_align = st.dio_mem_align - 1;
} else if (sb_bdev) {
zlo->block_size = bdev_physical_block_size(sb_bdev);
zlo->dio_mem_align = bdev_dma_alignment(sb_bdev);
} else {
zlo->block_size = SECTOR_SIZE;
zlo->dio_mem_align = SECTOR_SIZE - 1;
}
/*
* Prefer the FS block size for the device block size when it is no
* larger than 4K; otherwise keep the direct I/O / physical block size
* selected above.
*/
if (file_inode(zone->file)->i_sb->s_blocksize <= SZ_4K)
zlo->block_size = file_inode(zone->file)->i_sb->s_blocksize;
else if (!vfs_getattr(&zone->file->f_path, &st, STATX_DIOALIGN, 0) &&
(st.result_mask & STATX_DIOALIGN))
zlo->block_size = st.dio_offset_align;
else if (sb_bdev)
zlo->block_size = bdev_physical_block_size(sb_bdev);
else
zlo->block_size = SECTOR_SIZE;
if (zlo->zone_capacity & ((zlo->block_size >> SECTOR_SHIFT) - 1)) {
pr_err("Zone capacity is not aligned to block size %u\n",
@@ -1279,6 +1291,10 @@ static int zloop_ctl_add(struct zloop_options *opts)
lim.physical_block_size = zlo->block_size;
lim.logical_block_size = zlo->block_size;
/* Direct I/O forwards the request pages to the backing files as-is. */
if (!opts->buffered_io)
lim.dma_alignment = max_t(unsigned int, zlo->dio_mem_align,
SECTOR_SIZE - 1);
if (zlo->zone_append)
lim.max_hw_zone_append_sectors = lim.max_hw_sectors;
lim.max_open_zones = zlo->max_open_zones;
drivers/md/dm-io-rewind.c
+5 -5
View File
@@ -16,12 +16,12 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec *bv,
int idx;
iter->bi_size += bytes;
if (bytes <= iter->bi_bvec_done) {
iter->bi_bvec_done -= bytes;
if (bytes <= iter->bi_offset) {
iter->bi_offset -= bytes;
return true;
}
bytes -= iter->bi_bvec_done;
bytes -= iter->bi_offset;
idx = iter->bi_idx - 1;
while (idx >= 0 && bytes && bytes > bv[idx].bv_len) {
@@ -32,13 +32,13 @@ static inline bool dm_bvec_iter_rewind(const struct bio_vec *bv,
if (WARN_ONCE(idx < 0 && bytes,
"Attempted to rewind iter beyond bvec's boundaries\n")) {
iter->bi_size -= bytes;
iter->bi_bvec_done = 0;
iter->bi_offset = 0;
iter->bi_idx = 0;
return false;
}
iter->bi_idx = idx;
iter->bi_bvec_done = bv[idx].bv_len - bytes;
iter->bi_offset = bv[idx].bv_len - bytes;
return true;
}
drivers/md/dm-pcache/segment.c
+2 -2
View File
@@ -14,7 +14,7 @@ int segment_copy_to_bio(struct pcache_segment *segment,
iov_iter_bvec(&iter, ITER_DEST, &bio->bi_io_vec[bio->bi_iter.bi_idx],
bio_segments(bio), bio->bi_iter.bi_size);
iter.iov_offset = bio->bi_iter.bi_bvec_done;
iter.iov_offset = bio->bi_iter.bi_offset;
if (bio_off)
iov_iter_advance(&iter, bio_off);
@@ -35,7 +35,7 @@ int segment_copy_from_bio(struct pcache_segment *segment,
iov_iter_bvec(&iter, ITER_SOURCE, &bio->bi_io_vec[bio->bi_iter.bi_idx],
bio_segments(bio), bio->bi_iter.bi_size);
iter.iov_offset = bio->bi_iter.bi_bvec_done;
iter.iov_offset = bio->bi_iter.bi_offset;
if (bio_off)
iov_iter_advance(&iter, bio_off);
drivers/md/md-bitmap.c
+14 -8
View File
@@ -1730,6 +1730,13 @@ static void bitmap_start_write(struct mddev *mddev, sector_t offset,
}
}
static void bitmap_prepare_range(struct mddev *mddev, sector_t *offset,
unsigned long *sectors, bool discard)
{
if (mddev->pers->bitmap_sector)
mddev->pers->bitmap_sector(mddev, offset, sectors);
}
static void bitmap_end_write(struct mddev *mddev, sector_t offset,
unsigned long sectors)
{
@@ -2064,23 +2071,18 @@ static void bitmap_end_behind_write(struct mddev *mddev)
bitmap->mddev->bitmap_info.max_write_behind);
}
static bool bitmap_wait_behind_writes(struct mddev *mddev, bool nowait)
static void bitmap_wait_behind_writes(struct mddev *mddev)
{
struct bitmap *bitmap = mddev->bitmap;
/* wait for behind writes to complete */
if (bitmap && atomic_read(&bitmap->behind_writes) > 0) {
if (nowait)
return false;
pr_debug("md:%s: behind writes in progress - waiting to stop.\n",
mdname(mddev));
/* need to kick something here to make sure I/O goes? */
wait_event(bitmap->behind_wait,
atomic_read(&bitmap->behind_writes) == 0);
}
return true;
}
static void bitmap_destroy(struct mddev *mddev)
@@ -2090,7 +2092,7 @@ static void bitmap_destroy(struct mddev *mddev)
if (!bitmap) /* there was no bitmap */
return;
bitmap_wait_behind_writes(mddev, false);
bitmap_wait_behind_writes(mddev);
if (!test_bit(MD_SERIALIZE_POLICY, &mddev->flags))
mddev_destroy_serial_pool(mddev, NULL);
@@ -2629,10 +2631,12 @@ static ssize_t
location_store(struct mddev *mddev, const char *buf, size_t len)
{
int rv;
unsigned int noio_flags;
rv = mddev_suspend_and_lock(mddev);
if (rv)
return rv;
noio_flags = memalloc_noio_save();
if (mddev->pers) {
if (mddev->recovery || mddev->sync_thread) {
@@ -2719,6 +2723,7 @@ location_store(struct mddev *mddev, const char *buf, size_t len)
}
rv = 0;
out:
memalloc_noio_restore(noio_flags);
mddev_unlock_and_resume(mddev);
if (rv)
return rv;
@@ -2862,7 +2867,7 @@ backlog_store(struct mddev *mddev, const char *buf, size_t len)
if (!has_write_mostly) {
pr_warn_ratelimited("%s: can't set backlog, no write mostly device available\n",
mdname(mddev));
mddev_unlock(mddev);
mddev_unlock_and_resume(mddev);
return -EINVAL;
}
@@ -3083,6 +3088,7 @@ static struct bitmap_operations bitmap_ops = {
.flush = bitmap_flush,
.write_all = bitmap_write_all,
.dirty_bits = bitmap_dirty_bits,
.prepare_range = bitmap_prepare_range,
.unplug = bitmap_unplug,
.daemon_work = bitmap_daemon_work,
drivers/md/md-bitmap.h
+11 -1
View File
@@ -29,6 +29,7 @@ enum bitmap_state {
BITMAP_FIRST_USE = 3, /* llbitmap is just created */
BITMAP_CLEAN = 4, /* llbitmap is created with assume_clean */
BITMAP_DAEMON_BUSY = 5, /* llbitmap daemon is not finished after daemon_sleep */
BITMAP_SHUTDOWN = 6, /* llbitmap is being destroyed */
BITMAP_HOSTENDIAN =15,
};
@@ -93,12 +94,21 @@ struct bitmap_operations {
void (*write_all)(struct mddev *mddev);
void (*dirty_bits)(struct mddev *mddev, unsigned long s,
unsigned long e);
/* Prepare a range for this bitmap implementation. */
void (*prepare_range)(struct mddev *mddev,
sector_t *offset,
unsigned long *sectors,
bool discard);
void (*reshape_finish)(struct mddev *mddev);
int (*reshape_can_start)(struct mddev *mddev);
void (*reshape_mark)(struct mddev *mddev, sector_t old_pos,
sector_t new_pos);
void (*unplug)(struct mddev *mddev, bool sync);
void (*daemon_work)(struct mddev *mddev);
void (*start_behind_write)(struct mddev *mddev);
void (*end_behind_write)(struct mddev *mddev);
bool (*wait_behind_writes)(struct mddev *mddev, bool nowait);
void (*wait_behind_writes)(struct mddev *mddev);
md_bitmap_fn *start_write;
md_bitmap_fn *end_write;
drivers/md/md-linear.c
+1
View File
@@ -71,6 +71,7 @@ static int linear_set_limits(struct mddev *mddev)
int err;
md_init_stacking_limits(&lim);
lim.features |= BLK_FEAT_NOWAIT;
lim.max_hw_sectors = mddev->chunk_sectors;
lim.logical_block_size = mddev->logical_block_size;
lim.max_write_zeroes_sectors = mddev->chunk_sectors;
drivers/md/md-llbitmap.c
+681 -73
View File
File diff suppressed because it is too large Load Diff
drivers/md/md.c
+140 -38
View File
@@ -233,23 +233,22 @@ static int rdev_need_serial(struct md_rdev *rdev)
void mddev_create_serial_pool(struct mddev *mddev, struct md_rdev *rdev)
{
int ret = 0;
unsigned int noio_flags;
if (rdev && !rdev_need_serial(rdev) &&
if (!test_bit(MD_SERIALIZE_POLICY, &mddev->flags) &&
rdev && !rdev_need_serial(rdev) &&
!test_bit(CollisionCheck, &rdev->flags))
return;
noio_flags = memalloc_noio_save();
if (!rdev)
ret = rdevs_init_serial(mddev);
else
ret = rdev_init_serial(rdev);
if (ret)
return;
goto out;
if (mddev->serial_info_pool == NULL) {
/*
* already in memalloc noio context by
* mddev_suspend()
*/
mddev->serial_info_pool =
mempool_create_kmalloc_pool(NR_SERIAL_INFOS,
sizeof(struct serial_info));
@@ -258,6 +257,8 @@ void mddev_create_serial_pool(struct mddev *mddev, struct md_rdev *rdev)
pr_err("can't alloc memory pool for serialization\n");
}
}
out:
memalloc_noio_restore(noio_flags);
}
/*
@@ -516,9 +517,6 @@ int mddev_suspend(struct mddev *mddev, bool interruptible)
*/
WRITE_ONCE(mddev->suspended, mddev->suspended + 1);
/* restrict memory reclaim I/O during raid array is suspend */
mddev->noio_flag = memalloc_noio_save();
mutex_unlock(&mddev->suspend_mutex);
return 0;
}
@@ -535,9 +533,6 @@ static void __mddev_resume(struct mddev *mddev, bool recovery_needed)
return;
}
/* entred the memalloc scope from mddev_suspend() */
memalloc_noio_restore(mddev->noio_flag);
percpu_ref_resurrect(&mddev->active_io);
wake_up(&mddev->sb_wait);
@@ -1920,6 +1915,13 @@ static int super_1_load(struct md_rdev *rdev, struct md_rdev *refdev, int minor_
rdev->bb_page, REQ_OP_READ, true))
return -EIO;
bbp = (__le64 *)page_address(rdev->bb_page);
/* check for badblocks api. */
if (sb->bblog_shift >= BITS_PER_TYPE(sector_t)) {
pr_err("md: %pg: bogus bblog_shift %u for badblocks.\n",
rdev->bdev, sb->bblog_shift);
return -EINVAL;
}
rdev->badblocks.shift = sb->bblog_shift;
for (i = 0 ; i < (sectors << (9-3)) ; i++, bbp++) {
u64 bb = le64_to_cpu(*bbp);
@@ -4047,6 +4049,7 @@ level_store(struct mddev *mddev, const char *buf, size_t len)
char clevel[16];
ssize_t rv;
size_t slen = len;
unsigned int noio_flags;
struct md_personality *pers, *oldpers;
long level;
void *priv, *oldpriv;
@@ -4058,6 +4061,7 @@ level_store(struct mddev *mddev, const char *buf, size_t len)
rv = mddev_suspend_and_lock(mddev);
if (rv)
return rv;
noio_flags = memalloc_noio_save();
if (mddev->pers == NULL) {
memcpy(mddev->clevel, buf, slen);
@@ -4233,6 +4237,7 @@ level_store(struct mddev *mddev, const char *buf, size_t len)
md_new_event();
rv = len;
out_unlock:
memalloc_noio_restore(noio_flags);
mddev_unlock_and_resume(mddev);
return rv;
}
@@ -4412,6 +4417,7 @@ static ssize_t
raid_disks_store(struct mddev *mddev, const char *buf, size_t len)
{
unsigned int n;
unsigned int noio_flags;
int err;
err = kstrtouint(buf, 10, &n);
@@ -4421,6 +4427,7 @@ raid_disks_store(struct mddev *mddev, const char *buf, size_t len)
err = mddev_suspend_and_lock(mddev);
if (err)
return err;
noio_flags = memalloc_noio_save();
if (mddev->pers) {
if (n != mddev->raid_disks)
err = update_raid_disks(mddev, n);
@@ -4444,6 +4451,7 @@ raid_disks_store(struct mddev *mddev, const char *buf, size_t len)
} else
mddev->raid_disks = n;
out_unlock:
memalloc_noio_restore(noio_flags);
mddev_unlock_and_resume(mddev);
return err ? err : len;
}
@@ -4824,6 +4832,7 @@ new_dev_store(struct mddev *mddev, const char *buf, size_t len)
int minor;
dev_t dev;
struct md_rdev *rdev;
unsigned int noio_flags;
int err;
if (!*buf || *e != ':' || !e[1] || e[1] == '\n')
@@ -4839,6 +4848,7 @@ new_dev_store(struct mddev *mddev, const char *buf, size_t len)
err = mddev_suspend_and_lock(mddev);
if (err)
return err;
noio_flags = memalloc_noio_save();
if (mddev->persistent) {
rdev = md_import_device(dev, mddev->major_version,
mddev->minor_version);
@@ -4857,6 +4867,7 @@ new_dev_store(struct mddev *mddev, const char *buf, size_t len)
rdev = md_import_device(dev, -1, -1);
if (IS_ERR(rdev)) {
memalloc_noio_restore(noio_flags);
mddev_unlock_and_resume(mddev);
return PTR_ERR(rdev);
}
@@ -4864,6 +4875,7 @@ new_dev_store(struct mddev *mddev, const char *buf, size_t len)
out:
if (err)
export_rdev(rdev);
memalloc_noio_restore(noio_flags);
mddev_unlock_and_resume(mddev);
if (!err)
md_new_event();
@@ -5263,21 +5275,28 @@ action_store(struct mddev *mddev, const char *page, size_t len)
if (!mddev->pers || !mddev->pers->sync_request)
return -EINVAL;
action = md_sync_action_by_name(page);
if (action == ACTION_RESHAPE) {
ret = mddev_suspend(mddev, true);
if (ret)
return ret;
}
retry:
if (work_busy(&mddev->sync_work))
flush_work(&mddev->sync_work);
ret = mddev_lock(mddev);
if (ret)
if (ret) {
if (action == ACTION_RESHAPE)
mddev_resume(mddev);
return ret;
}
if (work_busy(&mddev->sync_work)) {
mddev_unlock(mddev);
goto retry;
}
action = md_sync_action_by_name(page);
/* TODO: mdadm rely on "idle" to start sync_thread. */
if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)) {
switch (action) {
@@ -5347,6 +5366,8 @@ retry:
out:
mddev_unlock(mddev);
if (action == ACTION_RESHAPE)
mddev_resume(mddev);
return ret;
}
@@ -6283,7 +6304,7 @@ void md_init_stacking_limits(struct queue_limits *lim)
{
blk_set_stacking_limits(lim);
lim->features = BLK_FEAT_WRITE_CACHE | BLK_FEAT_FUA |
BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT;
BLK_FEAT_IO_STAT;
}
EXPORT_SYMBOL_GPL(md_init_stacking_limits);
@@ -6631,7 +6652,6 @@ int md_run(struct mddev *mddev)
int err;
struct md_rdev *rdev;
struct md_personality *pers;
bool nowait = true;
if (list_empty(&mddev->disks))
/* cannot run an array with no devices.. */
@@ -6702,7 +6722,6 @@ int md_run(struct mddev *mddev)
}
}
sysfs_notify_dirent_safe(rdev->sysfs_state);
nowait = nowait && bdev_nowait(rdev->bdev);
}
pers = get_pers(mddev->level, mddev->clevel);
@@ -7050,7 +7069,7 @@ EXPORT_SYMBOL_GPL(md_stop_writes);
static void mddev_detach(struct mddev *mddev)
{
if (md_bitmap_enabled(mddev, false))
mddev->bitmap_ops->wait_behind_writes(mddev, false);
mddev->bitmap_ops->wait_behind_writes(mddev);
if (mddev->pers && mddev->pers->quiesce && !is_md_suspended(mddev)) {
mddev->pers->quiesce(mddev, 1);
mddev->pers->quiesce(mddev, 0);
@@ -7066,8 +7085,8 @@ static void __md_stop(struct mddev *mddev)
{
struct md_personality *pers = mddev->pers;
md_bitmap_destroy(mddev);
mddev_detach(mddev);
md_bitmap_destroy(mddev);
spin_lock(&mddev->lock);
mddev->pers = NULL;
spin_unlock(&mddev->lock);
@@ -8324,8 +8343,10 @@ static int md_ioctl(struct block_device *bdev, blk_mode_t mode,
unsigned int cmd, unsigned long arg)
{
int err = 0;
unsigned int noio_flags = 0;
void __user *argp = (void __user *)arg;
struct mddev *mddev = NULL;
bool suspend;
err = md_ioctl_valid(cmd);
if (err)
@@ -8375,13 +8396,15 @@ static int md_ioctl(struct block_device *bdev, blk_mode_t mode,
if (!md_is_rdwr(mddev))
flush_work(&mddev->sync_work);
err = md_ioctl_need_suspend(cmd) ? mddev_suspend_and_lock(mddev) :
mddev_lock(mddev);
suspend = md_ioctl_need_suspend(cmd);
err = suspend ? mddev_suspend_and_lock(mddev) : mddev_lock(mddev);
if (err) {
pr_debug("md: ioctl lock interrupted, reason %d, cmd %d\n",
err, cmd);
goto out;
}
if (suspend)
noio_flags = memalloc_noio_save();
if (cmd == SET_ARRAY_INFO) {
err = __md_set_array_info(mddev, argp);
@@ -8506,8 +8529,12 @@ unlock:
err != -EINVAL)
mddev->hold_active = 0;
md_ioctl_need_suspend(cmd) ? mddev_unlock_and_resume(mddev) :
mddev_unlock(mddev);
if (suspend) {
memalloc_noio_restore(noio_flags);
mddev_unlock_and_resume(mddev);
} else {
mddev_unlock(mddev);
}
out:
if (cmd == STOP_ARRAY_RO || (err && cmd == STOP_ARRAY))
@@ -9350,6 +9377,10 @@ void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
{
struct bio *discard_bio = NULL;
/* Discard is optional, so silently skip members that do not support it. */
if (unlikely(!bdev_max_discard_sectors(rdev->bdev)))
return;
__blkdev_issue_discard(rdev->bdev, start, size, GFP_NOIO, &discard_bio);
if (!discard_bio)
return;
@@ -9361,17 +9392,62 @@ void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
}
EXPORT_SYMBOL_GPL(md_submit_discard_bio);
struct bio *mddev_bio_split_at_reshape_offset(struct mddev *mddev,
struct bio *bio,
unsigned int *max_sectors,
struct bio_set *bs)
{
sector_t boundary;
sector_t start;
sector_t end;
unsigned int split_sectors;
split_sectors = bio_sectors(bio);
if (max_sectors && *max_sectors && *max_sectors < split_sectors)
split_sectors = *max_sectors;
if (!test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery))
goto split;
boundary = READ_ONCE(mddev->reshape_position);
start = bio->bi_iter.bi_sector;
end = bio_end_sector(bio);
if (start >= boundary || end <= boundary)
goto split;
if (boundary - start < split_sectors)
split_sectors = boundary - start;
split:
if (max_sectors)
*max_sectors = split_sectors;
if (split_sectors < bio_sectors(bio)) {
bio = bio_submit_split_bioset(bio, split_sectors, bs);
if (bio)
bio->bi_opf |= REQ_NOMERGE;
}
return bio;
}
EXPORT_SYMBOL_GPL(mddev_bio_split_at_reshape_offset);
static void md_bitmap_prepare_range(struct mddev *mddev, sector_t *offset,
unsigned long *sectors, bool discard)
{
mddev->bitmap_ops->prepare_range(mddev, offset, sectors, discard);
}
static void md_bitmap_start(struct mddev *mddev,
struct md_io_clone *md_io_clone)
{
md_bitmap_fn *fn = unlikely(md_io_clone->rw == STAT_DISCARD) ?
mddev->bitmap_ops->start_discard :
bool discard = md_io_clone->rw == STAT_DISCARD;
md_bitmap_fn *fn = discard ? mddev->bitmap_ops->start_discard :
mddev->bitmap_ops->start_write;
if (mddev->pers->bitmap_sector)
mddev->pers->bitmap_sector(mddev, &md_io_clone->offset,
&md_io_clone->sectors);
md_bitmap_prepare_range(mddev, &md_io_clone->offset,
&md_io_clone->sectors, discard);
if (!md_io_clone->sectors)
return;
fn(mddev, md_io_clone->offset, md_io_clone->sectors);
}
@@ -9392,7 +9468,8 @@ static void md_end_clone_io(struct bio *bio)
struct mddev *mddev = md_io_clone->mddev;
struct completion *reshape_completion = bio->bi_private;
if (bio_data_dir(orig_bio) == WRITE && md_bitmap_enabled(mddev, false))
if (bio_data_dir(orig_bio) == WRITE && md_io_clone->sectors &&
md_bitmap_enabled(mddev, false))
md_bitmap_end(mddev, md_io_clone);
if (bio->bi_status && !orig_bio->bi_status)
@@ -9419,10 +9496,14 @@ static void md_clone_bio(struct mddev *mddev, struct bio **bio)
md_io_clone = container_of(clone, struct md_io_clone, bio_clone);
md_io_clone->orig_bio = *bio;
md_io_clone->mddev = mddev;
md_io_clone->sectors = 0;
if (blk_queue_io_stat(bdev->bd_disk->queue))
md_io_clone->start_time = bio_start_io_acct(*bio);
else
md_io_clone->start_time = 0;
if (bio_data_dir(*bio) == WRITE && md_bitmap_enabled(mddev, false)) {
if (bio_data_dir(*bio) == WRITE && bio_sectors(*bio) &&
md_bitmap_enabled(mddev, false)) {
md_io_clone->offset = (*bio)->bi_iter.bi_sector;
md_io_clone->sectors = bio_sectors(*bio);
md_io_clone->rw = op_stat_group(bio_op(*bio));
@@ -9881,8 +9962,10 @@ update:
*/
md_new_event();
if (last_check + window > io_sectors || j == max_sectors)
if (last_check + window > io_sectors || j == max_sectors) {
cond_resched();
continue;
}
last_check = io_sectors;
repeat:
@@ -10175,19 +10258,34 @@ static void md_start_sync(struct work_struct *ws)
struct mddev *mddev = container_of(ws, struct mddev, sync_work);
int spares = 0;
bool suspend = false;
unsigned int noio_flags = 0;
char *name;
/*
* If reshape is still in progress, spares won't be added or removed
* from conf until reshape is done.
*/
if (mddev->reshape_position == MaxSector &&
if ((mddev->reshape_position == MaxSector || !md_is_rdwr(mddev)) &&
md_spares_need_change(mddev)) {
suspend = true;
mddev_suspend(mddev, false);
noio_flags = memalloc_noio_save();
}
mddev_lock_nointr(mddev);
/*
* The spare configuration can change before reconfig_mutex is acquired.
* Recheck while holding the lock and suspend if needed.
*/
if (!suspend && (mddev->reshape_position == MaxSector || !md_is_rdwr(mddev)) &&
md_spares_need_change(mddev)) {
mddev_unlock(mddev);
mddev_suspend_and_lock_nointr(mddev);
suspend = true;
noio_flags = memalloc_noio_save();
}
if (!md_is_rdwr(mddev)) {
/*
* On a read-only array we can:
@@ -10231,8 +10329,10 @@ static void md_start_sync(struct work_struct *ws)
* https://bugzilla.kernel.org/show_bug.cgi?id=218200
* Therefore, use __mddev_resume(mddev, false).
*/
if (suspend)
if (suspend) {
memalloc_noio_restore(noio_flags);
__mddev_resume(mddev, false);
}
md_wakeup_thread(mddev->sync_thread);
sysfs_notify_dirent_safe(mddev->sysfs_action);
md_new_event();
@@ -10251,8 +10351,10 @@ not_running:
* https://bugzilla.kernel.org/show_bug.cgi?id=218200
* Therefore, use __mddev_resume(mddev, false).
*/
if (suspend)
if (suspend) {
memalloc_noio_restore(noio_flags);
__mddev_resume(mddev, false);
}
wake_up(&resync_wait);
if (test_and_clear_bit(MD_RECOVERY_RECOVER, &mddev->recovery) &&
@@ -10553,7 +10655,7 @@ EXPORT_SYMBOL(md_finish_reshape);
/* Bad block management */
/* Returns true on success, false on failure */
bool rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
bool rdev_set_badblocks(struct md_rdev *rdev, sector_t s, sector_t sectors,
int is_new)
{
struct mddev *mddev = rdev->mddev;
@@ -10593,7 +10695,7 @@ bool rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
}
EXPORT_SYMBOL_GPL(rdev_set_badblocks);
void rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
void rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, sector_t sectors,
int is_new)
{
if (is_new)
drivers/md/md.h
+11 -4
View File
@@ -311,7 +311,7 @@ static inline int is_badblock(struct md_rdev *rdev, sector_t s, sector_t sectors
}
static inline int rdev_has_badblock(struct md_rdev *rdev, sector_t s,
int sectors)
sector_t sectors)
{
sector_t first_bad;
sector_t bad_sectors;
@@ -319,9 +319,9 @@ static inline int rdev_has_badblock(struct md_rdev *rdev, sector_t s,
return is_badblock(rdev, s, sectors, &first_bad, &bad_sectors);
}
extern bool rdev_set_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
extern bool rdev_set_badblocks(struct md_rdev *rdev, sector_t s, sector_t sectors,
int is_new);
extern void rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, int sectors,
extern void rdev_clear_badblocks(struct md_rdev *rdev, sector_t s, sector_t sectors,
int is_new);
struct md_cluster_info;
struct md_cluster_operations;
@@ -621,7 +621,6 @@ struct mddev {
struct md_cluster_info *cluster_info;
struct md_cluster_operations *cluster_ops;
unsigned int good_device_nr; /* good device num within cluster raid */
unsigned int noio_flag; /* for memalloc scope API */
/*
* Temporarily store rdev that will be finally removed when
@@ -798,6 +797,10 @@ struct md_personality
/* convert io ranges from array to bitmap */
void (*bitmap_sector)(struct mddev *mddev, sector_t *offset,
unsigned long *sectors);
void (*bitmap_sector_map)(struct mddev *mddev, sector_t *offset,
unsigned long *sectors, bool previous);
sector_t (*bitmap_sync_size)(struct mddev *mddev, bool previous);
sector_t (*bitmap_array_sectors)(struct mddev *mddev, bool previous);
};
struct md_sysfs_entry {
@@ -921,6 +924,10 @@ extern void md_error(struct mddev *mddev, struct md_rdev *rdev);
extern void md_finish_reshape(struct mddev *mddev);
void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
struct bio *bio, sector_t start, sector_t size);
struct bio *mddev_bio_split_at_reshape_offset(struct mddev *mddev,
struct bio *bio,
unsigned int *max_sectors,
struct bio_set *bs);
void md_account_bio(struct mddev *mddev, struct bio **bio);
extern bool __must_check md_flush_request(struct mddev *mddev, struct bio *bio);
drivers/md/raid0.c
+1
View File
@@ -385,6 +385,7 @@ static int raid0_set_limits(struct mddev *mddev)
int err;
md_init_stacking_limits(&lim);
lim.features |= BLK_FEAT_NOWAIT;
lim.max_hw_sectors = mddev->chunk_sectors;
lim.max_write_zeroes_sectors = mddev->chunk_sectors;
lim.max_hw_wzeroes_unmap_sectors = mddev->chunk_sectors;
drivers/md/raid1-10.c
+3 -5
View File
@@ -290,9 +290,8 @@ static inline bool raid1_should_read_first(struct mddev *mddev,
}
/*
* bio with REQ_RAHEAD or REQ_NOWAIT can fail at anytime, before such IO is
* submitted to the underlying disks, hence don't record badblocks or retry
* in this case.
* bio with REQ_RAHEAD can fail at anytime, before such IO is submitted to the
* underlying disks, hence don't record badblocks or retry in this case.
*
* BLK_STS_INVAL means the bio was not valid for the underlying device. This
* is a user error, not a device failure, so retrying or recording bad blocks
@@ -300,6 +299,5 @@ static inline bool raid1_should_read_first(struct mddev *mddev,
*/
static inline bool raid1_should_handle_error(struct bio *bio)
{
return !(bio->bi_opf & (REQ_RAHEAD | REQ_NOWAIT)) &&
bio->bi_status != BLK_STS_INVAL;
return !(bio->bi_opf & REQ_RAHEAD) && bio->bi_status != BLK_STS_INVAL;
}
drivers/md/raid1.c
+41 -98
View File
@@ -604,9 +604,9 @@ static void update_read_sectors(struct r1conf *conf, int disk,
struct raid1_info *info = &conf->mirrors[disk];
atomic_inc(&info->rdev->nr_pending);
if (info->next_seq_sect != this_sector)
info->seq_start = this_sector;
info->next_seq_sect = this_sector + len;
if (READ_ONCE(info->next_seq_sect) != this_sector)
WRITE_ONCE(info->seq_start, this_sector);
WRITE_ONCE(info->next_seq_sect, this_sector + len);
}
static int choose_first_rdev(struct r1conf *conf, struct r1bio *r1_bio,
@@ -735,8 +735,7 @@ static int choose_slow_rdev(struct r1conf *conf, struct r1bio *r1_bio,
static bool is_sequential(struct r1conf *conf, int disk, struct r1bio *r1_bio)
{
/* TODO: address issues with this check and concurrency. */
return conf->mirrors[disk].next_seq_sect == r1_bio->sector ||
return READ_ONCE(conf->mirrors[disk].next_seq_sect) == r1_bio->sector ||
READ_ONCE(conf->mirrors[disk].head_position) == r1_bio->sector;
}
@@ -747,15 +746,18 @@ static bool is_sequential(struct r1conf *conf, int disk, struct r1bio *r1_bio)
static bool should_choose_next(struct r1conf *conf, int disk)
{
struct raid1_info *mirror = &conf->mirrors[disk];
sector_t seq_start, next_seq_sect;
int opt_iosize;
if (!test_bit(Nonrot, &mirror->rdev->flags))
return false;
opt_iosize = bdev_io_opt(mirror->rdev->bdev) >> 9;
return opt_iosize > 0 && mirror->seq_start != MaxSector &&
mirror->next_seq_sect > opt_iosize &&
mirror->next_seq_sect - opt_iosize >= mirror->seq_start;
seq_start = READ_ONCE(mirror->seq_start);
next_seq_sect = READ_ONCE(mirror->next_seq_sect);
return opt_iosize > 0 && seq_start != MaxSector &&
next_seq_sect > opt_iosize &&
next_seq_sect - opt_iosize >= seq_start;
}
static bool rdev_readable(struct md_rdev *rdev, struct r1bio *r1_bio)
@@ -1051,10 +1053,8 @@ static void lower_barrier(struct r1conf *conf, sector_t sector_nr)
wake_up(&conf->wait_barrier);
}
static bool _wait_barrier(struct r1conf *conf, int idx, bool nowait)
static void _wait_barrier(struct r1conf *conf, int idx)
{
bool ret = true;
/*
* We need to increase conf->nr_pending[idx] very early here,
* then raise_barrier() can be blocked when it waits for
@@ -1085,7 +1085,7 @@ static bool _wait_barrier(struct r1conf *conf, int idx, bool nowait)
*/
if (!READ_ONCE(conf->array_frozen) &&
!atomic_read(&conf->barrier[idx]))
return ret;
return;
/*
* After holding conf->resync_lock, conf->nr_pending[idx]
@@ -1104,26 +1104,18 @@ static bool _wait_barrier(struct r1conf *conf, int idx, bool nowait)
wake_up_barrier(conf);
/* Wait for the barrier in same barrier unit bucket to drop. */
/* Return false when nowait flag is set */
if (nowait) {
ret = false;
} else {
wait_event_lock_irq(conf->wait_barrier,
!conf->array_frozen &&
!atomic_read(&conf->barrier[idx]),
conf->resync_lock);
atomic_inc(&conf->nr_pending[idx]);
}
wait_event_lock_irq(conf->wait_barrier, !conf->array_frozen &&
!atomic_read(&conf->barrier[idx]),
conf->resync_lock);
atomic_inc(&conf->nr_pending[idx]);
atomic_dec(&conf->nr_waiting[idx]);
spin_unlock_irq(&conf->resync_lock);
return ret;
}
static bool wait_read_barrier(struct r1conf *conf, sector_t sector_nr, bool nowait)
static void wait_read_barrier(struct r1conf *conf, sector_t sector_nr)
{
int idx = sector_to_idx(sector_nr);
bool ret = true;
/*
* Very similar to _wait_barrier(). The difference is, for read
@@ -1135,7 +1127,7 @@ static bool wait_read_barrier(struct r1conf *conf, sector_t sector_nr, bool nowa
atomic_inc(&conf->nr_pending[idx]);
if (!READ_ONCE(conf->array_frozen))
return ret;
return;
spin_lock_irq(&conf->resync_lock);
atomic_inc(&conf->nr_waiting[idx]);
@@ -1147,27 +1139,19 @@ static bool wait_read_barrier(struct r1conf *conf, sector_t sector_nr, bool nowa
wake_up_barrier(conf);
/* Wait for array to be unfrozen */
/* Return false when nowait flag is set */
if (nowait) {
/* Return false when nowait flag is set */
ret = false;
} else {
wait_event_lock_irq(conf->wait_barrier,
!conf->array_frozen,
conf->resync_lock);
atomic_inc(&conf->nr_pending[idx]);
}
wait_event_lock_irq(conf->wait_barrier, !conf->array_frozen,
conf->resync_lock);
atomic_inc(&conf->nr_pending[idx]);
atomic_dec(&conf->nr_waiting[idx]);
spin_unlock_irq(&conf->resync_lock);
return ret;
}
static bool wait_barrier(struct r1conf *conf, sector_t sector_nr, bool nowait)
static void wait_barrier(struct r1conf *conf, sector_t sector_nr)
{
int idx = sector_to_idx(sector_nr);
return _wait_barrier(conf, idx, nowait);
_wait_barrier(conf, idx);
}
static void _allow_barrier(struct r1conf *conf, int idx)
@@ -1342,7 +1326,6 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
int max_sectors;
int rdisk;
bool r1bio_existed = !!r1_bio;
bool nowait = bio->bi_opf & REQ_NOWAIT;
/*
* An md cloned bio indicates we are in the error path.
@@ -1362,16 +1345,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
* Still need barrier for READ in case that whole
* array is frozen.
*/
if (!wait_read_barrier(conf, bio->bi_iter.bi_sector, nowait)) {
bio_wouldblock_error(bio);
if (r1bio_existed) {
set_bit(R1BIO_Returned, &r1_bio->state);
raid_end_bio_io(r1_bio);
}
return;
}
wait_read_barrier(conf, bio->bi_iter.bi_sector);
if (!r1_bio)
r1_bio = alloc_r1bio(mddev, bio);
@@ -1406,14 +1380,10 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
md_bitmap_enabled(mddev, false)) {
/*
* Reading from a write-mostly device must take care not to
* over-take any writes that are 'behind'
*/
mddev_add_trace_msg(mddev, "raid1 wait behind writes");
if (!mddev->bitmap_ops->wait_behind_writes(mddev, nowait)) {
bio_wouldblock_error(bio);
set_bit(R1BIO_Returned, &r1_bio->state);
goto err_handle;
}
* over-take any writes that are 'behind'
*/
mddev_add_trace_msg(mddev, "raid1 wait behind writes");
mddev->bitmap_ops->wait_behind_writes(mddev);
}
if (max_sectors < bio_sectors(bio)) {
@@ -1435,7 +1405,6 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
}
read_bio = bio_alloc_clone(mirror->rdev->bdev, bio, gfp,
&mddev->bio_set);
read_bio->bi_opf &= ~REQ_NOWAIT;
r1_bio->bios[rdisk] = read_bio;
read_bio->bi_iter.bi_sector = r1_bio->sector +
@@ -1454,7 +1423,7 @@ err_handle:
raid_end_bio_io(r1_bio);
}
static bool wait_blocked_rdev(struct mddev *mddev, struct bio *bio)
static void wait_blocked_rdev(struct mddev *mddev, struct bio *bio)
{
struct r1conf *conf = mddev->private;
int disks = conf->raid_disks * 2;
@@ -1474,9 +1443,6 @@ retry:
set_bit(BlockedBadBlocks, &rdev->flags);
if (rdev_blocked(rdev)) {
if (bio->bi_opf & REQ_NOWAIT)
return false;
mddev_add_trace_msg(rdev->mddev, "raid1 wait rdev %d blocked",
rdev->raid_disk);
atomic_inc(&rdev->nr_pending);
@@ -1484,8 +1450,6 @@ retry:
goto retry;
}
}
return true;
}
static void raid1_start_write_behind(struct mddev *mddev, struct r1bio *r1_bio,
@@ -1521,18 +1485,13 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
unsigned long flags;
int first_clone;
bool write_behind = false;
bool nowait = bio->bi_opf & REQ_NOWAIT;
bool atomic = bio->bi_opf & REQ_ATOMIC;
bool is_discard = op_is_discard(bio->bi_opf);
sector_t sector = bio->bi_iter.bi_sector;
if (mddev_is_clustered(mddev) &&
mddev->cluster_ops->area_resyncing(mddev, WRITE, sector,
bio_end_sector(bio))) {
if (nowait) {
bio_wouldblock_error(bio);
return false;
}
wait_event_idle(conf->wait_barrier,
!mddev->cluster_ops->area_resyncing(mddev, WRITE,
sector,
@@ -1544,15 +1503,9 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
* thread has put up a bar for new requests.
* Continue immediately if no resync is active currently.
*/
if (!wait_barrier(conf, sector, nowait)) {
bio_wouldblock_error(bio);
return false;
}
wait_barrier(conf, sector);
if (!wait_blocked_rdev(mddev, bio)) {
bio_wouldblock_error(bio);
goto err_allow_barrier;
}
wait_blocked_rdev(mddev, bio);
r1_bio = alloc_r1bio(mddev, bio);
r1_bio->sectors = max_sectors;
@@ -1579,6 +1532,8 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
*/
if (!is_discard && rdev && test_bit(WriteMostly, &rdev->flags))
write_behind = true;
if (atomic && max_sectors > BIO_MAX_VECS * (PAGE_SIZE >> 9))
write_behind = false;
r1_bio->bios[i] = NULL;
if (!rdev || test_bit(Faulty, &rdev->flags))
@@ -1604,19 +1559,6 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
if (is_bad) {
int good_sectors;
/*
* We cannot atomically write this, so just
* error in that case. It could be possible to
* atomically write other mirrors, but the
* complexity of supporting that is not worth
* the benefit.
*/
if (bio->bi_opf & REQ_ATOMIC) {
bio->bi_status = BLK_STS_NOTSUPP;
bio_endio(bio);
goto err_dec_pending;
}
good_sectors = first_bad - sector;
if (good_sectors < max_sectors)
max_sectors = good_sectors;
@@ -1637,6 +1579,11 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
max_sectors = min_t(int, max_sectors,
BIO_MAX_VECS * (PAGE_SIZE >> 9));
if (max_sectors < bio_sectors(bio)) {
if (atomic) {
bio_io_error(bio);
goto err_dec_pending;
}
bio = bio_submit_split_bioset(bio, max_sectors,
&conf->bio_split);
if (!bio)
@@ -1681,7 +1628,6 @@ static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
wait_for_serialization(rdev, r1_bio);
}
mbio->bi_opf &= ~REQ_NOWAIT;
r1_bio->bios[i] = mbio;
mbio->bi_iter.bi_sector = sector + rdev->data_offset;
@@ -1720,8 +1666,6 @@ err_dec_pending:
}
free_r1bio(r1_bio);
err_allow_barrier:
allow_barrier(conf, sector);
return false;
@@ -1850,7 +1794,7 @@ static void close_sync(struct r1conf *conf)
int idx;
for (idx = 0; idx < BARRIER_BUCKETS_NR; idx++) {
_wait_barrier(conf, idx, false);
_wait_barrier(conf, idx);
_allow_barrier(conf, idx);
}
@@ -3228,6 +3172,7 @@ static int raid1_set_limits(struct mddev *mddev)
md_init_stacking_limits(&lim);
lim.max_write_zeroes_sectors = 0;
lim.max_hw_wzeroes_unmap_sectors = 0;
lim.chunk_sectors = BARRIER_UNIT_SECTOR_SIZE;
lim.logical_block_size = mddev->logical_block_size;
lim.features |= BLK_FEAT_ATOMIC_WRITES;
lim.features |= BLK_FEAT_PCI_P2PDMA;
@@ -3485,8 +3430,6 @@ static void *raid1_takeover(struct mddev *mddev)
mddev->new_chunk_sectors = 0;
conf = setup_conf(mddev);
if (!IS_ERR(conf)) {
/* Array must appear to be quiesced */
conf->array_frozen = 1;
mddev_clear_unsupported_flags(mddev,
UNSUPPORTED_MDDEV_FLAGS);
}
drivers/md/raid10.c
+126 -101
View File
@@ -103,13 +103,23 @@ static inline struct r10bio *get_resync_r10bio(struct bio *bio)
return get_resync_pages(bio)->raid_bio;
}
static void * r10bio_pool_alloc(gfp_t gfp_flags, void *data)
static inline int calc_r10bio_size(unsigned int raid_disks)
{
struct r10conf *conf = data;
int size = offsetof(struct r10bio, devs[conf->geo.raid_disks]);
return offsetof(struct r10bio, devs[raid_disks]);
}
/* allocate a r10bio with room for raid_disks entries in the
* bios array */
static mempool_t *create_r10bio_pool(unsigned int raid_disks)
{
int size = calc_r10bio_size(raid_disks);
return mempool_create_kmalloc_pool(NR_RAID_BIOS, size);
}
static struct r10bio *alloc_r10bio(unsigned int raid_disks, gfp_t gfp_flags)
{
int size = calc_r10bio_size(raid_disks);
/* allocate a r10bio sized for current geometry */
return kzalloc(size, gfp_flags);
}
@@ -137,7 +147,7 @@ static void * r10buf_pool_alloc(gfp_t gfp_flags, void *data)
int nalloc, nalloc_rp;
struct resync_pages *rps;
r10_bio = r10bio_pool_alloc(gfp_flags, conf);
r10_bio = alloc_r10bio(conf->geo.raid_disks, gfp_flags);
if (!r10_bio)
return NULL;
@@ -277,7 +287,7 @@ static void free_r10bio(struct r10bio *r10_bio)
struct r10conf *conf = r10_bio->mddev->private;
put_all_bios(conf, r10_bio);
mempool_free(r10_bio, &conf->r10bio_pool);
mempool_free(r10_bio, conf->r10bio_pool);
}
static void put_buf(struct r10bio *r10_bio)
@@ -321,20 +331,24 @@ static void raid_end_bio_io(struct r10bio *r10_bio)
{
struct bio *bio = r10_bio->master_bio;
struct r10conf *conf = r10_bio->mddev->private;
bool returned = true;
if (!test_and_set_bit(R10BIO_Returned, &r10_bio->state)) {
if (!test_bit(R10BIO_Uptodate, &r10_bio->state))
bio->bi_status = BLK_STS_IOERR;
bio_endio(bio);
returned = false;
}
free_r10bio(r10_bio);
if (!returned)
bio_endio(bio);
/*
* Wake up any possible resync thread that waits for the device
* to go idle.
*/
allow_barrier(conf);
free_r10bio(r10_bio);
}
/*
@@ -1002,32 +1016,22 @@ static bool wait_barrier_nolock(struct r10conf *conf)
return false;
}
static bool wait_barrier(struct r10conf *conf, bool nowait)
static void wait_barrier(struct r10conf *conf)
{
bool ret = true;
if (wait_barrier_nolock(conf))
return true;
return;
write_seqlock_irq(&conf->resync_lock);
if (conf->barrier) {
/* Return false when nowait flag is set */
if (nowait) {
ret = false;
} else {
conf->nr_waiting++;
mddev_add_trace_msg(conf->mddev, "raid10 wait barrier");
wait_event_barrier(conf, stop_waiting_barrier(conf));
conf->nr_waiting--;
}
conf->nr_waiting++;
mddev_add_trace_msg(conf->mddev, "raid10 wait barrier");
wait_event_barrier(conf, stop_waiting_barrier(conf));
conf->nr_waiting--;
if (!conf->nr_waiting)
wake_up(&conf->wait_barrier);
}
/* Only increment nr_pending when we wait */
if (ret)
atomic_inc(&conf->nr_pending);
atomic_inc(&conf->nr_pending);
write_sequnlock_irq(&conf->resync_lock);
return ret;
}
static void allow_barrier(struct r10conf *conf)
@@ -1119,30 +1123,22 @@ static void raid10_unplug(struct blk_plug_cb *cb, bool from_schedule)
* currently.
* 2. If IO spans the reshape position. Need to wait for reshape to pass.
*/
static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf,
static void regular_request_wait(struct mddev *mddev, struct r10conf *conf,
struct bio *bio, sector_t sectors)
{
/* Bail out if REQ_NOWAIT is set for the bio */
if (!wait_barrier(conf, bio->bi_opf & REQ_NOWAIT)) {
bio_wouldblock_error(bio);
return false;
}
wait_barrier(conf);
while (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
bio->bi_iter.bi_sector < conf->reshape_progress &&
bio->bi_iter.bi_sector + sectors > conf->reshape_progress) {
allow_barrier(conf);
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
return false;
}
mddev_add_trace_msg(conf->mddev, "raid10 wait reshape");
wait_event(conf->wait_barrier,
conf->reshape_progress <= bio->bi_iter.bi_sector ||
conf->reshape_progress >= bio->bi_iter.bi_sector +
sectors);
wait_barrier(conf, false);
wait_barrier(conf);
}
return true;
}
static void raid10_read_request(struct mddev *mddev, struct bio *bio,
@@ -1191,10 +1187,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
}
}
if (!regular_request_wait(mddev, conf, bio, r10_bio->sectors)) {
free_r10bio(r10_bio);
return;
}
regular_request_wait(mddev, conf, bio, r10_bio->sectors);
rdev = read_balance(conf, r10_bio, &max_sectors);
if (!rdev) {
@@ -1215,7 +1208,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
allow_barrier(conf);
bio = bio_submit_split_bioset(bio, max_sectors,
&conf->bio_split);
wait_barrier(conf, false);
wait_barrier(conf);
if (!bio) {
set_bit(R10BIO_Returned, &r10_bio->state);
goto err_handle;
@@ -1231,7 +1224,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
r10_bio->master_bio = bio;
}
read_bio = bio_alloc_clone(rdev->bdev, bio, gfp, &mddev->bio_set);
read_bio->bi_opf &= ~REQ_NOWAIT;
r10_bio->devs[slot].bio = read_bio;
r10_bio->devs[slot].rdev = rdev;
@@ -1265,7 +1257,6 @@ static void raid10_write_one_disk(struct mddev *mddev, struct r10bio *r10_bio,
conf->mirrors[devnum].rdev;
mbio = bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, &mddev->bio_set);
mbio->bi_opf &= ~REQ_NOWAIT;
if (replacement)
r10_bio->devs[n_copy].repl_bio = mbio;
else
@@ -1344,7 +1335,7 @@ retry_wait:
"raid10 %s wait rdev %d blocked",
__func__, blocked_rdev->raid_disk);
md_wait_for_blocked_rdev(blocked_rdev, mddev);
wait_barrier(conf, false);
wait_barrier(conf);
goto retry_wait;
}
}
@@ -1356,33 +1347,20 @@ static bool raid10_write_request(struct mddev *mddev, struct bio *bio,
int i, k;
sector_t sectors;
int max_sectors;
bool atomic = bio->bi_opf & REQ_ATOMIC;
if ((mddev_is_clustered(mddev) &&
mddev->cluster_ops->area_resyncing(mddev, WRITE,
bio->bi_iter.bi_sector,
bio_end_sector(bio)))) {
DEFINE_WAIT(w);
/* Bail out if REQ_NOWAIT is set for the bio */
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
return false;
}
for (;;) {
prepare_to_wait(&conf->wait_barrier,
&w, TASK_IDLE);
if (!mddev->cluster_ops->area_resyncing(mddev, WRITE,
bio->bi_iter.bi_sector, bio_end_sector(bio)))
break;
schedule();
}
finish_wait(&conf->wait_barrier, &w);
wait_event_idle(conf->wait_barrier,
!mddev->cluster_ops->area_resyncing(mddev, WRITE,
bio->bi_iter.bi_sector,
bio_end_sector(bio)));
}
sectors = r10_bio->sectors;
if (!regular_request_wait(mddev, conf, bio, sectors)) {
free_r10bio(r10_bio);
return false;
}
regular_request_wait(mddev, conf, bio, sectors);
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
(mddev->reshape_backwards
@@ -1395,11 +1373,6 @@ static bool raid10_write_request(struct mddev *mddev, struct bio *bio,
set_mask_bits(&mddev->sb_flags, 0,
BIT(MD_SB_CHANGE_DEVS) | BIT(MD_SB_CHANGE_PENDING));
md_wakeup_thread(mddev->thread);
if (bio->bi_opf & REQ_NOWAIT) {
allow_barrier(conf);
bio_wouldblock_error(bio);
return false;
}
mddev_add_trace_msg(conf->mddev,
"raid10 wait reshape metadata");
wait_event(mddev->sb_wait,
@@ -1462,16 +1435,6 @@ static bool raid10_write_request(struct mddev *mddev, struct bio *bio,
if (is_bad) {
int good_sectors;
/*
* We cannot atomically write this, so just
* error in that case. It could be possible to
* atomically write other mirrors, but the
* complexity of supporting that is not worth
* the benefit.
*/
if (bio->bi_opf & REQ_ATOMIC)
goto err_handle;
good_sectors = first_bad - dev_sector;
if (good_sectors < max_sectors)
max_sectors = good_sectors;
@@ -1491,10 +1454,11 @@ static bool raid10_write_request(struct mddev *mddev, struct bio *bio,
r10_bio->sectors = max_sectors;
if (r10_bio->sectors < bio_sectors(bio)) {
allow_barrier(conf);
if (atomic)
goto err_handle;
bio = bio_submit_split_bioset(bio, r10_bio->sectors,
&conf->bio_split);
wait_barrier(conf, false);
if (!bio) {
set_bit(R10BIO_Returned, &r10_bio->state);
goto err_handle;
@@ -1542,7 +1506,7 @@ static bool __make_request(struct mddev *mddev, struct bio *bio, int sectors)
struct r10conf *conf = mddev->private;
struct r10bio *r10_bio;
r10_bio = mempool_alloc(&conf->r10bio_pool, GFP_NOIO);
r10_bio = mempool_alloc(conf->r10bio_pool, GFP_NOIO);
r10_bio->master_bio = bio;
r10_bio->sectors = sectors;
@@ -1577,9 +1541,11 @@ static void raid_end_discard_bio(struct r10bio *r10bio)
free_r10bio(r10bio);
r10bio = first_r10bio;
} else {
struct bio *master_bio = r10bio->master_bio;
md_write_end(r10bio->mddev);
bio_endio(r10bio->master_bio);
free_r10bio(r10bio);
bio_endio(master_bio);
break;
}
}
@@ -1637,11 +1603,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery))
return -EAGAIN;
if (!wait_barrier(conf, bio->bi_opf & REQ_NOWAIT)) {
bio_wouldblock_error(bio);
md_write_end(mddev);
return 0;
}
wait_barrier(conf);
/*
* Check reshape again to avoid reshape happens after checking
@@ -1692,7 +1654,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
allow_barrier(conf);
/* Resend the fist split part */
submit_bio_noacct(split);
wait_barrier(conf, false);
wait_barrier(conf);
}
div_u64_rem(bio_end, stripe_size, &remainder);
if (remainder) {
@@ -1712,7 +1674,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
/* Resend the second split part */
submit_bio_noacct(bio);
bio = split;
wait_barrier(conf, false);
wait_barrier(conf);
}
bio_start = bio->bi_iter.bi_sector;
@@ -1742,7 +1704,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
(last_stripe_index << geo->chunk_shift);
retry_discard:
r10_bio = mempool_alloc(&conf->r10bio_pool, GFP_NOIO);
r10_bio = mempool_alloc(conf->r10bio_pool, GFP_NOIO);
r10_bio->mddev = mddev;
r10_bio->state = 0;
r10_bio->sectors = 0;
@@ -1870,7 +1832,7 @@ retry_discard:
end_disk_offset += geo->stride;
atomic_inc(&first_r10bio->remaining);
raid_end_discard_bio(r10_bio);
wait_barrier(conf, false);
wait_barrier(conf);
goto retry_discard;
}
@@ -1886,6 +1848,7 @@ static bool raid10_make_request(struct mddev *mddev, struct bio *bio)
{
struct r10conf *conf = mddev->private;
sector_t chunk_mask = (conf->geo.chunk_mask & conf->prev.chunk_mask);
const int rw = bio_data_dir(bio);
int chunk_sects = chunk_mask + 1;
int sectors = bio_sectors(bio);
@@ -1911,6 +1874,15 @@ static bool raid10_make_request(struct mddev *mddev, struct bio *bio)
sectors = chunk_sects -
(bio->bi_iter.bi_sector &
(chunk_sects - 1));
bio = mddev_bio_split_at_reshape_offset(mddev, bio, &sectors,
&conf->bio_split);
if (!bio) {
if (rw == WRITE)
md_write_end(mddev);
return true;
}
if (!__make_request(mddev, bio, sectors))
md_write_end(mddev);
@@ -2069,7 +2041,7 @@ static void print_conf(struct r10conf *conf)
static void close_sync(struct r10conf *conf)
{
wait_barrier(conf, false);
wait_barrier(conf);
allow_barrier(conf);
mempool_exit(&conf->r10buf_pool);
@@ -3365,7 +3337,7 @@ static sector_t raid10_sync_request(struct mddev *mddev, sector_t sector_nr,
struct md_rdev *rdev = conf->mirrors[j].rdev;
if (rdev == NULL || test_bit(Faulty, &rdev->flags)) {
still_degraded = false;
still_degraded = true;
break;
}
}
@@ -3844,7 +3816,7 @@ static void raid10_free_conf(struct r10conf *conf)
if (!conf)
return;
mempool_exit(&conf->r10bio_pool);
mempool_destroy(conf->r10bio_pool);
kfree(conf->mirrors);
kfree(conf->mirrors_old);
kfree(conf->mirrors_new);
@@ -3891,9 +3863,8 @@ static struct r10conf *setup_conf(struct mddev *mddev)
conf->geo = geo;
conf->copies = copies;
err = mempool_init(&conf->r10bio_pool, NR_RAID_BIOS, r10bio_pool_alloc,
rbio_pool_free, conf);
if (err)
conf->r10bio_pool = create_r10bio_pool(conf->geo.raid_disks);
if (!conf->r10bio_pool)
goto out;
err = bioset_init(&conf->bio_split, BIO_POOL_SIZE, 0, 0);
@@ -4284,6 +4255,10 @@ static int raid10_check_reshape(struct mddev *mddev)
if (conf->geo.far_copies != 1 && !conf->geo.far_offset)
return -EINVAL;
if (mddev->bitmap_id == ID_LLBITMAP &&
mddev->new_chunk_sectors &&
mddev->new_chunk_sectors < mddev->chunk_sectors)
return -EOPNOTSUPP;
if (setup_geo(&geo, mddev, geo_start) != conf->copies)
/* mustn't change number of copies */
@@ -4387,9 +4362,16 @@ static int raid10_start_reshape(struct mddev *mddev)
struct md_rdev *rdev;
int spares = 0;
int ret;
mempool_t *new_pool = NULL;
if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery))
return -EBUSY;
if (md_bitmap_enabled(mddev, false) &&
mddev->bitmap_ops->reshape_can_start) {
ret = mddev->bitmap_ops->reshape_can_start(mddev);
if (ret)
return ret;
}
if (setup_geo(&new, mddev, geo_start) != conf->copies)
return -EINVAL;
@@ -4423,6 +4405,11 @@ static int raid10_start_reshape(struct mddev *mddev)
return -EINVAL;
conf->offset_diff = min_offset_diff;
if (mddev->delta_disks > 0) {
new_pool = create_r10bio_pool(new.raid_disks);
if (!new_pool)
return -ENOMEM;
}
spin_lock_irq(&conf->device_lock);
if (conf->mirrors_new) {
memcpy(conf->mirrors_new, conf->mirrors,
@@ -4523,6 +4510,10 @@ out:
mddev->raid_disks = conf->geo.raid_disks;
mddev->reshape_position = conf->reshape_progress;
set_bit(MD_SB_CHANGE_DEVS, &mddev->sb_flags);
if (new_pool) {
mempool_destroy(conf->r10bio_pool);
conf->r10bio_pool = new_pool;
}
clear_bit(MD_RECOVERY_SYNC, &mddev->recovery);
clear_bit(MD_RECOVERY_CHECK, &mddev->recovery);
@@ -4545,6 +4536,7 @@ abort:
conf->reshape_safe = MaxSector;
mddev->reshape_position = MaxSector;
spin_unlock_irq(&conf->device_lock);
mempool_destroy(new_pool);
return ret;
}
@@ -4702,7 +4694,14 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr,
if (need_flush ||
time_after(jiffies, conf->reshape_checkpoint + 10*HZ)) {
/* Need to update reshape_position in metadata */
wait_barrier(conf, false);
wait_barrier(conf);
if (md_bitmap_enabled(mddev, false) &&
mddev->bitmap_ops->reshape_mark &&
conf->reshape_safe != conf->reshape_progress) {
mddev->bitmap_ops->reshape_mark(mddev, conf->reshape_safe,
conf->reshape_progress);
mddev->bitmap_ops->unplug(mddev, true);
}
mddev->reshape_position = conf->reshape_progress;
if (mddev->reshape_backwards)
mddev->curr_resync_completed = raid10_size(mddev, 0, 0)
@@ -4901,9 +4900,19 @@ static void reshape_request_write(struct mddev *mddev, struct r10bio *r10_bio)
static void end_reshape(struct r10conf *conf)
{
struct mddev *mddev = conf->mddev;
if (test_bit(MD_RECOVERY_INTR, &conf->mddev->recovery))
return;
if (md_bitmap_enabled(mddev, false) &&
mddev->bitmap_ops->reshape_mark &&
conf->reshape_safe != conf->reshape_progress) {
mddev->bitmap_ops->reshape_mark(mddev, conf->reshape_safe,
conf->reshape_progress);
mddev->bitmap_ops->unplug(mddev, true);
}
spin_lock_irq(&conf->device_lock);
conf->prev = conf->geo;
md_finish_reshape(conf->mddev);
@@ -5035,10 +5044,15 @@ static void end_reshape_request(struct r10bio *r10_bio)
static void raid10_finish_reshape(struct mddev *mddev)
{
struct r10conf *conf = mddev->private;
bool llbitmap = mddev->bitmap_id == ID_LLBITMAP &&
md_bitmap_enabled(mddev, false);
if (test_bit(MD_RECOVERY_INTR, &mddev->recovery))
return;
if (llbitmap && mddev->bitmap_ops->reshape_finish)
mddev->bitmap_ops->reshape_finish(mddev);
if (mddev->delta_disks > 0) {
if (mddev->resync_offset > mddev->resync_max_sectors) {
mddev->resync_offset = mddev->resync_max_sectors;
@@ -5065,6 +5079,15 @@ static void raid10_finish_reshape(struct mddev *mddev)
mddev->reshape_backwards = 0;
}
static sector_t raid10_bitmap_sync_size(struct mddev *mddev, bool previous)
{
struct r10conf *conf = mddev->private;
if (previous)
return raid10_size(mddev, 0, 0);
return raid10_size(mddev, 0, conf->geo.raid_disks);
}
static struct md_personality raid10_personality =
{
.head = {
@@ -5091,6 +5114,8 @@ static struct md_personality raid10_personality =
.start_reshape = raid10_start_reshape,
.finish_reshape = raid10_finish_reshape,
.update_reshape_pos = raid10_update_reshape_pos,
.bitmap_sync_size = raid10_bitmap_sync_size,
.bitmap_array_sectors = raid10_bitmap_sync_size,
};
static int __init raid10_init(void)
drivers/md/raid10.h
+1 -1
View File
@@ -87,7 +87,7 @@ struct r10conf {
*/
wait_queue_head_t wait_barrier;
mempool_t r10bio_pool;
mempool_t *r10bio_pool;
mempool_t r10buf_pool;
struct page *tmppage;
struct bio_set bio_split;
drivers/md/raid5-ppl.c
+3 -1
View File
@@ -643,8 +643,10 @@ static void ppl_do_flush(struct ppl_io_unit *io)
log->disk_flush_bitmap = 0;
for (i = flushed_disks ; i < raid_disks; i++) {
if (atomic_dec_and_test(&io->pending_flushes))
if (atomic_dec_and_test(&io->pending_flushes)) {
ppl_io_unit_finished(io);
break;
}
}
}
drivers/md/raid5.c
+246 -72
View File
@@ -801,7 +801,7 @@ static bool is_inactive_blocked(struct r5conf *conf, int hash)
return true;
return (atomic_read(&conf->active_stripes) <
(conf->max_nr_stripes * 3 / 4));
(READ_ONCE(conf->max_nr_stripes) * 3 / 4));
}
struct stripe_head *raid5_get_active_stripe(struct r5conf *conf,
@@ -2471,11 +2471,6 @@ static int scribble_alloc(struct raid5_percpu *percpu,
sizeof(unsigned int) * (num + 2);
void *scribble;
/*
* If here is in raid array suspend context, it is in memalloc noio
* context as well, there is no potential recursive memory reclaim
* I/Os with the GFP_KERNEL flag.
*/
scribble = kvmalloc_array(cnt, obj_size, GFP_KERNEL);
if (!scribble)
return -ENOMEM;
@@ -2490,6 +2485,7 @@ static int scribble_alloc(struct raid5_percpu *percpu,
static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
{
unsigned long cpu;
unsigned int noio_flags;
int err = 0;
/* Never shrink. */
@@ -2498,6 +2494,7 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
return 0;
raid5_quiesce(conf->mddev, true);
noio_flags = memalloc_noio_save();
cpus_read_lock();
for_each_present_cpu(cpu) {
@@ -2511,6 +2508,7 @@ static int resize_chunks(struct r5conf *conf, int new_disks, int new_sectors)
}
cpus_read_unlock();
memalloc_noio_restore(noio_flags);
raid5_quiesce(conf->mddev, false);
if (!err) {
@@ -2785,6 +2783,7 @@ static void raid5_end_read_request(struct bio * bi)
} else {
int retry = 0;
int set_bad = 0;
int max_nr_stripes = READ_ONCE(conf->max_nr_stripes);
clear_bit(R5_UPTODATE, &sh->dev[i].flags);
if (!(bi->bi_status == BLK_STS_PROTECTION))
@@ -2810,13 +2809,12 @@ static void raid5_end_read_request(struct bio * bi)
mdname(conf->mddev),
(unsigned long long)s,
rdev->bdev);
} else if (atomic_read(&rdev->read_errors)
> conf->max_nr_stripes) {
} else if (atomic_read(&rdev->read_errors) > max_nr_stripes) {
if (!test_bit(Faulty, &rdev->flags)) {
pr_warn("md/raid:%s: %d read_errors > %d stripes\n",
mdname(conf->mddev),
atomic_read(&rdev->read_errors),
conf->max_nr_stripes);
max_nr_stripes);
pr_warn("md/raid:%s: Too many read errors, failing device %pg.\n",
mdname(conf->mddev), rdev->bdev);
}
@@ -3553,7 +3551,7 @@ static void __add_stripe_bio(struct stripe_head *sh, struct bio *bi,
sh->dev[dd_idx].sector);
if (conf->mddev->bitmap && firstwrite && !sh->batch_head) {
sh->bm_seq = conf->seq_flush+1;
sh->bm_seq = READ_ONCE(conf->seq_flush) + 1;
set_bit(STRIPE_BIT_DELAY, &sh->state);
}
}
@@ -3745,17 +3743,89 @@ handle_failed_sync(struct r5conf *conf, struct stripe_head *sh,
md_sync_error(conf->mddev);
}
/*
* handle_failed_reshape - handle failed stripes when reshape failed and
* degraded devices >= max_degraded
*
* handle following kinds of stripe:
* 1. cleanup the following kinds of destination stripe:
* - new regions past the old end of the array, zero-filled in place,
* requires no source data.
* (STRIPE_EXPANDING | STRIPE_EXPAND_READY)
* - prepare source data chunks already done, and writeout failed
* (STRIPE_EXPAND_READY)
* 2. dest stripes that need source data (STRIPE_EXPANDING, no STRIPE_HANDLE)
* - these kind of stripes sit idle in the stripe cache and are never seen
* by handle_stripe(). So clean up indirectly when their source stripe
* (type 3) is processed.
* 3. src stripes (STRIPE_EXPAND_SOURCE)
* - hit handle_stripe() after their member disks are marked Faulty.
* - clear STRIPE_EXPAND_SOURCE, finds and cleanup all dependent destination
* stripes that were waiting for data.
* - walks the source's data disks, compute the corresponding destination
* sector, looks up the destination stripe, and do cleanup(clear flags,
* dec counters, call md_done_sync())
*/
static void handle_failed_reshape(struct r5conf *conf, struct stripe_head *sh,
struct stripe_head_state *s)
{
int i;
bool was_expanding = test_and_clear_bit(STRIPE_EXPANDING, &sh->state);
bool was_ready = test_and_clear_bit(STRIPE_EXPAND_READY, &sh->state);
if (was_expanding || was_ready) {
atomic_dec(&conf->reshape_stripes);
wake_up(&conf->wait_for_reshape);
md_done_sync(conf->mddev, RAID5_STRIPE_SECTORS(conf));
}
s->expanded = 0;
s->expanding = 0;
/* release the destination stripes that are waiting to be filled */
if (test_and_clear_bit(STRIPE_EXPAND_SOURCE, &sh->state)) {
for (i = 0; i < sh->disks; i++) {
int dd_idx;
struct stripe_head *sh2;
sector_t bn, sec;
if (i == sh->pd_idx)
continue;
if (conf->level == 6 && i == sh->qd_idx)
continue;
bn = raid5_compute_blocknr(sh, i, 1);
sec = raid5_compute_sector(conf, bn, 0, &dd_idx, NULL);
sh2 = raid5_get_active_stripe(conf, NULL, sec,
R5_GAS_NOBLOCK |
R5_GAS_NOQUIESCE);
if (!sh2)
continue;
if (test_and_clear_bit(STRIPE_EXPANDING, &sh2->state)) {
atomic_dec(&conf->reshape_stripes);
wake_up(&conf->wait_for_reshape);
md_done_sync(conf->mddev,
RAID5_STRIPE_SECTORS(conf));
}
clear_bit(STRIPE_EXPAND_READY, &sh2->state);
raid5_release_stripe(sh2);
}
}
}
static int want_replace(struct stripe_head *sh, int disk_idx)
{
struct md_rdev *rdev;
int rv = 0;
rdev = sh->raid_conf->disks[disk_idx].replacement;
if (rdev
&& !test_bit(Faulty, &rdev->flags)
&& !test_bit(In_sync, &rdev->flags)
&& (rdev->recovery_offset <= sh->sector
|| rdev->mddev->resync_offset <= sh->sector))
if (rdev && !test_bit(Faulty, &rdev->flags) &&
!test_bit(In_sync, &rdev->flags) &&
(READ_ONCE(rdev->recovery_offset) <= sh->sector ||
rdev->mddev->resync_offset <= sh->sector))
rv = 1;
return rv;
}
@@ -4672,7 +4742,8 @@ static void analyse_stripe(struct stripe_head *sh, struct stripe_head_state *s)
*/
rdev = conf->disks[i].replacement;
if (rdev && !test_bit(Faulty, &rdev->flags) &&
rdev->recovery_offset >= sh->sector + RAID5_STRIPE_SECTORS(conf) &&
READ_ONCE(rdev->recovery_offset) >=
sh->sector + RAID5_STRIPE_SECTORS(conf) &&
!rdev_has_badblock(rdev, sh->sector,
RAID5_STRIPE_SECTORS(conf)))
set_bit(R5_ReadRepl, &dev->flags);
@@ -4714,7 +4785,7 @@ static void analyse_stripe(struct stripe_head *sh, struct stripe_head_state *s)
} else if (test_bit(In_sync, &rdev->flags))
set_bit(R5_Insync, &dev->flags);
else if (sh->sector + RAID5_STRIPE_SECTORS(conf) <=
rdev->recovery_offset) {
READ_ONCE(rdev->recovery_offset)) {
/*
* in sync if:
* - normal IO, or
@@ -5025,6 +5096,8 @@ static void handle_stripe(struct stripe_head *sh)
handle_failed_stripe(conf, sh, &s, disks);
if (s.syncing + s.replacing)
handle_failed_sync(conf, sh, &s);
if (s.expanding + s.expanded)
handle_failed_reshape(conf, sh, &s);
}
/* Now we check to see if any write operations have recently
@@ -5458,13 +5531,13 @@ static int raid5_read_one_chunk(struct mddev *mddev, struct bio *raid_bio)
rdev = conf->disks[dd_idx].replacement;
if (!rdev || test_bit(Faulty, &rdev->flags) ||
rdev->recovery_offset < end_sector) {
READ_ONCE(rdev->recovery_offset) < end_sector) {
rdev = conf->disks[dd_idx].rdev;
if (!rdev)
return 0;
if (test_bit(Faulty, &rdev->flags) ||
!(test_bit(In_sync, &rdev->flags) ||
rdev->recovery_offset >= end_sector))
READ_ONCE(rdev->recovery_offset) >= end_sector))
return 0;
}
@@ -5718,13 +5791,8 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
struct bio *orig_bi = bi;
int stripe_sectors;
/* We need to handle this when io_uring supports discard/trim */
if (WARN_ON_ONCE(bi->bi_opf & REQ_NOWAIT))
return;
if (mddev->reshape_position != MaxSector)
/* Skip discard while reshape is happening */
return;
goto complete_again;
if (!raid5_discard_limits(mddev, bi))
return;
@@ -5799,7 +5867,7 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
}
spin_unlock_irq(&sh->stripe_lock);
if (conf->mddev->bitmap) {
sh->bm_seq = conf->seq_flush + 1;
sh->bm_seq = READ_ONCE(conf->seq_flush) + 1;
set_bit(STRIPE_BIT_DELAY, &sh->state);
}
@@ -5811,6 +5879,11 @@ static void make_discard_request(struct mddev *mddev, struct bio *bi)
}
bio_endio(bi);
return;
complete_again:
/* Skip discard while reshape is happening */
bio_endio_status(bi, BLK_STS_AGAIN);
}
static bool ahead_of_reshape(struct mddev *mddev, sector_t sector,
@@ -5942,6 +6015,34 @@ static enum reshape_loc get_reshape_loc(struct mddev *mddev,
return LOC_BEHIND_RESHAPE;
}
static void raid5_bitmap_sector_map(struct mddev *mddev, sector_t *offset,
unsigned long *sectors,
bool previous)
{
struct r5conf *conf = mddev->private;
sector_t start = *offset;
sector_t end = start + *sectors;
int sectors_per_chunk;
int dd_idx;
if (previous)
sectors_per_chunk = conf->prev_chunk_sectors *
(conf->previous_raid_disks - conf->max_degraded);
else
sectors_per_chunk = conf->chunk_sectors *
(conf->raid_disks - conf->max_degraded);
sector_div(start, sectors_per_chunk);
start *= sectors_per_chunk;
if (sector_div(end, sectors_per_chunk))
end++;
end *= sectors_per_chunk;
start = raid5_compute_sector(conf, start, previous, &dd_idx, NULL);
end = raid5_compute_sector(conf, end, previous, &dd_idx, NULL);
*offset = start;
*sectors = end - start;
}
static void raid5_bitmap_sector(struct mddev *mddev, sector_t *offset,
unsigned long *sectors)
{
@@ -5949,18 +6050,11 @@ static void raid5_bitmap_sector(struct mddev *mddev, sector_t *offset,
sector_t start = *offset;
sector_t end = start + *sectors;
sector_t prev_start = start;
sector_t prev_end = end;
int sectors_per_chunk;
unsigned long prev_sectors = end - start;
enum reshape_loc loc;
int dd_idx;
sectors_per_chunk = conf->chunk_sectors *
(conf->raid_disks - conf->max_degraded);
start = round_down(start, sectors_per_chunk);
end = round_up(end, sectors_per_chunk);
start = raid5_compute_sector(conf, start, 0, &dd_idx, NULL);
end = raid5_compute_sector(conf, end, 0, &dd_idx, NULL);
raid5_bitmap_sector_map(mddev, &start, sectors, false);
end = start + *sectors;
/*
* For LOC_INSIDE_RESHAPE, this IO will wait for reshape to make
@@ -5969,17 +6063,10 @@ static void raid5_bitmap_sector(struct mddev *mddev, sector_t *offset,
loc = get_reshape_loc(mddev, conf, prev_start);
if (likely(loc != LOC_AHEAD_OF_RESHAPE)) {
*offset = start;
*sectors = end - start;
return;
}
sectors_per_chunk = conf->prev_chunk_sectors *
(conf->previous_raid_disks - conf->max_degraded);
prev_start = round_down(prev_start, sectors_per_chunk);
prev_end = round_down(prev_end, sectors_per_chunk);
prev_start = raid5_compute_sector(conf, prev_start, 1, &dd_idx, NULL);
prev_end = raid5_compute_sector(conf, prev_end, 1, &dd_idx, NULL);
raid5_bitmap_sector_map(mddev, &prev_start, &prev_sectors, true);
/*
* for LOC_AHEAD_OF_RESHAPE, reshape can make progress before this IO
@@ -5987,7 +6074,7 @@ static void raid5_bitmap_sector(struct mddev *mddev, sector_t *offset,
* we set bits for both.
*/
*offset = min(start, prev_start);
*sectors = max(end, prev_end) - *offset;
*sectors = max(end, prev_start + prev_sectors) - *offset;
}
static enum stripe_result make_stripe_request(struct mddev *mddev,
@@ -6134,9 +6221,11 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
struct r5conf *conf = mddev->private;
const int rw = bio_data_dir(bi);
struct stripe_request_ctx *ctx;
struct bio *front_bio;
sector_t logical_sector;
enum stripe_result res;
int s, stripe_cnt;
bool split = false;
bool on_wq;
if (unlikely(bi->bi_opf & REQ_PREFLUSH)) {
@@ -6170,6 +6259,18 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
return true;
}
front_bio = bi;
bi = mddev_bio_split_at_reshape_offset(mddev, bi, NULL,
&conf->bio_split);
if (!bi) {
if (rw == WRITE)
md_write_end(mddev);
return true;
}
if (bi != front_bio)
split = true;
front_bio = bi;
logical_sector = bi->bi_iter.bi_sector & ~((sector_t)RAID5_STRIPE_SECTORS(conf)-1);
bi->bi_next = NULL;
@@ -6191,15 +6292,6 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
pr_debug("raid456: %s, logical %llu to %llu\n", __func__,
bi->bi_iter.bi_sector, ctx->last_sector);
/* Bail out if conflicts with reshape and REQ_NOWAIT is set */
if ((bi->bi_opf & REQ_NOWAIT) &&
get_reshape_loc(mddev, conf, logical_sector) == LOC_INSIDE_RESHAPE) {
bio_wouldblock_error(bi);
if (rw == WRITE)
md_write_end(mddev);
mempool_free(ctx, conf->ctx_pool);
return true;
}
md_account_bio(mddev, &bi);
/*
@@ -6270,6 +6362,11 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
bio_endio(bi);
wait_for_completion(&done);
front_bio->bi_status = BLK_STS_OK;
if (split) {
submit_bio_noacct(front_bio);
return true;
}
return false;
}
@@ -6419,6 +6516,13 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr, int *sk
|| test_bit(MD_RECOVERY_INTR, &mddev->recovery));
if (atomic_read(&conf->reshape_stripes) != 0)
return 0;
if (md_bitmap_enabled(mddev, false) &&
mddev->bitmap_ops->reshape_mark &&
conf->reshape_safe != conf->reshape_progress) {
mddev->bitmap_ops->reshape_mark(mddev, conf->reshape_safe,
conf->reshape_progress);
mddev->bitmap_ops->unplug(mddev, true);
}
mddev->reshape_position = conf->reshape_progress;
mddev->curr_resync_completed = sector_nr;
if (!mddev->reshape_backwards)
@@ -6427,8 +6531,8 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr, int *sk
if (rdev->raid_disk >= 0 &&
!test_bit(Journal, &rdev->flags) &&
!test_bit(In_sync, &rdev->flags) &&
rdev->recovery_offset < sector_nr)
rdev->recovery_offset = sector_nr;
READ_ONCE(rdev->recovery_offset) < sector_nr)
WRITE_ONCE(rdev->recovery_offset, sector_nr);
conf->reshape_checkpoint = jiffies;
set_bit(MD_SB_CHANGE_DEVS, &mddev->sb_flags);
@@ -6528,6 +6632,13 @@ finish:
|| test_bit(MD_RECOVERY_INTR, &mddev->recovery));
if (atomic_read(&conf->reshape_stripes) != 0)
goto ret;
if (md_bitmap_enabled(mddev, false) &&
mddev->bitmap_ops->reshape_mark &&
conf->reshape_safe != conf->reshape_progress) {
mddev->bitmap_ops->reshape_mark(mddev, conf->reshape_safe,
conf->reshape_progress);
mddev->bitmap_ops->unplug(mddev, true);
}
mddev->reshape_position = conf->reshape_progress;
mddev->curr_resync_completed = sector_nr;
if (!mddev->reshape_backwards)
@@ -6536,8 +6647,8 @@ finish:
if (rdev->raid_disk >= 0 &&
!test_bit(Journal, &rdev->flags) &&
!test_bit(In_sync, &rdev->flags) &&
rdev->recovery_offset < sector_nr)
rdev->recovery_offset = sector_nr;
READ_ONCE(rdev->recovery_offset) < sector_nr)
WRITE_ONCE(rdev->recovery_offset, sector_nr);
conf->reshape_checkpoint = jiffies;
set_bit(MD_SB_CHANGE_DEVS, &mddev->sb_flags);
md_wakeup_thread(mddev->thread);
@@ -6849,12 +6960,14 @@ static void raid5d(struct md_thread *thread)
if (
!list_empty(&conf->bitmap_list)) {
/* Now is a good time to flush some bitmap updates */
conf->seq_flush++;
int seq = conf->seq_flush + 1;
WRITE_ONCE(conf->seq_flush, seq);
spin_unlock_irq(&conf->device_lock);
if (md_bitmap_enabled(mddev, true))
mddev->bitmap_ops->unplug(mddev, true);
spin_lock_irq(&conf->device_lock);
conf->seq_write = conf->seq_flush;
conf->seq_write = seq;
activate_bit_delay(conf, conf->temp_inactive_list);
}
raid5_activate_delayed(conf);
@@ -7039,6 +7152,7 @@ raid5_store_stripe_size(struct mddev *mddev, const char *page, size_t len)
{
struct r5conf *conf;
unsigned long new;
unsigned int noio_flags = 0;
int err;
int size;
@@ -7079,6 +7193,7 @@ raid5_store_stripe_size(struct mddev *mddev, const char *page, size_t len)
goto out_unlock;
}
noio_flags = memalloc_noio_save();
mutex_lock(&conf->cache_size_mutex);
size = conf->max_nr_stripes;
@@ -7095,6 +7210,7 @@ raid5_store_stripe_size(struct mddev *mddev, const char *page, size_t len)
mutex_unlock(&conf->cache_size_mutex);
out_unlock:
memalloc_noio_restore(noio_flags);
mddev_unlock_and_resume(mddev);
return err ?: len;
}
@@ -8056,9 +8172,9 @@ static int raid5_run(struct mddev *mddev)
/* Hack because v0.91 doesn't store recovery_offset properly. */
if (mddev->major_version == 0 &&
mddev->minor_version > 90)
rdev->recovery_offset = reshape_offset;
WRITE_ONCE(rdev->recovery_offset, reshape_offset);
if (rdev->recovery_offset < reshape_offset) {
if (READ_ONCE(rdev->recovery_offset) < reshape_offset) {
/* We need to check old and new layout */
if (!only_parity(rdev->raid_disk,
conf->algorithm,
@@ -8213,10 +8329,10 @@ static int raid5_spare_active(struct mddev *mddev)
for (i = 0; i < conf->raid_disks; i++) {
rdev = conf->disks[i].rdev;
replacement = conf->disks[i].replacement;
if (replacement
&& replacement->recovery_offset == MaxSector
&& !test_bit(Faulty, &replacement->flags)
&& !test_and_set_bit(In_sync, &replacement->flags)) {
if (replacement &&
READ_ONCE(replacement->recovery_offset) == MaxSector &&
!test_bit(Faulty, &replacement->flags) &&
!test_and_set_bit(In_sync, &replacement->flags)) {
/* Replacement has just become active. */
if (!rdev
|| !test_and_clear_bit(In_sync, &rdev->flags))
@@ -8231,10 +8347,10 @@ static int raid5_spare_active(struct mddev *mddev)
rdev->sysfs_state);
}
sysfs_notify_dirent_safe(replacement->sysfs_state);
} else if (rdev
&& rdev->recovery_offset == MaxSector
&& !test_bit(Faulty, &rdev->flags)
&& !test_and_set_bit(In_sync, &rdev->flags)) {
} else if (rdev &&
READ_ONCE(rdev->recovery_offset) == MaxSector &&
!test_bit(Faulty, &rdev->flags) &&
!test_and_set_bit(In_sync, &rdev->flags)) {
count++;
sysfs_notify_dirent_safe(rdev->sysfs_state);
}
@@ -8479,6 +8595,8 @@ static int check_reshape(struct mddev *mddev)
return 0; /* nothing to do */
if (has_failed(conf))
return -EINVAL;
if (!mddev->new_chunk_sectors)
return -EINVAL;
if (mddev->delta_disks < 0 && mddev->reshape_position == MaxSector) {
/* We might be able to shrink, but the devices must
* be made bigger first.
@@ -8495,6 +8613,9 @@ static int check_reshape(struct mddev *mddev)
if (!check_stripe_cache(mddev))
return -ENOSPC;
if (mddev->bitmap_id == ID_LLBITMAP &&
mddev->new_chunk_sectors < mddev->chunk_sectors)
return -EOPNOTSUPP;
if (mddev->new_chunk_sectors > mddev->chunk_sectors ||
mddev->delta_disks > 0)
if (resize_chunks(conf,
@@ -8522,6 +8643,9 @@ static int raid5_start_reshape(struct mddev *mddev)
if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery))
return -EBUSY;
if (!mddev->new_chunk_sectors)
return -EINVAL;
if (!check_stripe_cache(mddev))
return -ENOSPC;
@@ -8557,6 +8681,12 @@ static int raid5_start_reshape(struct mddev *mddev)
mdname(mddev));
return -EINVAL;
}
if (md_bitmap_enabled(mddev, false) &&
mddev->bitmap_id == ID_LLBITMAP) {
i = mddev->bitmap_ops->resize(mddev, mddev->dev_sectors, 0);
if (i)
return i;
}
atomic_set(&conf->reshape_stripes, 0);
spin_lock_irq(&conf->device_lock);
@@ -8603,7 +8733,7 @@ static int raid5_start_reshape(struct mddev *mddev)
>= conf->previous_raid_disks)
set_bit(In_sync, &rdev->flags);
else
rdev->recovery_offset = 0;
WRITE_ONCE(rdev->recovery_offset, 0);
/* Failure here is OK */
sysfs_link_rdev(mddev, rdev);
@@ -8641,10 +8771,19 @@ static int raid5_start_reshape(struct mddev *mddev)
*/
static void end_reshape(struct r5conf *conf)
{
struct mddev *mddev = conf->mddev;
if (!test_bit(MD_RECOVERY_INTR, &conf->mddev->recovery)) {
struct md_rdev *rdev;
if (md_bitmap_enabled(mddev, false) &&
mddev->bitmap_ops->reshape_mark &&
conf->reshape_safe != conf->reshape_progress) {
mddev->bitmap_ops->reshape_mark(mddev, conf->reshape_safe,
conf->reshape_progress);
mddev->bitmap_ops->unplug(mddev, true);
}
spin_lock_irq(&conf->device_lock);
conf->previous_raid_disks = conf->raid_disks;
md_finish_reshape(conf->mddev);
@@ -8655,7 +8794,7 @@ static void end_reshape(struct r5conf *conf)
if (rdev->raid_disk >= 0 &&
!test_bit(Journal, &rdev->flags) &&
!test_bit(In_sync, &rdev->flags))
rdev->recovery_offset = MaxSector;
WRITE_ONCE(rdev->recovery_offset, MaxSector);
spin_unlock_irq(&conf->device_lock);
wake_up(&conf->wait_for_reshape);
@@ -8671,8 +8810,16 @@ static void raid5_finish_reshape(struct mddev *mddev)
{
struct r5conf *conf = mddev->private;
struct md_rdev *rdev;
bool llbitmap = mddev->bitmap_id == ID_LLBITMAP &&
md_bitmap_enabled(mddev, false);
if (!test_bit(MD_RECOVERY_INTR, &mddev->recovery)) {
if (llbitmap && mddev->bitmap_ops->reshape_finish)
mddev->bitmap_ops->reshape_finish(mddev);
if (llbitmap) {
mddev->resync_offset = 0;
mddev->resync_max_sectors = mddev->dev_sectors;
}
if (mddev->delta_disks <= 0) {
int d;
@@ -8975,6 +9122,7 @@ static void *raid6_takeover(struct mddev *mddev)
static int raid5_change_consistency_policy(struct mddev *mddev, const char *buf)
{
struct r5conf *conf;
unsigned int noio_flags;
int err;
err = mddev_suspend_and_lock(mddev);
@@ -8986,6 +9134,7 @@ static int raid5_change_consistency_policy(struct mddev *mddev, const char *buf)
return -ENODEV;
}
noio_flags = memalloc_noio_save();
if (strncmp(buf, "ppl", 3) == 0) {
/* ppl only works with RAID 5 */
if (!raid5_has_ppl(conf) && conf->level == 5) {
@@ -9025,6 +9174,7 @@ static int raid5_change_consistency_policy(struct mddev *mddev, const char *buf)
if (!err)
md_update_sb(mddev, 1);
memalloc_noio_restore(noio_flags);
mddev_unlock_and_resume(mddev);
return err;
@@ -9049,6 +9199,21 @@ static void raid5_prepare_suspend(struct mddev *mddev)
wake_up(&conf->wait_for_reshape);
}
static sector_t raid5_bitmap_sync_size(struct mddev *mddev, bool previous)
{
return mddev->dev_sectors;
}
static sector_t raid5_bitmap_array_sectors(struct mddev *mddev, bool previous)
{
struct r5conf *conf = mddev->private;
if (previous)
return raid5_size(mddev, mddev->dev_sectors,
conf->previous_raid_disks);
return raid5_size(mddev, mddev->dev_sectors, conf->raid_disks);
}
static struct md_personality raid6_personality =
{
.head = {
@@ -9078,6 +9243,9 @@ static struct md_personality raid6_personality =
.change_consistency_policy = raid5_change_consistency_policy,
.prepare_suspend = raid5_prepare_suspend,
.bitmap_sector = raid5_bitmap_sector,
.bitmap_sector_map = raid5_bitmap_sector_map,
.bitmap_sync_size = raid5_bitmap_sync_size,
.bitmap_array_sectors = raid5_bitmap_array_sectors,
};
static struct md_personality raid5_personality =
{
@@ -9108,6 +9276,9 @@ static struct md_personality raid5_personality =
.change_consistency_policy = raid5_change_consistency_policy,
.prepare_suspend = raid5_prepare_suspend,
.bitmap_sector = raid5_bitmap_sector,
.bitmap_sector_map = raid5_bitmap_sector_map,
.bitmap_sync_size = raid5_bitmap_sync_size,
.bitmap_array_sectors = raid5_bitmap_array_sectors,
};
static struct md_personality raid4_personality =
@@ -9139,6 +9310,9 @@ static struct md_personality raid4_personality =
.change_consistency_policy = raid5_change_consistency_policy,
.prepare_suspend = raid5_prepare_suspend,
.bitmap_sector = raid5_bitmap_sector,
.bitmap_sector_map = raid5_bitmap_sector_map,
.bitmap_sync_size = raid5_bitmap_sync_size,
.bitmap_array_sectors = raid5_bitmap_array_sectors,
};
static int __init raid5_init(void)
drivers/nvdimm/btt.c
+1 -1
View File
@@ -1163,7 +1163,7 @@ static int btt_rw_integrity(struct btt *btt, struct bio_integrity_payload *bip,
bv = bvec_iter_bvec(bip->bip_vec, bip->bip_iter);
/*
* The 'bv' obtained from bvec_iter_bvec has its .bv_len and
* .bv_offset already adjusted for iter->bi_bvec_done, and we
* .bv_offset already adjusted for iter->bi_offset, and we
* can use those directly
*/
drivers/nvme/common/auth.c
+6 -7
View File
@@ -692,8 +692,7 @@ int nvme_auth_derive_tls_psk(int hmac_id, const u8 *psk, size_t psk_len,
const char *psk_digest, u8 **ret_psk)
{
static const u8 default_salt[NVME_AUTH_MAX_DIGEST_SIZE];
static const char label[] = "tls13 nvme-tls-psk";
const size_t label_len = sizeof(label) - 1;
static const char label[18] = "tls13 nvme-tls-psk";
u8 prk[NVME_AUTH_MAX_DIGEST_SIZE];
size_t hash_len, ctx_len;
u8 *hmac_data = NULL, *tls_key;
@@ -729,7 +728,7 @@ int nvme_auth_derive_tls_psk(int hmac_id, const u8 *psk, size_t psk_len,
*/
hmac_data = kmalloc(/* output length */ 2 +
/* label */ 1 + label_len +
/* label */ 1 + sizeof(label) +
/* context (max) */ 1 + 3 + 1 + strlen(psk_digest) +
/* counter */ 1,
GFP_KERNEL);
@@ -743,10 +742,10 @@ int nvme_auth_derive_tls_psk(int hmac_id, const u8 *psk, size_t psk_len,
hmac_data[i++] = hash_len;
/* label */
static_assert(label_len <= 255);
hmac_data[i] = label_len;
memcpy(&hmac_data[i + 1], label, label_len);
i += 1 + label_len;
static_assert(sizeof(label) <= 255);
hmac_data[i] = sizeof(label);
memcpy(&hmac_data[i + 1], label, sizeof(label));
i += 1 + sizeof(label);
/* context */
ctx_len = sprintf(&hmac_data[i + 1], "%02d %s", hmac_id, psk_digest);
drivers/nvme/host/Makefile
+1
View File
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: GPL-2.0
CONTEXT_ANALYSIS := y
ccflags-y += -I$(src)
obj-$(CONFIG_NVME_CORE) += nvme-core.o
drivers/nvme/host/apple.c
+39 -27
View File
@@ -47,9 +47,6 @@
#define APPLE_ANS_BOOT_STATUS 0x1300
#define APPLE_ANS_BOOT_STATUS_OK 0xde71ce55
#define APPLE_ANS_UNKNOWN_CTRL 0x24008
#define APPLE_ANS_PRP_NULL_CHECK BIT(11)
#define APPLE_ANS_LINEAR_SQ_CTRL 0x24908
#define APPLE_ANS_LINEAR_SQ_EN BIT(0)
@@ -151,6 +148,23 @@ struct apple_nvme_queue {
bool enabled;
};
static inline bool apple_nvme_queue_enabled(struct apple_nvme_queue *q)
{
/* Pair with apple_nvme_enable_queue(). */
return smp_load_acquire(&q->enabled);
}
static inline void apple_nvme_enable_queue(struct apple_nvme_queue *q)
{
/* Publish queue initialization before setting q->enabled. */
smp_store_release(&q->enabled, true);
}
static inline void apple_nvme_disable_queue(struct apple_nvme_queue *q)
{
WRITE_ONCE(q->enabled, false);
}
/*
* The apple_nvme_iod describes the data in an I/O.
*
@@ -318,13 +332,15 @@ static void apple_nvme_submit_cmd_t8103(struct apple_nvme_queue *q,
u32 tag = nvme_tag_from_cid(cmd->common.command_id);
struct apple_nvmmu_tcb *tcb = &q->tcbs[tag];
tcb->opcode = cmd->common.opcode;
tcb->opcode = 0;
tcb->prp1 = cmd->common.dptr.prp1;
tcb->prp2 = cmd->common.dptr.prp2;
tcb->length = cmd->rw.length;
tcb->command_id = tag;
if (nvme_is_write(cmd))
if (!cmd->common.dptr.prp1)
tcb->dma_flags = 0;
else if (nvme_is_write(cmd))
tcb->dma_flags = APPLE_ANS_TCB_DMA_TO_DEVICE;
else
tcb->dma_flags = APPLE_ANS_TCB_DMA_FROM_DEVICE;
@@ -677,7 +693,7 @@ static bool apple_nvme_handle_cq(struct apple_nvme_queue *q, bool force)
bool found;
DEFINE_IO_COMP_BATCH(iob);
if (!READ_ONCE(q->enabled) && !force)
if (!apple_nvme_queue_enabled(q) && !force)
return false;
found = apple_nvme_poll_cq(q, &iob);
@@ -780,7 +796,7 @@ static blk_status_t apple_nvme_queue_rq(struct blk_mq_hw_ctx *hctx,
* We should not need to do this, but we're still using this to
* ensure we can drain requests on a dying queue.
*/
if (unlikely(!READ_ONCE(q->enabled)))
if (unlikely(!apple_nvme_queue_enabled(q)))
return BLK_STS_IOERR;
if (!nvme_check_ready(&anv->ctrl, req, true))
@@ -863,7 +879,7 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown)
nvme_quiesce_io_queues(&anv->ctrl);
if (!dead) {
if (READ_ONCE(anv->ioq.enabled)) {
if (apple_nvme_queue_enabled(&anv->ioq)) {
apple_nvme_remove_sq(anv);
apple_nvme_remove_cq(anv);
}
@@ -887,8 +903,8 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown)
nvme_disable_ctrl(&anv->ctrl, false);
}
WRITE_ONCE(anv->ioq.enabled, false);
WRITE_ONCE(anv->adminq.enabled, false);
apple_nvme_disable_queue(&anv->ioq);
apple_nvme_disable_queue(&anv->adminq);
mb(); /* ensure that nvme_queue_rq() sees that enabled is cleared */
nvme_quiesce_admin_queue(&anv->ctrl);
@@ -1016,8 +1032,7 @@ static void apple_nvme_init_queue(struct apple_nvme_queue *q)
memset(q->tcbs, 0, anv->hw->max_queue_depth
* sizeof(struct apple_nvmmu_tcb));
memset(q->cqes, 0, depth * sizeof(struct nvme_completion));
WRITE_ONCE(q->enabled, true);
wmb(); /* ensure the first interrupt sees the initialization */
apple_nvme_enable_queue(q);
}
static void apple_nvme_reset_work(struct work_struct *work)
@@ -1125,17 +1140,6 @@ static void apple_nvme_reset_work(struct work_struct *work)
/* Setup the NVMMU for the maximum admin and IO queue depth */
writel(anv->hw->max_queue_depth - 1,
anv->mmio_nvme + APPLE_NVMMU_NUM_TCBS);
/*
* This is probably a chicken bit: without it all commands
* where any PRP is set to zero (including those that don't use
* that field) fail and the co-processor complains about
* "completed with err BAD_CMD-" or a "NULL_PRP_PTR_ERR" in the
* syslog
*/
writel(readl(anv->mmio_nvme + APPLE_ANS_UNKNOWN_CTRL) &
~APPLE_ANS_PRP_NULL_CHECK,
anv->mmio_nvme + APPLE_ANS_UNKNOWN_CTRL);
}
/* Setup the admin queue */
@@ -1567,10 +1571,8 @@ static struct apple_nvme *apple_nvme_alloc(struct platform_device *pdev)
ret = devm_request_irq(anv->dev, anv->irq, apple_nvme_irq, 0,
"nvme-apple", anv);
if (ret) {
dev_err_probe(dev, ret, "Failed to request IRQ");
if (ret)
goto put_dev;
}
anv->rtk =
devm_apple_rtkit_init(dev, anv, NULL, 0, &apple_nvme_rtkit_ops);
@@ -1581,7 +1583,8 @@ static struct apple_nvme *apple_nvme_alloc(struct platform_device *pdev)
}
ret = nvme_init_ctrl(&anv->ctrl, anv->dev, &nvme_ctrl_ops,
NVME_QUIRK_SKIP_CID_GEN | NVME_QUIRK_IDENTIFY_CNS);
NVME_QUIRK_SKIP_CID_GEN | NVME_QUIRK_IDENTIFY_CNS |
NVME_QUIRK_ADMIN_PAGE_ALIGN);
if (ret) {
dev_err_probe(dev, ret, "Failed to initialize nvme_ctrl");
goto put_dev;
@@ -1636,6 +1639,15 @@ static void apple_nvme_remove(struct platform_device *pdev)
nvme_stop_ctrl(&anv->ctrl);
nvme_remove_namespaces(&anv->ctrl);
apple_nvme_disable(anv, true);
if (anv->ctrl.admin_q && !blk_queue_dying(anv->ctrl.admin_q)) {
/*
* If the controller was reset during removal, it's possible
* user requests may be waiting on a stopped queue. Start the
* queue to flush these to completion.
*/
nvme_unquiesce_admin_queue(&anv->ctrl);
blk_mq_destroy_queue(anv->ctrl.admin_q);
}
nvme_uninit_ctrl(&anv->ctrl);
if (apple_rtkit_is_running(anv->rtk)) {
drivers/nvme/host/auth.c
+2 -1
View File
@@ -8,6 +8,7 @@
#include <linux/prandom.h>
#include <linux/unaligned.h>
#include <crypto/dh.h>
#include <crypto/utils.h>
#include "nvme.h"
#include "fabrics.h"
#include <linux/nvme-auth.h>
@@ -361,7 +362,7 @@ static int nvme_auth_process_dhchap_success1(struct nvme_ctrl *ctrl,
return 0;
/* Validate controller response */
if (memcmp(chap->response, data->rval, data->hl)) {
if (crypto_memneq(chap->response, data->rval, data->hl)) {
dev_dbg(ctrl->device, "%s: qid %d ctrl response %*ph\n",
__func__, chap->qid, (int)chap->hash_len, data->rval);
dev_dbg(ctrl->device, "%s: qid %d host response %*ph\n",
drivers/nvme/host/core.c
+45 -7
View File
@@ -33,6 +33,13 @@
#define NVME_MINORS (1U << MINORBITS)
/*
* Write hints (bio->bi_write_stream) are u8, so FDP placement handles beyond
* U8_MAX can never be selected. Cap the handle count to bound both the RUH
* status buffer and the per-head plids array.
*/
#define NVME_MAX_PLIDS U8_MAX
struct nvme_ns_info {
struct nvme_ns_ids ids;
u32 nsid;
@@ -126,8 +133,8 @@ EXPORT_SYMBOL_GPL(nvme_reset_wq);
struct workqueue_struct *nvme_delete_wq;
EXPORT_SYMBOL_GPL(nvme_delete_wq);
static LIST_HEAD(nvme_subsystems);
DEFINE_MUTEX(nvme_subsystems_lock);
static LIST_HEAD_GUARDED(nvme_subsystems, nvme_subsystems_lock);
static DEFINE_IDA(nvme_instance_ida);
static dev_t nvme_ctrl_base_chr_devt;
@@ -693,6 +700,11 @@ static void nvme_free_ns_head(struct kref *ref)
kfree(head);
}
void nvme_get_ns_head(struct nvme_ns_head *head)
{
kref_get(&head->ref);
}
bool nvme_tryget_ns_head(struct nvme_ns_head *head)
{
return kref_get_unless_zero(&head->ref);
@@ -1268,7 +1280,7 @@ u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u8 opcode)
}
EXPORT_SYMBOL_NS_GPL(nvme_passthru_start, "NVME_TARGET_PASSTHRU");
void nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
u32 nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
struct nvme_command *cmd, int status)
{
if (effects & NVME_CMD_EFFECTS_CSE_MASK) {
@@ -1289,7 +1301,7 @@ void nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
flush_work(&ctrl->scan_work);
}
if (ns)
return;
return effects;
switch (cmd->common.opcode) {
case nvme_admin_set_features:
@@ -1310,6 +1322,8 @@ void nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
default:
break;
}
return effects;
}
EXPORT_SYMBOL_NS_GPL(nvme_passthru_end, "NVME_TARGET_PASSTHRU");
@@ -1583,8 +1597,12 @@ static int nvme_identify_ns_descs(struct nvme_ctrl *ctrl,
for (pos = 0; pos < NVME_IDENTIFY_DATA_SIZE; pos += len) {
struct nvme_ns_id_desc *cur = data + pos;
if (pos + sizeof(*cur) > NVME_IDENTIFY_DATA_SIZE)
break;
if (cur->nidl == 0)
break;
if (pos + sizeof(*cur) + cur->nidl > NVME_IDENTIFY_DATA_SIZE)
break;
len = nvme_process_ns_desc(ctrl, &info->ids, cur, &csi_seen);
if (len < 0)
@@ -2071,7 +2089,10 @@ static void nvme_set_ctrl_limits(struct nvme_ctrl *ctrl,
lim->max_integrity_segments = ctrl->max_integrity_segments;
lim->virt_boundary_mask = ctrl->ops->get_virt_boundary(ctrl, is_admin);
lim->max_segment_size = UINT_MAX;
lim->dma_alignment = 3;
if (is_admin && (ctrl->quirks & NVME_QUIRK_ADMIN_PAGE_ALIGN))
lim->dma_alignment = NVME_CTRL_PAGE_SIZE - 1;
else
lim->dma_alignment = 3;
}
static bool nvme_update_disk_info(struct nvme_ns *ns, struct nvme_id_ns *id,
@@ -2342,7 +2363,7 @@ static int nvme_query_fdp_info(struct nvme_ns *ns, struct nvme_ns_info *info)
if (!info->runs)
return ret;
size = struct_size(ruhs, ruhsd, S8_MAX - 1);
size = struct_size(ruhs, ruhsd, NVME_MAX_PLIDS);
ruhs = kzalloc(size, GFP_KERNEL);
if (!ruhs)
return -ENOMEM;
@@ -2357,7 +2378,7 @@ static int nvme_query_fdp_info(struct nvme_ns *ns, struct nvme_ns_info *info)
goto free;
}
head->nr_plids = le16_to_cpu(ruhs->nruhsd);
head->nr_plids = min(le16_to_cpu(ruhs->nruhsd), NVME_MAX_PLIDS);
if (!head->nr_plids)
goto free;
@@ -2592,11 +2613,15 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info)
lim.max_write_streams = ns_lim->max_write_streams;
lim.write_stream_granularity = ns_lim->write_stream_granularity;
ret = queue_limits_commit_update(ns->head->disk->queue, &lim);
if (ret)
goto unfreeze_head_queue;
set_capacity_and_notify(ns->head->disk, get_capacity(ns->disk));
set_disk_ro(ns->head->disk, nvme_ns_is_readonly(ns, info));
nvme_mpath_revalidate_paths(ns->head);
ret = nvme_mpath_revalidate_zones(ns->head);
unfreeze_head_queue:
blk_mq_unfreeze_queue(ns->head->disk->queue, memflags);
}
@@ -3198,6 +3223,7 @@ static void nvme_put_subsystem(struct nvme_subsystem *subsys)
}
static struct nvme_subsystem *__nvme_find_get_subsystem(const char *subsysnqn)
__must_hold(&nvme_subsystems_lock)
{
struct nvme_subsystem *subsys;
@@ -3242,6 +3268,7 @@ static inline bool nvme_is_io_ctrl(struct nvme_ctrl *ctrl)
static bool nvme_validate_cntlid(struct nvme_subsystem *subsys,
struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
__must_hold(&nvme_subsystems_lock)
{
struct nvme_ctrl *tmp;
@@ -3272,6 +3299,7 @@ static bool nvme_validate_cntlid(struct nvme_subsystem *subsys,
}
static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id)
__context_unsafe(/* initialize unpublished/lock-guarded variables */)
{
struct nvme_subsystem *subsys, *found;
int ret;
@@ -3843,6 +3871,7 @@ static const struct file_operations nvme_dev_fops = {
static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl,
unsigned nsid)
__must_hold(&ctrl->subsys->lock)
{
struct nvme_ns_head *h;
@@ -3865,6 +3894,7 @@ static struct nvme_ns_head *nvme_find_ns_head(struct nvme_ctrl *ctrl,
static int nvme_subsys_check_duplicate_ids(struct nvme_subsystem *subsys,
struct nvme_ns_ids *ids)
__must_hold(&subsys->lock)
{
bool has_uuid = !uuid_is_null(&ids->uuid);
bool has_nguid = memchr_inv(ids->nguid, 0, sizeof(ids->nguid));
@@ -3890,6 +3920,11 @@ static int nvme_subsys_check_duplicate_ids(struct nvme_subsystem *subsys,
static void nvme_cdev_rel(struct device *dev)
{
ida_free(&nvme_ns_chr_minor_ida, MINOR(dev->devt));
if (dev->parent->class == &nvme_class)
nvme_put_ns(container_of(dev, struct nvme_ns, cdev_device));
else
nvme_put_ns_head(container_of(dev, struct nvme_ns_head,
cdev_device));
}
void nvme_cdev_del(struct cdev *cdev, struct device *cdev_device)
@@ -3955,10 +3990,12 @@ static void nvme_add_ns_cdev(struct nvme_ns *ns)
snprintf(name, sizeof(name), "ng%dn%d", ns->ctrl->instance,
ns->head->instance);
nvme_get_ns(ns); /* Undone in nvme_cdev_rel() */
if (nvme_cdev_add(name, &ns->cdev, &ns->cdev_device,
&nvme_ns_chr_fops, ns->ctrl->ops->module)) {
dev_err(ns->ctrl->device, "Unable to create the %s device\n",
name);
nvme_put_ns(ns);
return;
}
set_bit(NVME_NS_CDEV_LIVE, &ns->flags);
@@ -3966,6 +4003,7 @@ static void nvme_add_ns_cdev(struct nvme_ns *ns)
static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl,
struct nvme_ns_info *info)
__must_hold(&ctrl->subsys->lock)
{
struct nvme_ns_head *head;
size_t size = sizeof(*head);
@@ -5195,7 +5233,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,
BUILD_BUG_ON(NVME_DSM_MAX_RANGES * sizeof(struct nvme_dsm_range) >
PAGE_SIZE);
ctrl->discard_page = alloc_page(GFP_KERNEL);
ctrl->discard_page = alloc_page(GFP_KERNEL | __GFP_ZERO);
if (!ctrl->discard_page) {
ret = -ENOMEM;
goto out;
drivers/nvme/host/fabrics.c
+2 -2
View File
@@ -14,11 +14,11 @@
#include "fabrics.h"
#include <linux/nvme-keyring.h>
static LIST_HEAD(nvmf_transports);
static DECLARE_RWSEM(nvmf_transports_rwsem);
static LIST_HEAD_GUARDED(nvmf_transports, nvmf_transports_rwsem);
static LIST_HEAD(nvmf_hosts);
static DEFINE_MUTEX(nvmf_hosts_mutex);
static LIST_HEAD_GUARDED(nvmf_hosts, nvmf_hosts_mutex);
static struct nvmf_host *nvmf_default_host;
drivers/nvme/host/fc.c
+7 -1
View File
@@ -2100,9 +2100,15 @@ __nvme_fc_init_request(struct nvme_fc_ctrl *ctrl,
dev_err(ctrl->dev,
"FCP Op failed - rspiu dma mapping failed.\n");
ret = -EFAULT;
goto out_unmap;
}
atomic_set(&op->state, FCPOP_STATE_IDLE);
return 0;
out_unmap:
fc_dma_unmap_single(ctrl->lport->dev, op->fcp_req.cmddma,
sizeof(op->cmd_iu), DMA_TO_DEVICE);
out_on_error:
return ret;
}
@@ -2318,7 +2324,7 @@ nvme_fc_create_hw_io_queues(struct nvme_fc_ctrl *ctrl, u16 qsize)
return 0;
delete_queues:
for (; i > 0; i--)
for (--i; i > 0; i--)
__nvme_fc_delete_hw_queue(ctrl, &ctrl->queues[i], i);
return ret;
}
drivers/nvme/host/ioctl.c
+79 -42
View File
@@ -14,45 +14,54 @@ enum {
NVME_IOCTL_PARTITION = (1 << 1),
};
static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
unsigned int flags, bool open_for_write)
static bool nvme_admin_cmd_allowed(struct nvme_ctrl *ctrl,
struct nvme_command *c)
{
u32 effects;
/*
* Do not allow unprivileged passthrough on partitions, as that allows an
* escape from the containment of the partition.
*/
if (flags & NVME_IOCTL_PARTITION)
goto admin;
/*
* Do not allow unprivileged processes to send vendor specific or fabrics
* commands as we can't be sure about their effects.
*/
if (c->common.opcode >= nvme_cmd_vendor_start ||
c->common.opcode == nvme_fabrics_command)
goto admin;
/*
* Do not allow unprivileged passthrough of admin commands except
* for a subset of identify commands that contain information required
* to form proper I/O commands in userspace and do not expose any
* potentially sensitive information.
*/
if (!ns) {
if (c->common.opcode == nvme_admin_identify) {
switch (c->identify.cns) {
case NVME_ID_CNS_NS:
case NVME_ID_CNS_CS_NS:
case NVME_ID_CNS_NS_CS_INDEP:
case NVME_ID_CNS_CS_CTRL:
case NVME_ID_CNS_CTRL:
return true;
}
switch (c->common.opcode) {
case nvme_admin_identify:
switch (c->identify.cns) {
case NVME_ID_CNS_NS:
case NVME_ID_CNS_CS_NS:
case NVME_ID_CNS_NS_CS_INDEP:
case NVME_ID_CNS_CS_CTRL:
case NVME_ID_CNS_CTRL:
return true;
}
goto admin;
break;
case nvme_admin_set_features:
/*
* Reject Set Features that change controller state the driver
* manages itself; setting them behind the driver's back from
* userspace leaves it unable to react correctly. Keep Alive is
* only armed for fabrics - on other transports it has no
* reserved tag and harms idle power states.
*/
switch (le32_to_cpu(c->features.fid) & 0xff) {
case NVME_FEAT_KATO:
if (ctrl->ops->flags & NVME_F_FABRICS)
break;
fallthrough;
case NVME_FEAT_HOST_BEHAVIOR:
case NVME_FEAT_HOST_MEM_BUF:
case NVME_FEAT_NUM_QUEUES:
case NVME_FEAT_AUTO_PST:
return false;
}
break;
}
return capable(CAP_SYS_ADMIN);
}
static bool nvme_ns_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
bool open_for_write)
{
u32 effects;
/*
* Check if the controller provides a Commands Supported and Effects log
@@ -61,7 +70,7 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
*/
effects = nvme_command_effects(ns->ctrl, ns, c->common.opcode);
if (!(effects & NVME_CMD_EFFECTS_CSUPP))
goto admin;
return capable(CAP_SYS_ADMIN);
/*
* Don't allow passthrough for command that have intrusive (or unknown)
@@ -70,7 +79,7 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
NVME_CMD_EFFECTS_UUID_SEL |
NVME_CMD_EFFECTS_SCOPE_MASK))
goto admin;
return capable(CAP_SYS_ADMIN);
/*
* Only allow I/O commands that transfer data to the controller or that
@@ -79,11 +88,34 @@ static bool nvme_cmd_allowed(struct nvme_ns *ns, struct nvme_command *c,
*/
if ((nvme_is_write(c) || (effects & NVME_CMD_EFFECTS_LBCC)) &&
!open_for_write)
goto admin;
return capable(CAP_SYS_ADMIN);
return true;
admin:
return capable(CAP_SYS_ADMIN);
}
static bool nvme_cmd_allowed(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
struct nvme_command *c, unsigned int flags,
bool open_for_write)
{
/*
* Do not allow unprivileged passthrough on partitions, as that
* allows an escape from the containment of the partition.
*/
if (flags & NVME_IOCTL_PARTITION)
return capable(CAP_SYS_ADMIN);
/*
* Do not allow unprivileged processes to send vendor specific or
* fabrics commands as we can't be sure about their effects.
*/
if (c->common.opcode >= nvme_cmd_vendor_start ||
c->common.opcode == nvme_fabrics_command)
return capable(CAP_SYS_ADMIN);
if (!ns)
return nvme_admin_cmd_allowed(ctrl, c);
return nvme_ns_cmd_allowed(ns, c, open_for_write);
}
/*
@@ -202,7 +234,8 @@ out_free_req:
return ret;
}
static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio,
unsigned int flags, bool open_for_write)
{
struct nvme_user_io io;
struct nvme_command c;
@@ -260,6 +293,9 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
c.rw.lbat = cpu_to_le16(io.apptag);
c.rw.lbatm = cpu_to_le16(io.appmask);
if (!nvme_cmd_allowed(ns->ctrl, ns, &c, flags, open_for_write))
return -EACCES;
return nvme_submit_user_cmd(ns->queue, &c, io.addr, length, metadata,
meta_len, NULL, 0, 0);
}
@@ -307,7 +343,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
c.common.cdw14 = cpu_to_le32(cmd.cdw14);
c.common.cdw15 = cpu_to_le32(cmd.cdw15);
if (!nvme_cmd_allowed(ns, &c, 0, open_for_write))
if (!nvme_cmd_allowed(ctrl, ns, &c, 0, open_for_write))
return -EACCES;
if (cmd.timeout_ms)
@@ -354,7 +390,7 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
c.common.cdw14 = cpu_to_le32(cmd.cdw14);
c.common.cdw15 = cpu_to_le32(cmd.cdw15);
if (!nvme_cmd_allowed(ns, &c, flags, open_for_write))
if (!nvme_cmd_allowed(ctrl, ns, &c, flags, open_for_write))
return -EACCES;
if (cmd.timeout_ms)
@@ -449,6 +485,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
const struct nvme_uring_cmd *cmd = io_uring_sqe128_cmd(ioucmd->sqe,
struct nvme_uring_cmd);
struct request_queue *q = ns ? ns->queue : ctrl->admin_q;
bool open_for_write = ioucmd->file->f_mode & FMODE_WRITE;
struct nvme_uring_data d;
struct nvme_command c;
struct iov_iter iter;
@@ -479,7 +516,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
c.common.cdw14 = cpu_to_le32(READ_ONCE(cmd->cdw14));
c.common.cdw15 = cpu_to_le32(READ_ONCE(cmd->cdw15));
if (!nvme_cmd_allowed(ns, &c, 0, ioucmd->file->f_mode & FMODE_WRITE))
if (!nvme_cmd_allowed(ctrl, ns, &c, 0, open_for_write))
return -EACCES;
d.metadata = READ_ONCE(cmd->metadata);
@@ -595,7 +632,7 @@ static int nvme_ns_ioctl(struct nvme_ns *ns, unsigned int cmd,
case NVME_IOCTL_SUBMIT_IO32:
#endif
case NVME_IOCTL_SUBMIT_IO:
return nvme_submit_io(ns, argp);
return nvme_submit_io(ns, argp, flags, open_for_write);
case NVME_IOCTL_IO64_CMD_VEC:
flags |= NVME_IOCTL_VEC;
fallthrough;
@@ -692,14 +729,14 @@ int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd,
static int nvme_ns_head_ctrl_ioctl(struct nvme_ns *ns, unsigned int cmd,
void __user *argp, struct nvme_ns_head *head, int srcu_idx,
bool open_for_write)
__releases(&head->srcu)
__releases_shared(&head->srcu)
{
struct nvme_ctrl *ctrl = ns->ctrl;
int ret;
nvme_get_ctrl(ns->ctrl);
srcu_read_unlock(&head->srcu, srcu_idx);
ret = nvme_ctrl_ioctl(ns->ctrl, cmd, argp, open_for_write);
ret = nvme_ctrl_ioctl(ctrl, cmd, argp, open_for_write);
nvme_put_ctrl(ctrl);
return ret;
drivers/nvme/host/multipath.c
+34 -30
View File
@@ -288,6 +288,25 @@ void nvme_mpath_revalidate_paths(struct nvme_ns_head *head)
kblockd_schedule_work(&head->requeue_work);
}
#ifdef CONFIG_BLK_DEV_ZONED
int nvme_mpath_revalidate_zones(struct nvme_ns_head *head)
{
struct gendisk *disk = head->disk;
int ret;
if (!disk || !blk_queue_is_zoned(disk->queue) ||
!test_bit(NVME_NSHEAD_DISK_LIVE, &head->flags))
return 0;
ret = blk_revalidate_disk_zones(disk);
if (ret)
dev_warn_ratelimited(disk_to_dev(disk),
"failed to revalidate zoned namespace head: %d\n",
ret);
return ret;
}
#endif /* CONFIG_BLK_DEV_ZONED */
static bool nvme_path_is_disabled(struct nvme_ns *ns)
{
enum nvme_ctrl_state state = nvme_ctrl_state(ns->ctrl);
@@ -306,6 +325,7 @@ static bool nvme_path_is_disabled(struct nvme_ns *ns)
}
static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)
__must_hold_shared(&head->srcu)
{
int found_distance = INT_MAX, fallback_distance = INT_MAX, distance;
struct nvme_ns *found = NULL, *fallback = NULL, *ns;
@@ -348,6 +368,7 @@ static struct nvme_ns *__nvme_find_path(struct nvme_ns_head *head, int node)
static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
struct nvme_ns *ns)
__must_hold_shared(&head->srcu)
{
ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct nvme_ns,
siblings);
@@ -357,6 +378,7 @@ static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
}
static struct nvme_ns *nvme_round_robin_path(struct nvme_ns_head *head)
__must_hold_shared(&head->srcu)
{
struct nvme_ns *ns, *found = NULL;
int node = numa_node_id();
@@ -405,6 +427,7 @@ out:
}
static struct nvme_ns *nvme_queue_depth_path(struct nvme_ns_head *head)
__must_hold_shared(&head->srcu)
{
struct nvme_ns *best_opt = NULL, *best_nonopt = NULL, *ns;
unsigned int min_depth_opt = UINT_MAX, min_depth_nonopt = UINT_MAX;
@@ -448,6 +471,7 @@ static inline bool nvme_path_is_optimized(struct nvme_ns *ns)
}
static struct nvme_ns *nvme_numa_path(struct nvme_ns_head *head)
__must_hold_shared(&head->srcu)
{
int node = numa_node_id();
struct nvme_ns *ns;
@@ -473,6 +497,7 @@ inline struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
}
static bool nvme_available_path(struct nvme_ns_head *head)
__must_hold_shared(&head->srcu)
{
struct nvme_ns *ns;
@@ -611,28 +636,8 @@ const struct block_device_operations nvme_ns_head_ops = {
.pr_ops = &nvme_pr_ops,
};
static inline struct nvme_ns_head *cdev_to_ns_head(struct cdev *cdev)
{
return container_of(cdev, struct nvme_ns_head, cdev);
}
static int nvme_ns_head_chr_open(struct inode *inode, struct file *file)
{
if (!nvme_tryget_ns_head(cdev_to_ns_head(inode->i_cdev)))
return -ENXIO;
return 0;
}
static int nvme_ns_head_chr_release(struct inode *inode, struct file *file)
{
nvme_put_ns_head(cdev_to_ns_head(inode->i_cdev));
return 0;
}
static const struct file_operations nvme_ns_head_chr_fops = {
.owner = THIS_MODULE,
.open = nvme_ns_head_chr_open,
.release = nvme_ns_head_chr_release,
.unlocked_ioctl = nvme_ns_head_chr_ioctl,
.compat_ioctl = compat_ptr_ioctl,
.uring_cmd = nvme_ns_head_chr_uring_cmd,
@@ -647,10 +652,12 @@ static void nvme_add_ns_head_cdev(struct nvme_ns_head *head)
snprintf(name, sizeof(name), "ng%dn%d", head->subsys->instance,
head->instance);
nvme_get_ns_head(head); /* Undone in nvme_cdev_rel() */
if (nvme_cdev_add(name, &head->cdev, &head->cdev_device,
&nvme_ns_head_chr_fops, THIS_MODULE)) {
dev_err(disk_to_dev(head->disk),
"Unable to create the %s device\n", name);
nvme_put_ns_head(head);
return;
}
set_bit(NVME_NSHEAD_CDEV_LIVE, &head->flags);
@@ -692,14 +699,15 @@ static void nvme_remove_head(struct nvme_ns_head *head)
{
if (test_and_clear_bit(NVME_NSHEAD_DISK_LIVE, &head->flags)) {
/*
* requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
* to allow multipath to fail all I/O.
* Requeue I/O after NVME_NSHEAD_DISK_LIVE has been cleared
* to allow multipath to fail all I/O. First synchronize to
* add any bios to the requeue list.
*/
synchronize_srcu(&head->srcu);
kblockd_schedule_work(&head->requeue_work);
if (test_and_clear_bit(NVME_NSHEAD_CDEV_LIVE, &head->flags))
nvme_cdev_del(&head->cdev, &head->cdev_device);
synchronize_srcu(&head->srcu);
del_gendisk(head->disk);
}
nvme_put_ns_head(head);
@@ -728,12 +736,10 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
struct queue_limits lim;
mutex_init(&head->lock);
bio_list_init(&head->requeue_list);
spin_lock_init(&head->requeue_lock);
INIT_WORK(&head->requeue_work, nvme_requeue_work);
INIT_WORK(&head->partition_scan_work, nvme_partition_scan_work);
INIT_DELAYED_WORK(&head->remove_work, nvme_remove_head_work);
head->delayed_removal_secs = 0;
/*
* If "multipath_always_on" is enabled, a multipath node is added
@@ -777,7 +783,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
set_bit(GD_SUPPRESS_PART_SCAN, &head->disk->state);
sprintf(head->disk->disk_name, "nvme%dn%d",
ctrl->subsys->instance, head->instance);
nvme_tryget_ns_head(head);
nvme_get_ns_head(head);
return 0;
}
@@ -819,12 +825,14 @@ static void nvme_mpath_set_live(struct nvme_ns *ns)
mutex_unlock(&head->lock);
synchronize_srcu(&head->srcu);
nvme_mpath_revalidate_zones(head);
kblockd_schedule_work(&head->requeue_work);
}
static int nvme_parse_ana_log(struct nvme_ctrl *ctrl, void *data,
int (*cb)(struct nvme_ctrl *ctrl, struct nvme_ana_group_desc *,
void *))
__must_hold(&ctrl->ana_lock)
{
void *base = ctrl->ana_log_buf;
size_t offset = sizeof(struct nvme_ana_rsp_hdr);
@@ -1375,10 +1383,6 @@ void nvme_mpath_add_disk(struct nvme_ns *ns, __le32 anagrpid)
nvme_mpath_set_live(ns);
}
#ifdef CONFIG_BLK_DEV_ZONED
if (blk_queue_is_zoned(ns->queue) && ns->head->disk)
ns->head->disk->nr_zones = ns->disk->nr_zones;
#endif
}
void nvme_mpath_remove_disk(struct nvme_ns_head *head)
drivers/nvme/host/nvme.h
+50 -17
View File
@@ -178,6 +178,11 @@ enum nvme_quirks {
* Align dma pool segment size to 512 bytes
*/
NVME_QUIRK_DMAPOOL_ALIGN_512 = (1 << 22),
/*
* Admin queue DMA buffers must be page aligned
*/
NVME_QUIRK_ADMIN_PAGE_ALIGN = (1 << 23),
};
static inline char *nvme_quirk_name(enum nvme_quirks q)
@@ -229,6 +234,8 @@ static inline char *nvme_quirk_name(enum nvme_quirks q)
return "broken_msi";
case NVME_QUIRK_DMAPOOL_ALIGN_512:
return "dmapool_align_512";
case NVME_QUIRK_ADMIN_PAGE_ALIGN:
return "admin_page_align";
}
return "unknown";
@@ -361,7 +368,8 @@ struct nvme_ctrl {
wait_queue_head_t state_wq;
struct nvme_subsystem *subsys;
struct list_head subsys_entry;
struct list_head subsys_entry
__guarded_by(&nvme_subsystems_lock);
struct opal_dev *opal_dev;
@@ -493,10 +501,13 @@ struct nvme_subsystem {
* a separate refcount.
*/
struct kref ref;
struct list_head entry;
struct list_head entry
__guarded_by(&nvme_subsystems_lock);
struct mutex lock;
struct list_head ctrls;
struct list_head nsheads;
struct list_head ctrls
__guarded_by(&nvme_subsystems_lock);
struct list_head nsheads
__guarded_by(&lock);
char subnqn[NVMF_NQN_SIZE];
char serial[20];
char model[40];
@@ -561,20 +572,22 @@ struct nvme_ns_head {
u16 nr_plids;
u16 *plids;
#ifdef CONFIG_NVME_MULTIPATH
struct bio_list requeue_list;
struct bio_list requeue_list
__guarded_by(&requeue_lock);
spinlock_t requeue_lock;
struct work_struct requeue_work;
struct work_struct partition_scan_work;
struct mutex lock;
unsigned long flags;
struct delayed_work remove_work;
unsigned int delayed_removal_secs;
unsigned int delayed_removal_secs
__guarded_by(&subsys->lock);
atomic_long_t io_requeue_no_usable_path_count;
atomic_long_t io_fail_no_available_path_count;
#define NVME_NSHEAD_DISK_LIVE 0
#define NVME_NSHEAD_QUEUE_IF_NO_PATH 1
#define NVME_NSHEAD_CDEV_LIVE 2
struct nvme_ns __rcu *current_path[];
struct nvme_ns __rcu_guarded *current_path[];
#endif
};
@@ -679,12 +692,12 @@ static inline struct request *nvme_find_rq(struct blk_mq_tags *tags,
rq = blk_mq_tag_to_rq(tags, tag);
if (unlikely(!rq)) {
pr_err("could not locate request for tag %#x\n",
tag);
pr_err_ratelimited("could not locate request for tag %#x\n",
tag);
return NULL;
}
if (unlikely(nvme_genctr_mask(nvme_req(rq)->genctr) != genctr)) {
dev_err(nvme_req(rq)->ctrl->device,
dev_err_ratelimited(nvme_req(rq)->ctrl->device,
"request %#x genctr mismatch (got %#x expected %#x)\n",
tag, genctr, nvme_genctr_mask(nvme_req(rq)->genctr));
return NULL;
@@ -995,6 +1008,7 @@ int nvme_delete_ctrl(struct nvme_ctrl *ctrl);
void nvme_queue_scan(struct nvme_ctrl *ctrl);
int nvme_get_log(struct nvme_ctrl *ctrl, u32 nsid, u8 log_page, u8 lsp, u8 csi,
void *log, size_t size, u64 offset);
void nvme_get_ns_head(struct nvme_ns_head *head);
bool nvme_tryget_ns_head(struct nvme_ns_head *head);
void nvme_put_ns_head(struct nvme_ns_head *head);
int nvme_cdev_add(const char *name, struct cdev *cdev,
@@ -1032,16 +1046,20 @@ extern const struct attribute_group *nvme_dev_attr_groups[];
extern const struct block_device_operations nvme_bdev_ops;
void nvme_delete_ctrl_sync(struct nvme_ctrl *ctrl);
struct nvme_ns *nvme_find_path(struct nvme_ns_head *head);
struct nvme_ns *nvme_find_path(struct nvme_ns_head *head)
__must_hold_shared(&head->srcu);
#ifdef CONFIG_NVME_MULTIPATH
static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
{
return ctrl->ana_log_buf != NULL;
}
void nvme_mpath_unfreeze(struct nvme_subsystem *subsys);
void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys);
void nvme_mpath_start_freeze(struct nvme_subsystem *subsys);
void nvme_mpath_unfreeze(struct nvme_subsystem *subsys)
__must_hold(&subsys->lock);
void nvme_mpath_wait_freeze(struct nvme_subsystem *subsys)
__must_hold(&subsys->lock);
void nvme_mpath_start_freeze(struct nvme_subsystem *subsys)
__must_hold(&subsys->lock);
void nvme_mpath_default_iopolicy(struct nvme_subsystem *subsys);
void nvme_failover_req(struct request *req);
void nvme_kick_requeue_lists(struct nvme_ctrl *ctrl);
@@ -1184,6 +1202,15 @@ static inline bool nvme_mpath_queue_if_no_path(struct nvme_ns_head *head)
}
#endif /* CONFIG_NVME_MULTIPATH */
#if defined(CONFIG_NVME_MULTIPATH) && defined(CONFIG_BLK_DEV_ZONED)
int nvme_mpath_revalidate_zones(struct nvme_ns_head *head);
#else
static inline int nvme_mpath_revalidate_zones(struct nvme_ns_head *head)
{
return 0;
}
#endif
int nvme_ns_get_unique_id(struct nvme_ns *ns, u8 id[16],
enum blk_unique_id type);
@@ -1290,10 +1317,16 @@ static inline void nvme_auth_revoke_tls_key(struct nvme_ctrl *ctrl) {};
u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
u8 opcode);
u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u8 opcode);
u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u8 opcode)
__cond_acquires(nonzero, &ctrl->subsys->lock)
__cond_acquires(nonzero, &ctrl->scan_lock);
int nvme_execute_rq(struct request *rq, bool at_head);
void nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
struct nvme_command *cmd, int status);
u32 nvme_passthru_end(struct nvme_ctrl *ctrl, struct nvme_ns *ns, u32 effects,
struct nvme_command *cmd, int status)
__cond_releases(nonzero, &ctrl->scan_lock)
__cond_releases(nonzero, &ctrl->subsys->lock);
struct nvme_ctrl *nvme_ctrl_from_file(struct file *file);
struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid);
bool nvme_get_ns(struct nvme_ns *ns);
drivers/nvme/host/pci.c
+24 -7
View File
@@ -213,6 +213,7 @@ static int quirks_param_set(const char *value, const struct kernel_param *kp)
if (nvme_parse_quirk_entry(field, &qlist[i])) {
pr_err("nvme: failed to parse quirk string %s\n",
value);
err = -EINVAL;
goto out_free_qlist;
}
@@ -366,7 +367,8 @@ struct nvme_queue {
struct nvme_dev *dev;
struct nvme_descriptor_pools descriptor_pools;
spinlock_t sq_lock;
void *sq_cmds;
void *sq_cmds
__guarded_by(&sq_lock);
/* only used for poll queues: */
spinlock_t cq_poll_lock ____cacheline_aligned_in_smp;
struct nvme_completion *cqes;
@@ -375,9 +377,11 @@ struct nvme_queue {
u32 __iomem *q_db;
u32 q_depth;
u16 cq_vector;
u16 sq_tail;
u16 last_sq_tail;
u16 cq_head;
u16 sq_tail
__guarded_by(&sq_lock);
u16 last_sq_tail
__guarded_by(&sq_lock);
u16 qid;
u8 cq_phase;
u8 sqes;
@@ -716,6 +720,7 @@ static void nvme_pci_map_queues(struct blk_mq_tag_set *set)
* Write sq tail if we are asked to, or if the next command would wrap.
*/
static inline void nvme_write_sq_db(struct nvme_queue *nvmeq, bool write_sq)
__must_hold(&nvmeq->sq_lock)
{
if (!write_sq) {
u16 next_tail = nvmeq->sq_tail + 1;
@@ -734,6 +739,7 @@ static inline void nvme_write_sq_db(struct nvme_queue *nvmeq, bool write_sq)
static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq,
struct nvme_command *cmd)
__must_hold(&nvmeq->sq_lock)
{
memcpy(nvmeq->sq_cmds + (nvmeq->sq_tail << nvmeq->sqes),
absolute_pointer(cmd), sizeof(*cmd));
@@ -1580,13 +1586,18 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq,
req = nvme_find_rq(nvme_queue_tagset(nvmeq), command_id);
if (unlikely(!req)) {
dev_warn(nvmeq->dev->ctrl.device,
"invalid id %d completed on queue %d\n",
command_id, le16_to_cpu(cqe->sq_id));
dev_warn_ratelimited(nvmeq->dev->ctrl.device,
"invalid id %d completed on queue %d\n",
command_id, le16_to_cpu(cqe->sq_id));
return;
}
trace_nvme_sq(req, cqe->sq_head, nvmeq->sq_tail);
/*
* Tracing only; annotate a lockless snapshot of nvmeq->sq_tail using
* data_race(). This would also help suppress context analysis warning
* while accessing nvmeq->sq_tail without acquiring ->sq_lock.
*/
trace_nvme_sq(req, cqe->sq_head, data_race(nvmeq->sq_tail));
if (!nvme_try_complete_req(req, cqe->status, cqe->result) &&
!blk_mq_add_to_batch(req, iob,
nvme_req(req)->status != NVME_SC_SUCCESS,
@@ -2013,6 +2024,7 @@ disable:
}
static void nvme_free_queue(struct nvme_queue *nvmeq)
__context_unsafe(/* frees queue which is no longer in use */)
{
dma_free_coherent(nvmeq->dev->dev, CQ_SIZE(nvmeq),
(void *)nvmeq->cqes, nvmeq->cq_dma_addr);
@@ -2107,6 +2119,7 @@ static int nvme_cmb_qdepth(struct nvme_dev *dev, int nr_io_queues,
static int nvme_alloc_sq_cmds(struct nvme_dev *dev, struct nvme_queue *nvmeq,
int qid)
__context_unsafe(/* safe to allocate sq_cmds without any protection */)
{
struct pci_dev *pdev = to_pci_dev(dev->dev);
@@ -2181,6 +2194,7 @@ static int queue_request_irq(struct nvme_queue *nvmeq)
}
static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
__context_unsafe(/* initialize unpublished/lock-guarded variables */)
{
struct nvme_dev *dev = nvmeq->dev;
@@ -2199,6 +2213,7 @@ static void nvme_init_queue(struct nvme_queue *nvmeq, u16 qid)
* Try getting shutdown_lock while setting up IO queues.
*/
static int nvme_setup_io_queues_trylock(struct nvme_dev *dev)
__cond_acquires(0, &dev->shutdown_lock)
{
/*
* Give up if the lock is being held by nvme_dev_disable.
@@ -2400,6 +2415,7 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)
result = queue_request_irq(nvmeq);
if (result) {
dev->online_queues--;
nvme_disable_ctrl(&dev->ctrl, false);
return result;
}
@@ -3838,6 +3854,7 @@ out_disable:
nvme_dev_remove_admin(dev);
nvme_dbbuf_dma_free(dev);
nvme_free_queues(dev, 0);
nvme_release_descriptor_pools(dev);
out_release_iod_mempool:
mempool_destroy(dev->dmavec_mempool);
out_dev_unmap:
drivers/nvme/host/rdma.c
+97 -63
View File
@@ -16,6 +16,7 @@
#include <linux/types.h>
#include <linux/list.h>
#include <linux/mutex.h>
#include <linux/async.h>
#include <linux/scatterlist.h>
#include <linux/nvme.h>
#include <linux/unaligned.h>
@@ -39,11 +40,18 @@
#define NVME_RDMA_METADATA_SGL_SIZE \
(sizeof(struct scatterlist) * NVME_INLINE_METADATA_SG_CNT)
static DEFINE_MUTEX(device_list_mutex);
static LIST_HEAD_GUARDED(device_list, device_list_mutex);
static DEFINE_MUTEX(nvme_rdma_ctrl_mutex);
static LIST_HEAD_GUARDED(nvme_rdma_ctrl_list, nvme_rdma_ctrl_mutex);
struct nvme_rdma_device {
struct ib_device *dev;
struct ib_pd *pd;
struct kref ref;
struct list_head entry;
struct list_head entry
__guarded_by(&device_list_mutex);
unsigned int num_inline_segments;
};
@@ -100,6 +108,11 @@ struct nvme_rdma_queue {
struct mutex queue_lock;
};
struct nvme_rdma_setup_ctx {
struct nvme_rdma_queue *queue;
int *err;
};
struct nvme_rdma_ctrl {
/* read only in the hot path */
struct nvme_rdma_queue *queues;
@@ -112,7 +125,8 @@ struct nvme_rdma_ctrl {
struct delayed_work reconnect_work;
struct list_head list;
struct list_head list
__guarded_by(&nvme_rdma_ctrl_mutex);
struct blk_mq_tag_set admin_tag_set;
struct nvme_rdma_device *device;
@@ -132,12 +146,6 @@ static inline struct nvme_rdma_ctrl *to_rdma_ctrl(struct nvme_ctrl *ctrl)
return container_of(ctrl, struct nvme_rdma_ctrl, ctrl);
}
static LIST_HEAD(device_list);
static DEFINE_MUTEX(device_list_mutex);
static LIST_HEAD(nvme_rdma_ctrl_list);
static DEFINE_MUTEX(nvme_rdma_ctrl_mutex);
/*
* Disabling this option makes small I/O goes faster, but is fundamentally
* unsafe. With it turned off we will have to register a global rkey that
@@ -566,16 +574,14 @@ out_put_dev:
return ret;
}
static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
int idx, size_t queue_size)
static int nvme_rdma_alloc_queue(struct nvme_rdma_queue *queue)
{
struct nvme_rdma_queue *queue;
struct nvme_rdma_ctrl *ctrl = queue->ctrl;
int idx = nvme_rdma_queue_idx(queue);
struct sockaddr *src_addr = NULL;
int ret;
queue = &ctrl->queues[idx];
mutex_init(&queue->queue_lock);
queue->ctrl = ctrl;
if (idx && ctrl->ctrl.max_integrity_segments)
queue->pi_support = true;
else
@@ -587,8 +593,6 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl,
else
queue->cmnd_capsule_len = sizeof(struct nvme_command);
queue->queue_size = queue_size;
queue->cm_id = rdma_create_id(&init_net, nvme_rdma_cm_handler, queue,
RDMA_PS_TCP, IB_QPT_RC);
if (IS_ERR(queue->cm_id)) {
@@ -694,59 +698,68 @@ static int nvme_rdma_start_queue(struct nvme_rdma_ctrl *ctrl, int idx)
return ret;
}
static int nvme_rdma_start_io_queues(struct nvme_rdma_ctrl *ctrl,
int first, int last)
static void nvme_rdma_setup_queue_async(void *data, async_cookie_t cookie)
{
int i, ret = 0;
struct nvme_rdma_setup_ctx *ctx = data;
struct nvme_rdma_queue *queue;
int ret;
for (i = first; i < last; i++) {
ret = nvme_rdma_start_queue(ctrl, i);
if (ret)
goto out_stop_queues;
}
queue = ctx->queue;
ret = nvme_rdma_alloc_queue(queue);
if (ret)
goto out_err;
return 0;
ret = nvme_rdma_start_queue(queue->ctrl, nvme_rdma_queue_idx(queue));
if (ret)
goto out_err;
out_stop_queues:
for (i--; i >= first; i--)
nvme_rdma_stop_queue(&ctrl->queues[i]);
return ret;
return;
out_err:
WRITE_ONCE(*ctx->err, ret);
}
static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl)
static int nvme_rdma_setup_io_queues(struct nvme_rdma_ctrl *ctrl,
unsigned int first, unsigned int last, size_t queue_size)
{
struct nvmf_ctrl_options *opts = ctrl->ctrl.opts;
unsigned int nr_io_queues;
int i, ret;
ASYNC_DOMAIN_EXCLUSIVE(queue_domain);
struct nvme_rdma_setup_ctx *ctxs;
int nr_queues = last - first;
int err = 0, i, ret;
nr_io_queues = nvmf_nr_io_queues(opts);
ret = nvme_set_queue_count(&ctrl->ctrl, &nr_io_queues);
if (ret)
return ret;
if (nr_io_queues == 0) {
dev_err(ctrl->ctrl.device,
"unable to set any I/O queues\n");
ctxs = kmalloc_objs(*ctxs, nr_queues);
if (!ctxs)
return -ENOMEM;
for (i = 0; i < nr_queues; i++) {
struct nvme_rdma_queue *queue = &ctrl->queues[first + i];
queue->ctrl = ctrl;
queue->queue_size = queue_size;
ctxs[i].queue = queue;
ctxs[i].err = &err;
async_schedule_domain(nvme_rdma_setup_queue_async, &ctxs[i],
&queue_domain);
}
ctrl->ctrl.queue_count = nr_io_queues + 1;
dev_info(ctrl->ctrl.device,
"creating %d I/O queues.\n", nr_io_queues);
async_synchronize_full_domain(&queue_domain);
kfree(ctxs);
nvmf_set_io_queues(opts, nr_io_queues, ctrl->io_queues);
for (i = 1; i < ctrl->ctrl.queue_count; i++) {
ret = nvme_rdma_alloc_queue(ctrl, i,
ctrl->ctrl.sqsize + 1);
if (ret)
goto out_free_queues;
}
ret = READ_ONCE(err);
if (ret)
goto out_free_queues;
return 0;
out_free_queues:
for (i--; i >= 1; i--)
nvme_rdma_free_queue(&ctrl->queues[i]);
for (i = 0; i < nr_queues; i++) {
struct nvme_rdma_queue *queue =
&ctrl->queues[first + i];
if (test_bit(NVME_RDMA_Q_LIVE, &queue->flags))
nvme_rdma_stop_queue(queue);
if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
nvme_rdma_free_queue(queue);
}
return ret;
}
@@ -783,7 +796,9 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl,
bool pi_capable = false;
int error;
error = nvme_rdma_alloc_queue(ctrl, 0, NVME_AQ_DEPTH);
ctrl->queues[0].ctrl = ctrl;
ctrl->queues[0].queue_size = NVME_AQ_DEPTH;
error = nvme_rdma_alloc_queue(&ctrl->queues[0]);
if (error)
return error;
@@ -863,12 +878,23 @@ out_free_queue:
static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
{
unsigned int nr_io_queues;
int ret, nr_queues;
ret = nvme_rdma_alloc_io_queues(ctrl);
nr_io_queues = nvmf_nr_io_queues(ctrl->ctrl.opts);
ret = nvme_set_queue_count(&ctrl->ctrl, &nr_io_queues);
if (ret)
return ret;
if (nr_io_queues == 0) {
dev_err(ctrl->ctrl.device, "unable to set any I/O queues\n");
return -ENOMEM;
}
ctrl->ctrl.queue_count = nr_io_queues + 1;
dev_info(ctrl->ctrl.device, "creating %d I/O queues.\n", nr_io_queues);
nvmf_set_io_queues(ctrl->ctrl.opts, nr_io_queues, ctrl->io_queues);
if (new) {
ret = nvme_rdma_alloc_tag_set(&ctrl->ctrl);
if (ret)
@@ -881,7 +907,9 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
* queue number might have changed.
*/
nr_queues = min(ctrl->tag_set.nr_hw_queues + 1, ctrl->ctrl.queue_count);
ret = nvme_rdma_start_io_queues(ctrl, 1, nr_queues);
ret = nvme_rdma_setup_io_queues(ctrl, 1, nr_queues,
ctrl->ctrl.sqsize + 1);
if (ret)
goto out_cleanup_tagset;
@@ -905,12 +933,15 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)
/*
* If the number of queues has increased (reconnect case)
* start all new queues now.
* setup all new queues now.
*/
ret = nvme_rdma_start_io_queues(ctrl, nr_queues,
ctrl->tag_set.nr_hw_queues + 1);
if (ret)
goto out_wait_freeze_timed_out;
if (ctrl->tag_set.nr_hw_queues + 1 > nr_queues) {
ret = nvme_rdma_setup_io_queues(ctrl, nr_queues,
ctrl->tag_set.nr_hw_queues + 1,
ctrl->ctrl.sqsize + 1);
if (ret)
goto out_wait_freeze_timed_out;
}
return 0;
@@ -969,7 +1000,7 @@ static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl)
{
struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl);
if (list_empty(&ctrl->list))
if (list_empty_careful(&ctrl->list))
goto free_ctrl;
mutex_lock(&nvme_rdma_ctrl_mutex);
@@ -2254,7 +2285,10 @@ static struct nvme_rdma_ctrl *nvme_rdma_alloc_ctrl(struct device *dev,
if (!ctrl)
return ERR_PTR(-ENOMEM);
ctrl->ctrl.opts = opts;
INIT_LIST_HEAD(&ctrl->list);
/*
* Safe to init list while allocating ctrl object.
*/
context_unsafe(INIT_LIST_HEAD(&ctrl->list));
if (!(opts->mask & NVMF_OPT_TRSVCID)) {
opts->trsvcid =
drivers/nvme/host/sysfs.c
+3 -1
View File
@@ -240,8 +240,10 @@ static ssize_t nuse_show(struct device *dev, struct device_attribute *attr,
ret = ns_head_update_nuse(head);
else
ret = ns_update_nuse(disk->private_data);
if (ret)
if (ret < 0)
return ret;
else if (ret > 0)
return -EIO;
return sysfs_emit(buf, "%llu\n", head->nuse);
}
drivers/nvme/host/tcp.c
+65 -8
View File
@@ -80,6 +80,7 @@ struct nvme_tcp_request {
struct bio *curr_bio;
struct iov_iter iter;
u32 data_recvd;
/* send state */
size_t offset;
@@ -108,6 +109,7 @@ struct nvme_tcp_queue {
struct mutex queue_lock;
struct mutex send_mutex;
struct mutex pf_cache_lock;
struct llist_head req_list;
struct list_head send_list;
@@ -149,13 +151,17 @@ struct nvme_tcp_queue {
#endif
};
static DEFINE_MUTEX(nvme_tcp_ctrl_mutex);
static LIST_HEAD_GUARDED(nvme_tcp_ctrl_list, nvme_tcp_ctrl_mutex);
struct nvme_tcp_ctrl {
/* read only in the hot path */
struct nvme_tcp_queue *queues;
struct blk_mq_tag_set tag_set;
/* other member variables */
struct list_head list;
struct list_head list
__guarded_by(&nvme_tcp_ctrl_mutex);
struct blk_mq_tag_set admin_tag_set;
struct sockaddr_storage addr;
struct sockaddr_storage src_addr;
@@ -167,8 +173,6 @@ struct nvme_tcp_ctrl {
u32 io_queues[HCTX_MAX_TYPES];
};
static LIST_HEAD(nvme_tcp_ctrl_list);
static DEFINE_MUTEX(nvme_tcp_ctrl_mutex);
static struct workqueue_struct *nvme_tcp_wq;
static const struct blk_mq_ops nvme_tcp_mq_ops;
static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
@@ -357,7 +361,7 @@ static void nvme_tcp_init_iter(struct nvme_tcp_request *req,
iov_iter_bvec(&req->iter, dir,
__bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter), nr_bvec,
bio->bi_iter.bi_size);
req->iter.iov_offset = bio->bi_iter.bi_bvec_done;
req->iter.iov_offset = bio->bi_iter.bi_offset;
}
}
@@ -550,9 +554,11 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set,
struct nvme_tcp_queue *queue = &ctrl->queues[queue_idx];
u8 hdgst = nvme_tcp_hdgst_len(queue);
mutex_lock(&queue->pf_cache_lock);
req->pdu = page_frag_alloc(&queue->pf_cache,
sizeof(struct nvme_tcp_cmd_pdu) + hdgst,
GFP_KERNEL | __GFP_ZERO);
mutex_unlock(&queue->pf_cache_lock);
if (!req->pdu)
return -ENOMEM;
@@ -612,6 +618,29 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl)
queue_work(nvme_reset_wq, &to_tcp_ctrl(ctrl)->err_work);
}
/*
* NVMe has no short read: a read that completes successfully must
* have transferred everything it asked for.
*/
static bool nvme_tcp_data_in_short(struct nvme_tcp_queue *queue,
struct request *rq)
{
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
if (le16_to_cpu(req->status) >> 1)
return false;
if (req_op(rq) != REQ_OP_READ || !req->data_len)
return false;
if (likely(req->data_recvd == req->data_len))
return false;
dev_err(queue->ctrl->ctrl.device,
"queue %d tag %#x short data-in: got %u of %u\n",
nvme_tcp_queue_id(queue), rq->tag,
req->data_recvd, req->data_len);
return true;
}
static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
struct nvme_completion *cqe)
{
@@ -631,6 +660,9 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
if (req->status == cpu_to_le16(NVME_SC_SUCCESS))
req->status = cqe->status;
if (unlikely(nvme_tcp_data_in_short(queue, rq)))
return -EPROTO;
if (!nvme_try_complete_req(rq, req->status, cqe->result))
nvme_complete_rq(rq);
queue->nr_cqe++;
@@ -641,6 +673,7 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue,
struct nvme_tcp_data_pdu *pdu)
{
struct nvme_tcp_request *req;
struct request *rq;
rq = nvme_find_rq(nvme_tcp_tagset(queue), pdu->command_id);
@@ -651,7 +684,8 @@ static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue,
return -ENOENT;
}
if (!blk_rq_payload_bytes(rq)) {
req = blk_mq_rq_to_pdu(rq);
if (!blk_rq_payload_bytes(rq) || !req->curr_bio || !req->data_len) {
dev_err(queue->ctrl->ctrl.device,
"queue %d tag %#x unexpected data\n",
nvme_tcp_queue_id(queue), rq->tag);
@@ -745,6 +779,13 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
}
req = blk_mq_rq_to_pdu(rq);
if (unlikely(rq_data_dir(rq) != WRITE)) {
dev_err(queue->ctrl->ctrl.device,
"req %d unexpected r2t for a non-write command\n",
rq->tag);
return -EPROTO;
}
if (unlikely(!r2t_length)) {
dev_err(queue->ctrl->ctrl.device,
"req %d r2t len is %u, probably a bug...\n",
@@ -953,6 +994,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
*len -= recv_len;
*offset += recv_len;
queue->data_remaining -= recv_len;
req->data_recvd += recv_len;
}
if (!queue->data_remaining) {
@@ -961,6 +1003,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
queue->ddgst_remaining = NVME_TCP_DIGEST_LENGTH;
} else {
if (pdu->hdr.flags & NVME_TCP_F_DATA_SUCCESS) {
if (unlikely(nvme_tcp_data_in_short(queue, rq)))
return -EPROTO;
nvme_tcp_end_request(rq,
le16_to_cpu(req->status));
queue->nr_cqe++;
@@ -1009,6 +1053,9 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,
pdu->command_id);
struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq);
if (unlikely(nvme_tcp_data_in_short(queue, rq)))
return -EPROTO;
nvme_tcp_end_request(rq, le16_to_cpu(req->status));
queue->nr_cqe++;
}
@@ -1417,9 +1464,11 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
struct nvme_tcp_request *async = &ctrl->async_req;
u8 hdgst = nvme_tcp_hdgst_len(queue);
mutex_lock(&queue->pf_cache_lock);
async->pdu = page_frag_alloc(&queue->pf_cache,
sizeof(struct nvme_tcp_cmd_pdu) + hdgst,
GFP_KERNEL | __GFP_ZERO);
mutex_unlock(&queue->pf_cache_lock);
if (!async->pdu)
return -ENOMEM;
@@ -1461,6 +1510,7 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
kfree(queue->pdu);
mutex_destroy(&queue->send_mutex);
mutex_destroy(&queue->queue_lock);
mutex_destroy(&queue->pf_cache_lock);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
lockdep_unregister_key(&queue->nvme_tcp_sk_key);
@@ -1788,6 +1838,7 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
INIT_LIST_HEAD(&queue->send_list);
mutex_init(&queue->send_mutex);
INIT_WORK(&queue->io_work, nvme_tcp_io_work);
mutex_init(&queue->pf_cache_lock);
if (qid > 0)
queue->cmnd_capsule_len = nctrl->ioccsz * 16;
@@ -1928,6 +1979,7 @@ err_sock:
err_destroy_mutex:
mutex_destroy(&queue->send_mutex);
mutex_destroy(&queue->queue_lock);
mutex_destroy(&queue->pf_cache_lock);
return ret;
}
@@ -2577,7 +2629,7 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)
{
struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
if (list_empty(&ctrl->list))
if (list_empty_careful(&ctrl->list))
goto free_ctrl;
mutex_lock(&nvme_tcp_ctrl_mutex);
@@ -2736,6 +2788,7 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns,
req->status = cpu_to_le16(NVME_SC_SUCCESS);
req->offset = 0;
req->data_sent = 0;
req->data_recvd = 0;
req->pdu_len = 0;
req->pdu_sent = 0;
req->h2cdata_left = 0;
@@ -2919,7 +2972,10 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
if (!ctrl)
return ERR_PTR(-ENOMEM);
INIT_LIST_HEAD(&ctrl->list);
/*
* Safe to init list while allocating ctrl object.
*/
context_unsafe(INIT_LIST_HEAD(&ctrl->list));
ctrl->ctrl.opts = opts;
ctrl->ctrl.queue_count = opts->nr_io_queues + opts->nr_write_queues +
opts->nr_poll_queues + 1;
@@ -2960,7 +3016,8 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
}
if (opts->mask & NVMF_OPT_HOST_IFACE) {
if (!__dev_get_by_name(&init_net, opts->host_iface)) {
if (!__dev_get_by_name(current->nsproxy->net_ns,
opts->host_iface)) {
pr_err("invalid interface passed: %s\n",
opts->host_iface);
ret = -ENODEV;
drivers/nvme/host/zns.c
+8 -3
View File
@@ -155,7 +155,8 @@ static int nvme_zone_parse_entry(struct nvme_ns *ns,
struct blk_zone zone = { };
if ((entry->zt & 0xf) != NVME_ZONE_TYPE_SEQWRITE_REQ) {
dev_err(ns->ctrl->device, "invalid zone type %#x\n", entry->zt);
dev_err(ns->ctrl->device, "invalid zone type %#x at zone %u\n",
entry->zt, idx);
return -EINVAL;
}
@@ -178,7 +179,7 @@ int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector,
struct nvme_zone_report *report;
struct nvme_command c = { };
int ret, zone_idx = 0;
unsigned int nz, i;
unsigned int max_in_buf, nz, i;
size_t buflen;
if (ns->head->ids.csi != NVME_CSI_ZNS)
@@ -188,6 +189,9 @@ int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector,
if (!report)
return -ENOMEM;
max_in_buf = (buflen - sizeof(struct nvme_zone_report)) /
sizeof(struct nvme_zone_descriptor);
c.zmr.opcode = nvme_cmd_zone_mgmt_recv;
c.zmr.nsid = cpu_to_le32(ns->head->ns_id);
c.zmr.numd = cpu_to_le32(nvme_bytes_to_numd(buflen));
@@ -207,7 +211,8 @@ int nvme_ns_report_zones(struct nvme_ns *ns, sector_t sector,
goto out_free;
}
nz = min((unsigned int)le64_to_cpu(report->nr_zones), nr_zones);
nz = min3((unsigned int)le64_to_cpu(report->nr_zones),
nr_zones - zone_idx, max_in_buf);
if (!nz)
break;
drivers/nvme/target/admin-cmd.c
+8 -6
View File
@@ -309,8 +309,10 @@ static void nvmet_execute_get_log_page_rmi(struct nvmet_req *req)
}
log = kzalloc_obj(*log);
if (!log)
if (!log) {
status = NVME_SC_INTERNAL;
goto out;
}
log->endgid = req->cmd->get_log_page.lsi;
disk = req->ns->bdev->bd_disk;
@@ -958,7 +960,7 @@ static void nvmet_execute_identify_nslist(struct nvmet_req *req, bool match_css)
nvmet_for_each_enabled_ns(&ctrl->subsys->namespaces, idx, ns) {
if (ns->nsid <= min_nsid)
continue;
if (match_css && req->ns->csi != req->cmd->identify.csi)
if (match_css && ns->csi != req->cmd->identify.csi)
continue;
list[i++] = cpu_to_le32(ns->nsid);
if (i == buf_size / sizeof(__le32))
@@ -1335,7 +1337,7 @@ static u16 nvmet_set_feat_arbitration(struct nvmet_req *req)
void nvmet_execute_set_features(struct nvmet_req *req)
{
struct nvmet_subsys *subsys = nvmet_req_subsys(req);
struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
u16 status = 0;
@@ -1357,7 +1359,7 @@ void nvmet_execute_set_features(struct nvmet_req *req)
break;
}
nvmet_set_result(req,
(subsys->max_qid - 1) | ((subsys->max_qid - 1) << 16));
(ctrl->max_qid - 1) | ((ctrl->max_qid - 1) << 16));
break;
case NVME_FEAT_IRQ_COALESCE:
status = nvmet_set_feat_irq_coalesce(req);
@@ -1494,7 +1496,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req)
void nvmet_execute_get_features(struct nvmet_req *req)
{
struct nvmet_subsys *subsys = nvmet_req_subsys(req);
struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
u16 status = 0;
@@ -1534,7 +1536,7 @@ void nvmet_execute_get_features(struct nvmet_req *req)
break;
case NVME_FEAT_NUM_QUEUES:
nvmet_set_result(req,
(subsys->max_qid-1) | ((subsys->max_qid-1) << 16));
(ctrl->max_qid-1) | ((ctrl->max_qid-1) << 16));
break;
case NVME_FEAT_KATO:
nvmet_get_feat_kato(req);
drivers/nvme/target/configfs.c
+6 -4
View File
@@ -312,15 +312,17 @@ static ssize_t nvmet_param_mdts_store(struct config_item *item,
const char *page, size_t count)
{
struct nvmet_port *port = to_nvmet_port(item);
int ret;
int ret, mdts;
if (nvmet_is_port_enabled(port, __func__))
return -EACCES;
ret = kstrtoint(page, 0, &port->mdts);
if (ret) {
pr_err("Invalid value '%s' for mdts\n", page);
ret = kstrtoint(page, 0, &mdts);
if (ret || mdts < 0 || mdts > NVMET_MAX_MDTS) {
pr_err("Invalid value '%s' for mdts, should be 0-%d\n",
page, NVMET_MAX_MDTS);
return -EINVAL;
}
port->mdts = mdts;
return count;
}
drivers/nvme/target/core.c
+32 -24
View File
@@ -610,12 +610,14 @@ int nvmet_ns_enable(struct nvmet_ns *ns)
goto out_dev_put;
}
if (percpu_ref_init(&ns->ref, nvmet_destroy_namespace, 0, GFP_KERNEL))
ret = percpu_ref_init(&ns->ref, nvmet_destroy_namespace, 0, GFP_KERNEL);
if (ret)
goto out_pr_exit;
nvmet_ns_changed(subsys, ns->nsid);
ns->enabled = true;
xa_set_mark(&subsys->namespaces, ns->nsid, NVMET_NS_ENABLED);
nvmet_debugfs_ns_setup(ns);
ret = 0;
out_unlock:
mutex_unlock(&subsys->lock);
@@ -642,6 +644,7 @@ void nvmet_ns_disable(struct nvmet_ns *ns)
ns->enabled = false;
xa_clear_mark(&subsys->namespaces, ns->nsid, NVMET_NS_ENABLED);
nvmet_debugfs_ns_free(ns);
list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry)
pci_dev_put(radix_tree_delete(&ctrl->p2p_ns_map, ns->nsid));
@@ -875,7 +878,7 @@ u16 nvmet_check_cqid(struct nvmet_ctrl *ctrl, u16 cqid, bool create)
if (!ctrl->cqs)
return NVME_SC_INTERNAL | NVME_STATUS_DNR;
if (cqid > ctrl->subsys->max_qid)
if (cqid > ctrl->max_qid)
return NVME_SC_QID_INVALID | NVME_STATUS_DNR;
if ((create && ctrl->cqs[cqid]) || (!create && !ctrl->cqs[cqid]))
@@ -923,7 +926,7 @@ u16 nvmet_check_sqid(struct nvmet_ctrl *ctrl, u16 sqid,
if (!ctrl->sqs)
return NVME_SC_INTERNAL | NVME_STATUS_DNR;
if (sqid > ctrl->subsys->max_qid)
if (sqid > ctrl->max_qid)
return NVME_SC_QID_INVALID | NVME_STATUS_DNR;
if ((create && ctrl->sqs[sqid]) ||
@@ -1652,23 +1655,6 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
if (!ctrl->changed_ns_list)
goto out_free_ctrl;
ctrl->sqs = kzalloc_objs(struct nvmet_sq *, subsys->max_qid + 1);
if (!ctrl->sqs)
goto out_free_changed_ns_list;
ctrl->cqs = kzalloc_objs(struct nvmet_cq *, subsys->max_qid + 1);
if (!ctrl->cqs)
goto out_free_sqs;
ret = ida_alloc_range(&cntlid_ida,
subsys->cntlid_min, subsys->cntlid_max,
GFP_KERNEL);
if (ret < 0) {
args->status = NVME_SC_CONNECT_CTRL_BUSY | NVME_STATUS_DNR;
goto out_free_cqs;
}
ctrl->cntlid = ret;
/*
* Discovery controllers may use some arbitrary high value
* in order to cleanup stale discovery sessions
@@ -1682,9 +1668,28 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
ctrl->err_counter = 0;
spin_lock_init(&ctrl->error_lock);
nvmet_start_keep_alive_timer(ctrl);
down_read(&nvmet_config_sem);
mutex_lock(&subsys->lock);
ctrl->max_qid = subsys->max_qid;
ctrl->sqs = kzalloc_objs(struct nvmet_sq *, ctrl->max_qid + 1);
if (!ctrl->sqs)
goto out_free_changed_ns_list;
ctrl->cqs = kzalloc_objs(struct nvmet_cq *, ctrl->max_qid + 1);
if (!ctrl->cqs)
goto out_free_sqs;
ret = ida_alloc_range(&cntlid_ida,
subsys->cntlid_min, subsys->cntlid_max,
GFP_KERNEL);
if (ret < 0) {
args->status = NVME_SC_CONNECT_CTRL_BUSY | NVME_STATUS_DNR;
goto out_free_cqs;
}
ctrl->cntlid = ret;
ret = nvmet_ctrl_init_pr(ctrl);
if (ret)
goto init_pr_fail;
@@ -1692,6 +1697,9 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
nvmet_setup_p2p_ns_map(ctrl, args->p2p_client);
nvmet_debugfs_ctrl_setup(ctrl);
mutex_unlock(&subsys->lock);
up_read(&nvmet_config_sem);
nvmet_start_keep_alive_timer(ctrl);
if (args->hostid)
uuid_copy(&ctrl->hostid, args->hostid);
@@ -1721,14 +1729,14 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
return ctrl;
init_pr_fail:
mutex_unlock(&subsys->lock);
nvmet_stop_keep_alive_timer(ctrl);
ida_free(&cntlid_ida, ctrl->cntlid);
out_free_cqs:
kfree(ctrl->cqs);
out_free_sqs:
kfree(ctrl->sqs);
out_free_changed_ns_list:
mutex_unlock(&subsys->lock);
up_read(&nvmet_config_sem);
kfree(ctrl->changed_ns_list);
out_free_ctrl:
kfree(ctrl);
drivers/nvme/target/debugfs.c
+103
View File
@@ -153,6 +153,109 @@ static int nvmet_ctrl_tls_concat_show(struct seq_file *m, void *p)
NVMET_DEBUGFS_ATTR(nvmet_ctrl_tls_concat);
#endif
static const char *const nvmet_pr_type_names[] = {
[NVME_PR_WRITE_EXCLUSIVE] = "write_exclusive",
[NVME_PR_EXCLUSIVE_ACCESS] = "exclusive_access",
[NVME_PR_WRITE_EXCLUSIVE_REG_ONLY] = "write_exclusive_reg_only",
[NVME_PR_EXCLUSIVE_ACCESS_REG_ONLY] = "exclusive_access_reg_only",
[NVME_PR_WRITE_EXCLUSIVE_ALL_REGS] = "write_exclusive_all_regs",
[NVME_PR_EXCLUSIVE_ACCESS_ALL_REGS] = "exclusive_access_all_regs",
};
static const char *nvmet_pr_type_to_str(enum nvme_pr_type type)
{
if (type < ARRAY_SIZE(nvmet_pr_type_names) &&
nvmet_pr_type_names[type])
return nvmet_pr_type_names[type];
return "unknown";
}
static const char *const nvmet_pr_notify_names[] = {
[NVME_PR_NOTIFY_BIT_REG_PREEMPTED] = "reg_preempted",
[NVME_PR_NOTIFY_BIT_RESV_RELEASED] = "resv_released",
[NVME_PR_NOTIFY_BIT_RESV_PREEMPTED] = "resv_preempted",
};
static void nvmet_pr_notify_mask_to_str(struct seq_file *m, unsigned long mask)
{
bool sep = false;
int i;
if (!mask) {
seq_puts(m, "none");
return;
}
for (i = 0; i < ARRAY_SIZE(nvmet_pr_notify_names); i++) {
if (!test_bit(i, &mask) || !nvmet_pr_notify_names[i])
continue;
if (sep)
seq_putc(m, ',');
seq_puts(m, nvmet_pr_notify_names[i]);
sep = true;
}
}
static int nvmet_ns_pr_show(struct seq_file *m, void *p)
{
struct nvmet_ns *ns = m->private;
struct nvmet_pr *pr = &ns->pr;
struct nvmet_pr_registrant *holder, *reg;
seq_printf(m, "enable=%d\n", pr->enable);
if (!pr->enable)
return 0;
seq_printf(m, "generation=%u\n", atomic_read(&pr->generation));
seq_puts(m, "notify_mask=");
nvmet_pr_notify_mask_to_str(m, pr->notify_mask);
seq_putc(m, '\n');
rcu_read_lock();
holder = rcu_dereference(pr->holder);
if (holder) {
seq_printf(m, "rtype=%s\n",
nvmet_pr_type_to_str(holder->rtype));
seq_printf(m, "holder=%pUb,0x%llx\n",
&holder->hostid, holder->rkey);
} else {
seq_puts(m, "rtype=none\n");
seq_puts(m, "holder=none\n");
}
list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
seq_printf(m, "reg=%pUb,0x%llx\n",
&reg->hostid, reg->rkey);
}
rcu_read_unlock();
return 0;
}
NVMET_DEBUGFS_ATTR(nvmet_ns_pr);
void nvmet_debugfs_ns_setup(struct nvmet_ns *ns)
{
char name[16];
struct dentry *parent = ns->subsys->debugfs_dir;
if (!parent)
return;
snprintf(name, sizeof(name), "ns%u", ns->nsid);
ns->debugfs_dir = debugfs_create_dir(name, parent);
if (IS_ERR(ns->debugfs_dir)) {
ns->debugfs_dir = NULL;
return;
}
debugfs_create_file("reservation", 0400, ns->debugfs_dir, ns,
&nvmet_ns_pr_fops);
}
void nvmet_debugfs_ns_free(struct nvmet_ns *ns)
{
debugfs_remove_recursive(ns->debugfs_dir);
ns->debugfs_dir = NULL;
}
int nvmet_debugfs_ctrl_setup(struct nvmet_ctrl *ctrl)
{
char name[32];
drivers/nvme/target/debugfs.h
+5
View File
@@ -14,6 +14,8 @@ int nvmet_debugfs_subsys_setup(struct nvmet_subsys *subsys);
void nvmet_debugfs_subsys_free(struct nvmet_subsys *subsys);
int nvmet_debugfs_ctrl_setup(struct nvmet_ctrl *ctrl);
void nvmet_debugfs_ctrl_free(struct nvmet_ctrl *ctrl);
void nvmet_debugfs_ns_setup(struct nvmet_ns *ns);
void nvmet_debugfs_ns_free(struct nvmet_ns *ns);
int __init nvmet_init_debugfs(void);
void nvmet_exit_debugfs(void);
@@ -30,6 +32,9 @@ static inline int nvmet_debugfs_ctrl_setup(struct nvmet_ctrl *ctrl)
}
static inline void nvmet_debugfs_ctrl_free(struct nvmet_ctrl *ctrl) {}
static inline void nvmet_debugfs_ns_setup(struct nvmet_ns *ns) {}
static inline void nvmet_debugfs_ns_free(struct nvmet_ns *ns) {}
static inline int __init nvmet_init_debugfs(void)
{
return 0;
drivers/nvme/target/fabrics-cmd-auth.c
+13 -4
View File
@@ -9,6 +9,7 @@
#include <linux/random.h>
#include <linux/nvme-auth.h>
#include <crypto/kpp.h>
#include <crypto/utils.h>
#include "nvmet.h"
static void nvmet_auth_expired_work(struct work_struct *work)
@@ -30,12 +31,16 @@ void nvmet_auth_sq_init(struct nvmet_sq *sq)
sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
}
static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)
{
struct nvmet_ctrl *ctrl = req->sq->ctrl;
struct nvmf_auth_dhchap_negotiate_data *data = d;
int i, hash_id = 0, fallback_hash_id = 0, dhgid, fallback_dhgid;
if (tl < sizeof(*data) +
sizeof(struct nvmf_auth_dhchap_protocol_descriptor))
return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
pr_debug("%s: ctrl %d qid %d: data sc_d %d napd %d authid %d halen %d dhlen %d\n",
__func__, ctrl->cntlid, req->sq->qid,
data->sc_c, data->napd, data->auth_protocol[0].dhchap.authid,
@@ -71,6 +76,10 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
NVME_AUTH_DHCHAP_AUTH_ID)
return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
if (data->auth_protocol[0].dhchap.dhlen > NVME_AUTH_DHCHAP_MAX_DH_IDS ||
data->auth_protocol[0].dhchap.halen > NVME_AUTH_DHCHAP_MAX_HASH_IDS)
return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
for (i = 0; i < data->auth_protocol[0].dhchap.halen; i++) {
u8 host_hmac_id = data->auth_protocol[0].dhchap.idlist[i];
@@ -177,7 +186,7 @@ static u8 nvmet_auth_reply(struct nvmet_req *req, void *d, u32 tl)
return NVME_AUTH_DHCHAP_FAILURE_FAILED;
}
if (memcmp(data->rval, response, data->hl)) {
if (crypto_memneq(data->rval, response, data->hl)) {
pr_info("ctrl %d qid %d host response mismatch\n",
ctrl->cntlid, req->sq->qid);
pr_debug("ctrl %d qid %d rval %*ph\n",
@@ -316,7 +325,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
} else if (data->auth_id != req->sq->dhchap_step)
goto done_failure1;
/* Validate negotiation parameters */
dhchap_status = nvmet_auth_negotiate(req, d);
dhchap_status = nvmet_auth_negotiate(req, d, tl);
if (dhchap_status == 0)
req->sq->dhchap_step =
NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE;
@@ -557,7 +566,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
return;
}
d = kmalloc(al, GFP_KERNEL);
d = kzalloc(al, GFP_KERNEL);
if (!d) {
status = NVME_SC_INTERNAL;
goto done;
drivers/nvme/target/fabrics-cmd.c
+1 -1
View File
@@ -370,7 +370,7 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
goto out;
}
if (unlikely(qid > ctrl->subsys->max_qid)) {
if (unlikely(qid > ctrl->max_qid)) {
pr_warn("invalid queue id (%d)\n", qid);
status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
req->cqe->result.u32 = IPO_IATTR_CONNECT_SQE(qid);
drivers/nvme/target/fc.c
+1 -1
View File
@@ -566,7 +566,7 @@ out_fail:
list_del(&iod->ls_rcv_list);
}
kfree(iod);
kfree(tgtport->iod);
return -EFAULT;
}
drivers/nvme/target/nvmet.h
+9
View File
@@ -128,6 +128,9 @@ struct nvmet_ns {
u8 csi;
struct nvmet_pr pr;
struct xarray pr_per_ctrl_refs;
#ifdef CONFIG_NVME_TARGET_DEBUGFS
struct dentry *debugfs_dir;
#endif
};
static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item)
@@ -265,6 +268,7 @@ struct nvmet_ctrl {
uuid_t hostid;
u16 cntlid;
u16 max_qid;
u32 kato;
struct nvmet_port *port;
@@ -753,6 +757,11 @@ static inline struct nvmet_subsys *nvmet_req_subsys(struct nvmet_req *req)
return req->sq->ctrl->subsys;
}
static inline struct nvmet_ctrl *nvmet_req_ctrl(struct nvmet_req *req)
{
return req->sq->ctrl;
}
static inline bool nvmet_is_disc_subsys(struct nvmet_subsys *subsys)
{
return subsys->type != NVME_NQN_NVME;
drivers/nvme/target/passthru.c
+9
View File
@@ -53,13 +53,22 @@ static u16 nvmet_passthru_override_id_descs(struct nvmet_req *req)
for (pos = 0; pos < NVME_IDENTIFY_DATA_SIZE; pos += len) {
struct nvme_ns_id_desc *cur = data + pos;
if (pos + sizeof(*cur) > NVME_IDENTIFY_DATA_SIZE)
break;
if (cur->nidl == 0)
break;
if (cur->nidt == NVME_NIDT_CSI) {
if (pos + sizeof(*cur) + NVME_NIDT_CSI_LEN >
NVME_IDENTIFY_DATA_SIZE)
break;
memcpy(&csi, cur + 1, NVME_NIDT_CSI_LEN);
csi_seen = true;
break;
}
len = sizeof(struct nvme_ns_id_desc) + cur->nidl;
}
drivers/nvme/target/pci-epf.c
+8 -4
View File
@@ -1339,6 +1339,7 @@ err_unmap_queue:
nvmet_pci_epf_mem_unmap(ctrl->nvme_epf, &cq->pci_map);
err_internal:
status = NVME_SC_INTERNAL | NVME_STATUS_DNR;
nvmet_cq_put(&cq->nvme_cq);
err:
if (test_and_clear_bit(NVMET_PCI_EPF_Q_IRQ_ENABLED, &cq->flags))
nvmet_pci_epf_remove_irq_vector(ctrl, cq->vector);
@@ -1594,6 +1595,7 @@ static void nvmet_pci_epf_exec_iod_work(struct work_struct *work)
struct nvmet_pci_epf_iod *iod =
container_of(work, struct nvmet_pci_epf_iod, work);
struct nvmet_req *req = &iod->req;
bool no_wait;
int ret;
if (!iod->ctrl->link_up) {
@@ -1638,14 +1640,16 @@ static void nvmet_pci_epf_exec_iod_work(struct work_struct *work)
}
}
req->execute(req);
/*
* If we do not have data to transfer after the command execution
* finishes, nvmet_pci_epf_queue_response() will complete the command
* directly. No need to wait for the completion in this case.
*/
if (!iod->data_len || iod->dma_dir != DMA_TO_DEVICE)
no_wait = !iod->data_len || iod->dma_dir != DMA_TO_DEVICE;
req->execute(req);
if (no_wait)
return;
wait_for_completion(&iod->done);
@@ -2077,7 +2081,7 @@ static int nvmet_pci_epf_create_ctrl(struct nvmet_pci_epf *nvme_epf,
}
/* Allocate our queues, up to the maximum number. */
ctrl->nr_queues = min(ctrl->tctrl->subsys->max_qid + 1, max_nr_queues);
ctrl->nr_queues = min(ctrl->tctrl->max_qid + 1, max_nr_queues);
ret = nvmet_pci_epf_alloc_queues(ctrl);
if (ret)
goto out_put_ctrl;
drivers/nvme/target/pr.c
+30 -7
View File
@@ -8,7 +8,7 @@
#include <linux/unaligned.h>
#include "nvmet.h"
#define NVMET_PR_NOTIFI_MASK_ALL \
#define NVMET_PR_NOTIFY_MASK_ALL \
(1 << NVME_PR_NOTIFY_BIT_REG_PREEMPTED | \
1 << NVME_PR_NOTIFY_BIT_RESV_RELEASED | \
1 << NVME_PR_NOTIFY_BIT_RESV_PREEMPTED)
@@ -44,7 +44,7 @@ u16 nvmet_set_feat_resv_notif_mask(struct nvmet_req *req, u32 mask)
unsigned long idx;
u16 status;
if (mask & ~(NVMET_PR_NOTIFI_MASK_ALL)) {
if (mask & ~(NVMET_PR_NOTIFY_MASK_ALL)) {
req->error_loc = offsetof(struct nvme_common_command, cdw11);
return NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
}
@@ -169,7 +169,7 @@ static void nvmet_pr_resv_released(struct nvmet_pr *pr, uuid_t *hostid)
nvmet_pr_add_resv_log(ctrl,
NVME_PR_LOG_RESERVATION_RELEASED, ns->nsid);
nvmet_add_async_event(ctrl, NVME_AER_CSS,
NVME_AEN_RESV_LOG_PAGE_AVALIABLE,
NVME_AEN_RESV_LOG_PAGE_AVAILABLE,
NVME_LOG_RESERVATION);
}
}
@@ -188,7 +188,7 @@ static void nvmet_pr_send_event_to_host(struct nvmet_pr *pr, uuid_t *hostid,
if (uuid_equal(hostid, &ctrl->hostid)) {
nvmet_pr_add_resv_log(ctrl, log_type, ns->nsid);
nvmet_add_async_event(ctrl, NVME_AER_CSS,
NVME_AEN_RESV_LOG_PAGE_AVALIABLE,
NVME_AEN_RESV_LOG_PAGE_AVAILABLE,
NVME_LOG_RESERVATION);
}
}
@@ -201,7 +201,7 @@ static void nvmet_pr_resv_preempted(struct nvmet_pr *pr, uuid_t *hostid)
return;
nvmet_pr_send_event_to_host(pr, hostid,
NVME_PR_LOG_RESERVATOIN_PREEMPTED);
NVME_PR_LOG_RESERVATION_PREEMPTED);
}
static void nvmet_pr_registration_preempted(struct nvmet_pr *pr,
@@ -355,9 +355,15 @@ static u16 nvmet_pr_replace(struct nvmet_req *req,
u16 status = NVME_SC_RESERVATION_CONFLICT | NVME_STATUS_DNR;
struct nvmet_ctrl *ctrl = req->sq->ctrl;
struct nvmet_pr *pr = &req->ns->pr;
struct nvmet_pr_registrant *reg;
struct nvmet_pr_registrant *reg, *new = NULL;
u64 nrkey = le64_to_cpu(d->nrkey);
if (ignore_key && nrkey) {
new = kzalloc_obj(*new);
if (!new)
return NVME_SC_INTERNAL;
}
down(&pr->pr_sem);
list_for_each_entry_rcu(reg, &pr->registrant_list, entry) {
if (uuid_equal(&reg->hostid, &ctrl->hostid)) {
@@ -365,9 +371,26 @@ static u16 nvmet_pr_replace(struct nvmet_req *req,
status = nvmet_pr_update_reg_attr(pr, reg,
nvmet_pr_update_reg_rkey,
&nrkey);
break;
goto free_data;
}
}
if (ignore_key) {
if (!nrkey) {
status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
goto free_data;
}
INIT_LIST_HEAD(&new->entry);
new->rkey = nrkey;
uuid_copy(&new->hostid, &ctrl->hostid);
list_add_tail_rcu(&new->entry, &pr->registrant_list);
status = NVME_SC_SUCCESS;
goto out;
}
free_data:
kfree(new);
out:
up(&pr->pr_sem);
return status;
}
drivers/nvme/target/rdma.c
+28 -3
View File
@@ -657,18 +657,25 @@ static void nvmet_rdma_rw_ctx_destroy(struct nvmet_rdma_rsp *rsp)
req->sg, req->sg_cnt, nvmet_data_dir(req));
}
static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
static void nvmet_rdma_free_rsp_resources(struct nvmet_rdma_rsp *rsp)
{
struct nvmet_rdma_queue *queue = rsp->queue;
atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail);
if (rsp->n_rdma)
nvmet_rdma_rw_ctx_destroy(rsp);
if (rsp->req.sg < rsp->cmd->inline_sg ||
rsp->req.sg >= rsp->cmd->inline_sg + queue->dev->inline_page_count)
nvmet_req_free_sgls(&rsp->req);
}
static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
{
struct nvmet_rdma_queue *queue = rsp->queue;
atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail);
nvmet_rdma_free_rsp_resources(rsp);
if (unlikely(!list_empty_careful(&queue->rsp_wr_wait_list)))
nvmet_rdma_process_wr_wait_list(queue);
@@ -1338,9 +1345,27 @@ err_destroy_cq:
goto out;
}
static bool nvmet_rdma_reclaim_rsp(struct sbitmap *sb, unsigned int bitnr,
void *data)
{
struct nvmet_rdma_queue *queue = data;
nvmet_rdma_free_rsp_resources(&queue->rsps[bitnr]);
return true;
}
static void nvmet_rdma_destroy_queue_ib(struct nvmet_rdma_queue *queue)
{
ib_drain_qp(queue->qp);
/*
* Reclaim resources of a response that is still in-flight when the
* queue is being torn down. This happens when the connection was
* forcefully disconnected while an I/O is in flight.
*/
sbitmap_for_each_set(&queue->rsp_tags, nvmet_rdma_reclaim_rsp, queue);
if (queue->cm_id)
rdma_destroy_id(queue->cm_id);
ib_destroy_qp(queue->qp);
drivers/nvme/target/tcp.c
+17 -2
View File
@@ -422,6 +422,19 @@ static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd)
if (!len)
return 0;
/*
* inline_data_size only bounds the in-capsule (type 0x01) SGL
* descriptor below. A non-inline transport SGL data-block
* descriptor skips that check entirely and would otherwise reach
* sgl_alloc() with an attacker-controlled len of up to 4 GiB,
* pinning that much kernel memory for a command that may never
* complete. Bound every descriptor type here, before allocating
* anything, using the same ceiling this file already applies to
* per-PDU H2C data.
*/
if (len > NVMET_TCP_MAXH2CDATA)
return NVME_SC_SGL_INVALID_DATA | NVME_STATUS_DNR;
if (sgl->type == ((NVME_SGL_FMT_DATA_DESC << 4) |
NVME_SGL_FMT_OFFSET)) {
if (!nvme_is_write(cmd->req.cmd))
@@ -433,13 +446,15 @@ static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd)
}
cmd->req.transfer_len += len;
cmd->req.sg = sgl_alloc(len, GFP_KERNEL, &cmd->req.sg_cnt);
cmd->req.sg = sgl_alloc(len, GFP_KERNEL | __GFP_NOWARN,
&cmd->req.sg_cnt);
if (!cmd->req.sg)
return NVME_SC_INTERNAL;
cmd->cur_sg = cmd->req.sg;
if (nvmet_tcp_has_data_in(cmd)) {
cmd->iov = kmalloc_objs(*cmd->iov, cmd->req.sg_cnt);
cmd->iov = kmalloc_objs(*cmd->iov, cmd->req.sg_cnt,
GFP_KERNEL | __GFP_NOWARN);
if (!cmd->iov)
goto err;
}
drivers/nvme/target/zns.c
+11 -4
View File
@@ -116,7 +116,7 @@ void nvmet_execute_identify_ns_zns(struct nvmet_req *req)
mutex_unlock(&req->ns->subsys->lock);
}
if (!bdev_is_zoned(req->ns->bdev)) {
if (!req->ns->bdev || !bdev_is_zoned(req->ns->bdev)) {
status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
req->error_loc = offsetof(struct nvme_identify, nsid);
goto out;
@@ -295,11 +295,18 @@ static void nvmet_bdev_zone_zmgmt_recv_work(struct work_struct *w)
}
/*
* When partial bit is set nr_zones must indicate the number of zone
* descriptors actually transferred.
* Partial report (PR bit set): the host accepts an incomplete listing,
* so cap Number of Zones to the descriptors that fit in the buffer.
* Full report (PR bit clear): Number of Zones is the match count; fail
* if the buffer cannot hold every matching zone descriptor.
*/
if (req->cmd->zmr.pr)
if (req->cmd->zmr.pr) {
rz_data.nr_zones = min(rz_data.nr_zones, rz_data.out_nr_zones);
} else if (rz_data.nr_zones > rz_data.out_nr_zones) {
req->error_loc = offsetof(struct nvme_zone_mgmt_recv_cmd, numd);
status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
goto out;
}
nr_zones = cpu_to_le64(rz_data.nr_zones);
status = nvmet_copy_to_sgl(req, 0, &nr_zones, sizeof(nr_zones));
drivers/s390/block/Makefile
+2
View File
@@ -3,6 +3,8 @@
# S/390 block devices
#
CONTEXT_ANALYSIS := y
dasd_eckd_mod-objs := dasd_eckd.o dasd_3990_erp.o dasd_alias.o
dasd_fba_mod-objs := dasd_fba.o
dasd_diag_mod-objs := dasd_diag.o
drivers/s390/block/dasd.c
+191 -55
View File
@@ -87,35 +87,57 @@ struct dasd_device *dasd_alloc_device(void)
{
struct dasd_device *device;
device = kzalloc_obj(struct dasd_device, GFP_ATOMIC);
device = kzalloc_obj(struct dasd_device, GFP_KERNEL);
if (!device)
return ERR_PTR(-ENOMEM);
/* Get two pages for normal block device operations. */
device->ccw_mem = (void *) __get_free_pages(GFP_ATOMIC | GFP_DMA, 1);
/*
* Four pages: a full-track ITCW is roughly twice the size of a plain
* track-mode one, so this keeps two maximum-size requests in flight.
*/
device->ccw_mem = (void *)__get_free_pages(GFP_KERNEL | GFP_DMA, 2);
if (!device->ccw_mem) {
kfree(device);
return ERR_PTR(-ENOMEM);
}
/* Get one page for error recovery. */
device->erp_mem = (void *) get_zeroed_page(GFP_ATOMIC | GFP_DMA);
if (!device->erp_mem) {
free_pages((unsigned long) device->ccw_mem, 1);
/* per-request track-filler buffers (R0 + count records) */
device->fill_mem = (void *)__get_free_pages(GFP_KERNEL | GFP_DMA, 1);
if (!device->fill_mem) {
free_pages((unsigned long)device->ccw_mem, 2);
kfree(device);
return ERR_PTR(-ENOMEM);
}
/* Get two pages for ese format. */
device->ese_mem = (void *)__get_free_pages(GFP_ATOMIC | GFP_DMA, 1);
/* Get one page for error recovery. */
device->erp_mem = (void *)get_zeroed_page(GFP_KERNEL | GFP_DMA);
if (!device->erp_mem) {
free_pages((unsigned long)device->fill_mem, 1);
free_pages((unsigned long)device->ccw_mem, 2);
kfree(device);
return ERR_PTR(-ENOMEM);
}
/* sized like ccw_chunks: two max-size NRF format requests in flight */
device->ese_mem = (void *)__get_free_pages(GFP_KERNEL | GFP_DMA, 2);
if (!device->ese_mem) {
free_page((unsigned long) device->erp_mem);
free_pages((unsigned long) device->ccw_mem, 1);
free_page((unsigned long)device->erp_mem);
free_pages((unsigned long)device->fill_mem, 1);
free_pages((unsigned long)device->ccw_mem, 2);
kfree(device);
return ERR_PTR(-ENOMEM);
}
device->nulldata = (void *)get_zeroed_page(GFP_KERNEL | GFP_DMA);
if (!device->nulldata) {
free_page((unsigned long)device->erp_mem);
free_pages((unsigned long)device->fill_mem, 1);
free_pages((unsigned long)device->ccw_mem, 2);
free_pages((unsigned long)device->ese_mem, 2);
kfree(device);
return ERR_PTR(-ENOMEM);
}
dasd_init_chunklist(&device->ccw_chunks, device->ccw_mem, PAGE_SIZE*2);
dasd_init_chunklist(&device->ccw_chunks, device->ccw_mem, PAGE_SIZE * 4);
dasd_init_chunklist(&device->fill_chunks, device->fill_mem, PAGE_SIZE * 2);
dasd_init_chunklist(&device->erp_chunks, device->erp_mem, PAGE_SIZE);
dasd_init_chunklist(&device->ese_chunks, device->ese_mem, PAGE_SIZE * 2);
dasd_init_chunklist(&device->ese_chunks, device->ese_mem, PAGE_SIZE * 4);
spin_lock_init(&device->mem_lock);
atomic_set(&device->tasklet_scheduled, 0);
tasklet_init(&device->tasklet, dasd_device_tasklet,
@@ -138,9 +160,11 @@ struct dasd_device *dasd_alloc_device(void)
void dasd_free_device(struct dasd_device *device)
{
kfree(device->private);
free_pages((unsigned long) device->ese_mem, 1);
free_page((unsigned long) device->erp_mem);
free_pages((unsigned long) device->ccw_mem, 1);
free_pages((unsigned long)device->ese_mem, 2);
free_page((unsigned long)device->erp_mem);
free_pages((unsigned long)device->fill_mem, 1);
free_pages((unsigned long)device->ccw_mem, 2);
free_page((unsigned long)device->nulldata);
kfree(device);
}
@@ -164,6 +188,8 @@ struct dasd_block *dasd_alloc_block(void)
spin_lock_init(&block->queue_lock);
INIT_LIST_HEAD(&block->format_list);
spin_lock_init(&block->format_lock);
INIT_LIST_HEAD(&block->ese_staging);
spin_lock_init(&block->ese_lock);
timer_setup(&block->timer, dasd_block_timeout, 0);
spin_lock_init(&block->profile.lock);
@@ -328,17 +354,19 @@ static int dasd_state_basic_to_ready(struct dasd_device *device)
*/
lim.dma_alignment = lim.logical_block_size - 1;
if (device->discipline->has_discard) {
if (device->features & DASD_FEATURE_DISCARD) {
unsigned int max_bytes;
lim.discard_granularity = block->bp_block;
/* Calculate max_discard_sectors and make it PAGE aligned */
max_bytes = USHRT_MAX * block->bp_block;
max_bytes = ALIGN_DOWN(max_bytes, PAGE_SIZE);
lim.max_hw_discard_sectors = max_bytes / block->bp_block;
lim.max_write_zeroes_sectors = lim.max_hw_discard_sectors;
if (device->discipline->disc_limits) {
device->discipline->disc_limits(block, &lim);
} else {
lim.discard_granularity = block->bp_block;
/* Calculate max_discard_sectors and make it PAGE aligned */
max_bytes = USHRT_MAX * block->bp_block;
max_bytes = ALIGN_DOWN(max_bytes, PAGE_SIZE);
lim.max_hw_discard_sectors = max_bytes / block->bp_block;
lim.max_write_zeroes_sectors = lim.max_hw_discard_sectors;
}
}
rc = queue_limits_commit_update(block->gdp->queue, &lim);
if (rc)
@@ -364,7 +392,8 @@ int _wait_for_empty_queues(struct dasd_device *device)
{
if (device->block)
return list_empty(&device->ccw_queue) &&
list_empty(&device->block->ccw_queue);
list_empty(&device->block->ccw_queue) &&
list_empty(&device->block->ese_staging);
else
return list_empty(&device->ccw_queue);
}
@@ -1224,7 +1253,18 @@ void dasd_sfree_request(struct dasd_ccw_req *cqr, struct dasd_device *device)
unsigned long flags;
spin_lock_irqsave(&device->mem_lock, flags);
dasd_free_chunk(&device->ccw_chunks, cqr->mem_chunk);
/*
* Free the request block from the pool it came from: smalloc() sets
* mem_chunk (ccw_chunks), fmalloc() leaves it NULL (ese_chunks). A
* full-track request also frees its track-filler buffer.
*/
if (cqr->filldata)
dasd_free_chunk(&device->fill_chunks, cqr->filldata);
if (cqr->mem_chunk)
dasd_free_chunk(&device->ccw_chunks, cqr->mem_chunk);
else
dasd_free_chunk(&device->ese_chunks, cqr);
spin_unlock_irqrestore(&device->mem_lock, flags);
dasd_put_device(device);
}
@@ -1235,6 +1275,8 @@ void dasd_ffree_request(struct dasd_ccw_req *cqr, struct dasd_device *device)
unsigned long flags;
spin_lock_irqsave(&device->mem_lock, flags);
if (cqr->filldata)
dasd_free_chunk(&device->fill_chunks, cqr->filldata);
dasd_free_chunk(&device->ese_chunks, cqr);
spin_unlock_irqrestore(&device->mem_lock, flags);
dasd_put_device(device);
@@ -1362,13 +1404,6 @@ int dasd_start_IO(struct dasd_ccw_req *cqr)
if (!cqr->lpm)
cqr->lpm = dasd_path_get_opm(device);
}
/*
* remember the amount of formatted tracks to prevent double format on
* ESE devices
*/
if (cqr->block)
cqr->trkcount = atomic_read(&cqr->block->trkcount);
if (cqr->cpmode == 1) {
rc = ccw_device_tm_start(device->cdev, cqr->cpaddr,
(long) cqr, cqr->lpm);
@@ -1585,7 +1620,7 @@ static int dasd_ese_oos_cond(u8 *sense)
void dasd_int_handler(struct ccw_device *cdev, unsigned long intparm,
struct irb *irb)
{
struct dasd_ccw_req *cqr, *next, *fcqr;
struct dasd_ccw_req *cqr, *next;
struct dasd_device *device;
unsigned long now;
int nrf_suppressed = 0;
@@ -1698,33 +1733,32 @@ void dasd_int_handler(struct ccw_device *cdev, unsigned long intparm,
return;
}
if (rq_data_dir(req) == READ) {
device->discipline->ese_read(cqr, irb);
cqr->status = DASD_CQR_SUCCESS;
if (device->discipline->ese_read(cqr, irb))
cqr->status = DASD_CQR_ERROR;
else
cqr->status = DASD_CQR_SUCCESS;
cqr->stopclk = now;
dasd_device_clear_timer(device);
dasd_schedule_device_bh(device);
return;
}
fcqr = device->discipline->ese_format(device, cqr, irb);
if (IS_ERR(fcqr)) {
if (PTR_ERR(fcqr) == -EINVAL) {
cqr->status = DASD_CQR_ERROR;
return;
}
if (cqr->filldata) {
/*
* If we can't format now, let the request go
* one extra round. Maybe we can format later.
* A WRITE_FULL_TRACK cqr carries the complete
* track image; INV_TRACK_FORMAT here means the
* generated image or the media itself is bad, not
* that the track still needs formatting - retrying
* via ese_format() would just resubmit the same
* write. Let it fail instead.
*/
cqr->status = DASD_CQR_QUEUED;
dasd_schedule_device_bh(device);
return;
} else {
fcqr->status = DASD_CQR_QUEUED;
cqr->status = DASD_CQR_QUEUED;
list_add(&fcqr->devlist, &device->ccw_queue);
cqr->status = DASD_CQR_ERROR;
cqr->stopclk = now;
dasd_device_clear_timer(device);
dasd_schedule_device_bh(device);
return;
}
device->discipline->ese_format(device, cqr, irb);
return;
}
/* Check for clear pending */
@@ -1883,6 +1917,17 @@ static void __dasd_process_cqr(struct dasd_device *device,
case DASD_CQR_CLEARED:
cqr->status = DASD_CQR_TERMINATED;
break;
case DASD_CQR_ABORT:
cqr->status = DASD_CQR_ABORTED;
/*
* ABORT is only set on the block-layer origin write that a
* full-track format replaces. Clear the callback so the request
* is not completed here - the replacement completes it. Internal
* requests never take this path, so no sleep_on waiter is left
* without its wakeup.
*/
cqr->callback = NULL;
break;
default:
dev_err(&device->cdev->dev,
"Unexpected CQR status %02x", cqr->status);
@@ -2210,6 +2255,7 @@ EXPORT_SYMBOL(dasd_add_request_tail);
void dasd_wakeup_cb(struct dasd_ccw_req *cqr, void *data)
{
spin_lock_irq(get_ccwdev_lock(cqr->startdev->cdev));
cqr->endclk = get_tod_clock();
cqr->callback_data = DASD_SLEEPON_END_TAG;
spin_unlock_irq(get_ccwdev_lock(cqr->startdev->cdev));
wake_up(&generic_waitq);
@@ -2686,6 +2732,13 @@ static void __dasd_process_erp(struct dasd_device *device,
if (cqr->status == DASD_CQR_DONE)
DBF_DEV_EVENT(DBF_NOTICE, device, "%s", "ERP successful");
else if (cqr->status == DASD_CQR_ABORTED)
/*
* ESE format aborts the request and replaces it with a format
* CQR - this is not an ERP failure.
*/
DBF_DEV_EVENT(DBF_NOTICE, device, "%s",
"ERP request aborted, replaced by ESE format");
else
dev_err(&device->cdev->dev, "ERP failed for the DASD\n");
erp_fn = device->discipline->erp_postaction(cqr);
@@ -2697,17 +2750,23 @@ static void __dasd_cleanup_cqr(struct dasd_ccw_req *cqr)
struct request *req;
blk_status_t error = BLK_STS_OK;
unsigned int proc_bytes;
int status;
int status, intrc;
req = (struct request *) cqr->callback_data;
dasd_profile_end(cqr->block, cqr, req);
/*
* free_cp() returns the request block to its memory pool, so snapshot
* everything still needed from cqr before calling it - another CPU can
* reallocate and overwrite the block right after.
*/
proc_bytes = cqr->proc_bytes;
intrc = cqr->intrc;
status = cqr->block->base->discipline->free_cp(cqr, req);
if (status < 0)
error = errno_to_blk_status(status);
else if (status == 0) {
switch (cqr->intrc) {
switch (intrc) {
case -EPERM:
/*
* DASD doesn't implement SCSI/NVMe reservations, but it
@@ -2726,6 +2785,9 @@ static void __dasd_cleanup_cqr(struct dasd_ccw_req *cqr)
error = BLK_STS_IOERR;
break;
}
} else if (status == DASD_CQR_ABORTED) {
/* aborted requests are replaced with a new one so do not complete this */
return;
}
/*
@@ -2771,7 +2833,8 @@ restart:
if (cqr->status != DASD_CQR_DONE &&
cqr->status != DASD_CQR_FAILED &&
cqr->status != DASD_CQR_NEED_ERP &&
cqr->status != DASD_CQR_TERMINATED)
cqr->status != DASD_CQR_TERMINATED &&
cqr->status != DASD_CQR_ABORTED)
continue;
if (cqr->status == DASD_CQR_TERMINATED) {
@@ -2819,6 +2882,28 @@ restart:
static void dasd_return_cqr_cb(struct dasd_ccw_req *cqr, void *data)
{
struct dasd_ccw_req *temp_cqr;
struct dasd_block *block;
/* only format CQRs are candidates */
if (!cqr->block || unlikely(!cqr->format))
goto out;
block = cqr->block;
/*
* Mark in-flight (IN_IO) CQRs that overlap this just-completed format
* range so they re-check in test_and_set_format on completion; FILLED
* or QUEUED CQRs re-check the format_list on their next round anyway.
*/
list_for_each_entry(temp_cqr, &block->ccw_queue, blocklist) {
if (temp_cqr != cqr &&
temp_cqr->status != DASD_CQR_FILLED &&
temp_cqr->status != DASD_CQR_QUEUED &&
dasd_req_conflict(cqr, temp_cqr)) {
WRITE_ONCE(temp_cqr->collision, true);
}
}
out:
dasd_schedule_block_bh(cqr->block);
}
@@ -2882,6 +2967,14 @@ static void dasd_block_tasklet(unsigned long data)
atomic_set(&block->tasklet_scheduled, 0);
INIT_LIST_HEAD(&final_queue);
spin_lock_irq(&block->queue_lock);
/*
* Splice the hardirq-staged ESE format CQRs onto ccw_queue. Splice to
* the tail so an aborted origin request (already on ccw_queue) is
* retired before its format-CQR replacement completes and requeues it.
*/
spin_lock(&block->ese_lock);
list_splice_tail_init(&block->ese_staging, &block->ccw_queue);
spin_unlock(&block->ese_lock);
/* Finish off requests on ccw queue */
__dasd_process_block_ccw_queue(block, &final_queue);
spin_unlock_irq(&block->queue_lock);
@@ -2941,6 +3034,15 @@ static int _dasd_requests_to_flushqueue(struct dasd_block *block,
int rc, i;
spin_lock_irqsave(&block->queue_lock, flags);
/*
* Splice any hardirq-staged ESE format CQRs onto ccw_queue first so
* they are seen and canceled by the walk below instead of being
* orphaned across this flush / state transition. Mirrors the splice
* in dasd_block_tasklet().
*/
spin_lock(&block->ese_lock);
list_splice_tail_init(&block->ese_staging, &block->ccw_queue);
spin_unlock(&block->ese_lock);
rc = 0;
restart:
list_for_each_entry_safe(cqr, n, &block->ccw_queue, blocklist) {
@@ -3036,6 +3138,7 @@ static blk_status_t do_dasd_request(struct blk_mq_hw_ctx *hctx,
struct dasd_device *basedev;
struct dasd_ccw_req *cqr;
blk_status_t rc = BLK_STS_OK;
bool complete_noop = false;
basedev = block->base;
spin_lock_irq(&dq->lock);
@@ -3084,6 +3187,17 @@ static blk_status_t do_dasd_request(struct blk_mq_hw_ctx *hctx,
rc = BLK_STS_RESOURCE;
} else if (PTR_ERR(cqr) == -EINVAL) {
rc = BLK_STS_INVAL;
} else if (PTR_ERR(cqr) == -EOPNOTSUPP) {
/*
* A discard that covers no whole extent releases
* nothing. Discard is advisory, so complete it as a
* benign no-op: the device does support discard, this
* range just does not align to the large ESE extent
* granularity.
* Completed after the lock is dropped.
*/
rc = BLK_STS_OK;
complete_noop = true;
} else {
DBF_DEV_EVENT(DBF_ERR, basedev,
"CCW creation failed (rc=%ld) on request %p",
@@ -3092,6 +3206,13 @@ static blk_status_t do_dasd_request(struct blk_mq_hw_ctx *hctx,
}
goto out;
}
if (!cqr) {
/* build_cp may collapse a non-transient build error to NULL */
DBF_DEV_EVENT(DBF_ERR, basedev,
"CCW creation returned NULL on request %p", req);
rc = BLK_STS_IOERR;
goto out;
}
/*
* Note: callback is set to dasd_return_cqr_cb in
* __dasd_block_start_head to cover erp requests as well
@@ -3110,6 +3231,8 @@ static blk_status_t do_dasd_request(struct blk_mq_hw_ctx *hctx,
out:
spin_unlock_irq(&dq->lock);
if (complete_noop)
blk_mq_end_request(req, BLK_STS_OK);
return rc;
}
@@ -3885,6 +4008,19 @@ restart_cb:
*/
goto restart_cb;
}
/*
* An aborted request was replaced by a full-track write and is
* retired by that replacement; do not requeue it, just release
* it (mirrors the DASD_CQR_ABORTED handling in
* __dasd_cleanup_cqr()).
*/
if (cqr->status == DASD_CQR_ABORTED) {
struct request *req = cqr->callback_data;
list_del_init(&cqr->blocklist);
cqr->block->base->discipline->free_cp(cqr, req);
continue;
}
_dasd_requeue_request(cqr);
list_del_init(&cqr->blocklist);
cqr->block->base->discipline->free_cp(
drivers/s390/block/dasd_3990_erp.c
+1
View File
@@ -2400,6 +2400,7 @@ static struct dasd_ccw_req *dasd_3990_erp_add_erp(struct dasd_ccw_req *cqr)
erp->startdev = device;
erp->memdev = device;
erp->block = cqr->block;
erp->filldata = cqr->filldata;
erp->magic = cqr->magic;
erp->expires = cqr->expires;
erp->retries = device->default_retries;
drivers/s390/block/dasd_devmap.c
+88 -2
View File
@@ -1630,6 +1630,88 @@ dasd_expires_store(struct device *dev, struct device_attribute *attr,
static DEVICE_ATTR(expires, 0644, dasd_expires_show, dasd_expires_store);
/* ESE fulltrack write aggressiveness knob (0..100, see DASD_FT_BIAS_*) */
static ssize_t
full_track_bias_show(struct device *dev, struct device_attribute *attr, char *buf)
{
struct dasd_device *device;
int len;
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
return -ENODEV;
len = sysfs_emit(buf, "%u\n", device->ft_bias);
dasd_put_device(device);
return len;
}
static ssize_t full_track_bias_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
struct dasd_device *device;
unsigned int val;
if (kstrtouint(buf, 0, &val) || val > DASD_FT_BIAS_MAX)
return -EINVAL;
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
return -ENODEV;
/*
* ft_bias is the tuning target; fulltrack is a best-effort mode hint
* that the per-IO heuristic also updates locklessly. A racing writer can
* at most leave a transient mismatch that self-corrects on the next IO,
* never corruption, so the update is left unlocked.
*/
device->ft_bias = val;
dasd_ft_bias_apply(device);
dasd_put_device(device);
return count;
}
static DEVICE_ATTR_RW(full_track_bias);
static const char * const dasd_ese_heu_state_names[] = {
[DASD_ESE_HEU_FT1_ACTIVE] = "fulltrack active",
[DASD_ESE_HEU_PROBING] = "probing",
[DASD_ESE_HEU_FT0_STABLE] = "fulltrack inactive",
};
/* read-only: current full-track mode / adaptive FSM state, for observability */
static ssize_t
ese_heuristic_state_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
struct dasd_device *device;
unsigned int state;
int len;
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
return -ENODEV;
if (device->ft_bias == 0) {
len = sysfs_emit(buf, "fulltrack deactivated\n");
} else if (device->ft_bias >= DASD_FT_BIAS_MAX) {
len = sysfs_emit(buf, "fulltrack permanent active\n");
} else if (!dasd_ese_adaptive(device)) {
/* adaptive range but not ESE: the heuristic does not run */
len = sysfs_emit(buf, "fulltrack deactivated\n");
} else {
state = device->ese_probe_state;
if (state < ARRAY_SIZE(dasd_ese_heu_state_names))
len = sysfs_emit(buf, "%s\n", dasd_ese_heu_state_names[state]);
else
len = sysfs_emit(buf, "unknown\n");
}
dasd_put_device(device);
return len;
}
static DEVICE_ATTR_RO(ese_heuristic_state);
static ssize_t
dasd_retries_show(struct device *dev, struct device_attribute *attr, char *buf)
{
@@ -2400,9 +2482,10 @@ static ssize_t dasd_##_name##_show(struct device *dev, \
\
return sysfs_emit(buf, "%d\n", val); \
} \
static DEVICE_ATTR(_name, 0444, dasd_##_name##_show, NULL); \
static DEVICE_ATTR(_name, 0444, dasd_##_name##_show, NULL);
DASD_DEFINE_ATTR(ese, device->discipline->is_ese);
DASD_DEFINE_ATTR(ese, device->discipline->ese_capable);
DASD_DEFINE_ATTR(on_demand_formatting, device->discipline->on_demand_format);
DASD_DEFINE_ATTR(extent_size, device->discipline->ext_size);
DASD_DEFINE_ATTR(pool_id, device->discipline->ext_pool_id);
DASD_DEFINE_ATTR(space_configured, device->discipline->space_configured);
@@ -2425,6 +2508,8 @@ static struct attribute * dasd_attrs[] = {
&dev_attr_erplog.attr,
&dev_attr_failfast.attr,
&dev_attr_expires.attr,
&dev_attr_full_track_bias.attr,
&dev_attr_ese_heuristic_state.attr,
&dev_attr_retries.attr,
&dev_attr_timeout.attr,
&dev_attr_reservation_policy.attr,
@@ -2438,6 +2523,7 @@ static struct attribute * dasd_attrs[] = {
&dev_attr_path_reset.attr,
&dev_attr_hpf.attr,
&dev_attr_ese.attr,
&dev_attr_on_demand_formatting.attr,
&dev_attr_fc_security.attr,
&dev_attr_copy_pair.attr,
&dev_attr_copy_role.attr,
drivers/s390/block/dasd_eckd.c
+1133 -137
View File
File diff suppressed because it is too large Load Diff
drivers/s390/block/dasd_eckd.h
+67 -1
View File
@@ -131,7 +131,7 @@
/*
* Maximum number of blocks to be chained
*/
#define DASD_ECKD_MAX_BLOCKS 190
#define DASD_ECKD_MAX_BLOCKS 180
#define DASD_ECKD_MAX_BLOCKS_RAW 256
/*****************************************************************************
@@ -146,6 +146,52 @@ struct eckd_count {
__u16 dl;
} __attribute__ ((packed));
struct eckd_r0 {
struct eckd_count count;
__u8 data[8];
} __packed;
/*
* Extended Address Volume track address: the head field carries the actual
* head in its low-order 4 bits; the cylinder bits that do not fit the 16-bit
* cyl field are shifted in just above them.
*/
#define DASD_EAV_CYL_HI_SHIFT 16 /* cylinder bits beyond the 16-bit cyl field */
#define DASD_EAV_HEAD_HI_SHIFT 4 /* head occupies the low-order 4 bits of head */
/*
* On-disk DASD format label.
*
* Written into track 0, head 0, record 4 (R4 - the first non-special CDL
* record) as part of the same channel program that formats track 0, so it is
* stored atomically with the track: either both the track format and the label
* make it to disk or neither does. Its presence with a valid magic therefore
* marks a completed format and can be used for format detection.
*
* The structure is exactly the smallest supported block size (512 bytes) so it
* always fits into a single record.
* For larger block sizes the rest of the record is zero padded.
* The magic together with the version is used to recognise a valid label.
*/
#define DASD_ESE_LABEL_MAGIC 0xC4C1E2C4C6D4E3F1ULL /* EBCDIC "DASDFMT1" */
#define DASD_ESE_LABEL_VERSION 1
/* dasd_format_label.flags */
#define DASD_ESE_LABEL_F_ESE 0x00000001 /* volume is extent space efficient */
#define DASD_ESE_LABEL_F_QUICK 0x00000002 /* quick (space released) format */
#define DASD_ESE_LABEL_F_FULL 0x00000004 /* full format */
struct dasd_format_label {
__u64 magic; /* DASD_ESE_LABEL_MAGIC */
__u32 version; /* DASD_ESE_LABEL_VERSION */
__u32 flags; /* DASD_ESE_LABEL_F_* */
__u32 blksize; /* block size the volume was formatted with */
__u32 reserved0;
__u64 format_tod; /* TOD clock at format time */
__u8 kernel_version[64]; /* NUL terminated kernel release (uname -r) */
__u8 reserved[416]; /* pad the struct to 512 bytes */
} __packed;
struct ch_t {
__u16 cyl;
__u16 head;
@@ -690,12 +736,32 @@ struct dasd_eckd_private {
/* alias management */
struct dasd_uid uid;
/*
* Cached copies of conf.ned->ID (the LSS) and conf.ned->unit_addr,
* refreshed under ccwdev_lock. Kept outside uid because create_uid()
* memsets uid before repopulating it, which would expose a transient
* zero to the lockless CCW-build readers.
*/
__u8 ned_lss;
__u8 ned_ua;
struct alias_pav_group *pavgroup;
struct alias_lcu *lcu;
int count;
u32 fcx_max_data;
char suc_reason;
/*
* Set when the whole volume's space was released (full RAS); consumed by
* the next format to mark the on-disk label as a quick (vs full) format.
*/
int ese_format_quick;
/*
* Cached on-disk format label (R4), read at online and refreshed on
* format. When valid, is_ese() is derived from it; otherwise it falls
* back to the hardware ESE field (vsq.vol_info.ese).
*/
struct dasd_format_label ese_label;
bool ese_label_valid;
};
drivers/s390/block/dasd_erp.c
+9 -2
View File
@@ -120,16 +120,19 @@ dasd_default_erp_action(struct dasd_ccw_req *cqr)
*/
struct dasd_ccw_req *dasd_default_erp_postaction(struct dasd_ccw_req *cqr)
{
int success;
int success, aborted;
unsigned long startclk, stopclk;
struct dasd_device *startdev;
unsigned int proc_bytes;
BUG_ON(cqr->refers == NULL || cqr->function == NULL);
success = cqr->status == DASD_CQR_DONE;
aborted = cqr->status == DASD_CQR_ABORTED;
startclk = cqr->startclk;
stopclk = cqr->stopclk;
startdev = cqr->startdev;
proc_bytes = cqr->proc_bytes;
/* free all ERPs - but NOT the original cqr */
while (cqr->refers != NULL) {
@@ -147,7 +150,11 @@ struct dasd_ccw_req *dasd_default_erp_postaction(struct dasd_ccw_req *cqr)
cqr->startclk = startclk;
cqr->stopclk = stopclk;
cqr->startdev = startdev;
if (success)
cqr->proc_bytes = proc_bytes;
if (aborted)
/* base request is owned by the ESE format replacement CQR */
cqr->status = DASD_CQR_ABORTED;
else if (success)
cqr->status = DASD_CQR_DONE;
else {
cqr->status = DASD_CQR_FAILED;
drivers/s390/block/dasd_int.h
+145 -4
View File
@@ -159,6 +159,11 @@ struct dasd_ccw_req {
void *callback_data;
unsigned int proc_bytes; /* bytes for partial completion */
unsigned int trkcount; /* count formatted tracks */
void *filldata; /* address of filler data */
struct dasd_format_entry *format;
sector_t start_trk;
sector_t end_trk;
bool collision;
};
/*
@@ -170,6 +175,7 @@ struct dasd_ccw_req {
#define DASD_CQR_IN_ERP 0x03 /* request is in recovery */
#define DASD_CQR_FAILED 0x04 /* request is finally failed */
#define DASD_CQR_TERMINATED 0x05 /* request was stopped by driver */
#define DASD_CQR_ABORTED 0x06 /* request was replaced and will be deleted */
#define DASD_CQR_QUEUED 0x80 /* request is queued to be processed */
#define DASD_CQR_IN_IO 0x81 /* request is currently in IO */
@@ -177,6 +183,7 @@ struct dasd_ccw_req {
#define DASD_CQR_CLEAR_PENDING 0x83 /* request is clear pending */
#define DASD_CQR_CLEARED 0x84 /* request was cleared */
#define DASD_CQR_SUCCESS 0x85 /* request was successful */
#define DASD_CQR_ABORT 0x86 /* request was replaced and will not be handled */
/* default expiration time*/
#define DASD_EXPIRES 300
@@ -394,6 +401,11 @@ struct dasd_discipline {
* Extent Space Efficient (ESE) relevant functions
*/
int (*is_ese)(struct dasd_device *);
int (*ese_capable)(struct dasd_device *);
/* Whether the volume is formatted on demand (thin), from the label */
int (*on_demand_format)(struct dasd_device *);
/* Fill discard queue limits */
void (*disc_limits)(struct dasd_block *, struct queue_limits *);
/* Capacity */
int (*space_allocated)(struct dasd_device *);
int (*space_configured)(struct dasd_device *);
@@ -406,8 +418,7 @@ struct dasd_discipline {
int (*ext_pool_warn_thrshld)(struct dasd_device *);
int (*ext_pool_oos)(struct dasd_device *);
int (*ext_pool_exhaust)(struct dasd_device *, struct dasd_ccw_req *);
struct dasd_ccw_req *(*ese_format)(struct dasd_device *,
struct dasd_ccw_req *, struct irb *);
void (*ese_format)(struct dasd_device *, struct dasd_ccw_req *, struct irb *);
int (*ese_read)(struct dasd_ccw_req *, struct irb *);
int (*pprc_status)(struct dasd_device *, struct dasd_pprc_data_sc4 *);
bool (*pprc_enabled)(struct dasd_device *);
@@ -538,9 +549,17 @@ struct dasd_profile {
spinlock_t lock;
};
/*
* concurrent ESE format ranges in flight; also caps a WRITE_FULL_TRACK's
* track count, which the LRE track bitmask limits to 16
*/
#define DASD_NR_FORMAT_ENTRIES 16
struct dasd_format_entry {
struct list_head list;
sector_t track;
struct dasd_ccw_req *cqr;
sector_t start_trk;
sector_t end_trk;
};
struct dasd_device {
@@ -573,9 +592,12 @@ struct dasd_device {
struct list_head ccw_queue;
spinlock_t mem_lock;
void *ccw_mem;
void *fill_mem;
void *erp_mem;
void *ese_mem;
void *nulldata;
struct list_head ccw_chunks;
struct list_head fill_chunks;
struct list_head erp_chunks;
struct list_head ese_chunks;
@@ -607,11 +629,24 @@ struct dasd_device {
struct dentry *debugfs_dentry;
struct dentry *hosts_dentry;
struct dasd_profile profile;
struct dasd_format_entry format_entry;
struct dasd_format_entry format_entry[DASD_NR_FORMAT_ENTRIES];
struct kset *paths_info;
struct dasd_copy_relation *copy;
unsigned long aq_mask;
unsigned int aq_timeouts;
/* ESE fulltrack write control (see full_track_bias sysfs attribute) */
unsigned int ft_bias; /* aggressiveness 0..100: 0=off, 100=always */
unsigned int fulltrack; /* internal: use WRITE_FULL_TRACK for aligned writes */
/* adaptive heuristic (active for ft_bias 1..99), derived from ft_bias */
unsigned int ese_probe_state; /* heuristic FSM state */
unsigned int ese_probe_interval; /* IOs between evaluations */
atomic_t ese_io_cnt; /* IO counter for current window */
atomic_t ese_nrf_window; /* NRF/INV_TRACK_FORMAT events in window */
unsigned int ese_heu_start_interval; /* IOs before first probe */
unsigned int ese_heu_probe_window; /* IOs in probe window */
unsigned int ese_heu_max_interval; /* max IOs between probes (backoff cap) */
unsigned int ese_heu_nrf_high; /* NRF per-mille threshold → activate ft1 */
};
struct dasd_block {
@@ -640,6 +675,15 @@ struct dasd_block {
struct list_head format_list;
spinlock_t format_lock;
atomic_t trkcount;
/*
* ESE format CQRs staged from hardirq, spliced into
* ccw_queue in dasd_block_tasklet under queue_lock. Direct enqueue from
* the IRQ handler would invert the queue_lock / ccwdev_lock order.
*/
struct list_head ese_staging;
/* lock for ese_staging */
spinlock_t ese_lock;
};
struct dasd_attention_data {
@@ -660,6 +704,39 @@ struct dasd_queue {
#define DASD_STOPPED_PPRC 32 /* PPRC swap */
#define DASD_STOPPED_NOSPC 128 /* no space left */
/*
* ESE fulltrack write aggressiveness (full_track_bias sysfs attribute), 0..100:
* 0 - never use proactively WRITE_FULL_TRACK
* 100 - always use proactively WRITE_FULL_TRACK, no probing
* 1..99 - adaptive; higher means switch to ft more eagerly
* WRITE_FULL_TRACK has an advantage on sparse formatted ESE devices
* but it has an overall penalty for maximum throughput for fully
* formatted devices.
* The default of 50 tries to balance both and do some probing in between
* to choose the best mode for default IO.
*/
#define DASD_FT_BIAS_MAX 100
#define DASD_FT_BIAS_DEFAULT 50
/* ESE fulltrack heuristic FSM states (adaptive range, ft_bias 1..99) */
#define DASD_ESE_HEU_FT1_ACTIVE 0 /* fulltrack write active */
#define DASD_ESE_HEU_PROBING 1 /* ft0 probe window, measuring NRF rate */
#define DASD_ESE_HEU_FT0_STABLE 2 /* device formatted, ft0 active */
/*
* Heuristic parameters are derived from ft_bias by linear interpolation,
* anchored so that ft_bias == 50 reproduces the previously shipped defaults
* and ft_bias == 100 is the most aggressive end of the range.
* probe_window is constant.
*/
#define DASD_ESE_HEU_PROBE_WINDOW 100
#define DASD_ESE_HEU_NRF_HIGH_A50 10 /* NRF per-mille threshold */
#define DASD_ESE_HEU_NRF_HIGH_A100 1
#define DASD_ESE_HEU_START_A50 2000 /* IOs before first probe */
#define DASD_ESE_HEU_START_A100 500
#define DASD_ESE_HEU_MAX_A50 500000 /* backoff cap */
#define DASD_ESE_HEU_MAX_A100 20000
/* per device flags */
#define DASD_FLAG_OFFLINE 3 /* device is in offline processing */
#define DASD_FLAG_EER_SNSS 4 /* A SNSS is required */
@@ -815,6 +892,70 @@ static inline void *dasd_get_callback_data(struct dasd_ccw_req *cqr)
return cqr->callback_data;
}
static inline bool dasd_req_conflict(struct dasd_ccw_req *cqr1,
struct dasd_ccw_req *cqr2)
{
return !(cqr1->format->end_trk < cqr2->start_trk ||
cqr2->end_trk < cqr1->format->start_trk);
}
/*
* true when device is ese device and ft_bias selects the adaptive
* heuristic (neither hard endpoint)
*/
static inline bool dasd_ese_adaptive(struct dasd_device *device)
{
return device->discipline &&
device->discipline->is_ese &&
device->discipline->is_ese(device) &&
device->ft_bias > 0 &&
device->ft_bias < DASD_FT_BIAS_MAX;
}
/*
* Linear interpolation of a heuristic parameter between its value at aggr==50
* (v50) and its value at aggr==100 (v100).
*/
static inline unsigned int dasd_ese_lerp(unsigned int v50, unsigned int v100,
unsigned int aggr)
{
return (unsigned int)((int)v50 +
((int)v100 - (int)v50) * ((int)aggr - 50) / 50);
}
/*
* Apply the ft_bias knob. For the hard endpoints just pin the mode; for the
* adaptive range derive the heuristic parameters from ft_bias and (re)start
* the FSM in ft1 so a freshly sparse device avoids the NRF penalty right away.
*/
static inline void dasd_ft_bias_apply(struct dasd_device *device)
{
unsigned int a = device->ft_bias;
if (!dasd_ese_adaptive(device)) {
device->fulltrack = (a >= DASD_FT_BIAS_MAX) ? 1 : 0;
device->ese_probe_state = DASD_ESE_HEU_FT1_ACTIVE;
return;
}
device->ese_heu_nrf_high =
dasd_ese_lerp(DASD_ESE_HEU_NRF_HIGH_A50,
DASD_ESE_HEU_NRF_HIGH_A100, a);
device->ese_heu_start_interval =
dasd_ese_lerp(DASD_ESE_HEU_START_A50,
DASD_ESE_HEU_START_A100, a);
device->ese_heu_max_interval =
dasd_ese_lerp(DASD_ESE_HEU_MAX_A50,
DASD_ESE_HEU_MAX_A100, a);
device->ese_heu_probe_window = DASD_ESE_HEU_PROBE_WINDOW;
device->ese_probe_state = DASD_ESE_HEU_FT1_ACTIVE;
device->ese_probe_interval = device->ese_heu_start_interval;
device->fulltrack = 1;
atomic_set(&device->ese_io_cnt, 0);
atomic_set(&device->ese_nrf_window, 0);
}
/* externals in dasd.c */
#define DASD_PROFILE_OFF 0
#define DASD_PROFILE_ON 1
fs/aio.c
+1 -1
View File
@@ -35,7 +35,7 @@
#include <linux/workqueue.h>
#include <linux/security.h>
#include <linux/eventfd.h>
#include <linux/blkdev.h>
#include <linux/blk_plug.h>
#include <linux/compat.h>
#include <linux/migrate.h>
#include <linux/ramfs.h>
fs/btrfs/misc.h
+1 -1
View File
@@ -74,7 +74,7 @@ static inline struct bvec_iter init_bvec_iter_for_bio(struct bio *bio)
.bi_sector = 0,
.bi_size = bio_size,
.bi_idx = 0,
.bi_bvec_done = 0,
.bi_offset = 0,
};
}
fs/buffer.c
+3
View File
@@ -1107,6 +1107,9 @@ static void __bh_submit(struct buffer_head *bh, blk_opf_t opf,
bio = bio_alloc(bh->b_bdev, 1, opf, GFP_NOIO);
if (folio_test_dropbehind(bh->b_folio) && op_is_write(opf))
bio_set_flag(bio, BIO_COMPLETE_IN_TASK);
if (IS_ENABLED(CONFIG_FS_ENCRYPTION))
buffer_set_crypto_ctx(bio, bh, GFP_NOIO);
fs/erofs/zdata.c
+1 -10
View File
@@ -1432,15 +1432,6 @@ static void z_erofs_decompressqueue_kthread_work(struct kthread_work *work)
}
#endif
/* Use (kthread_)work in atomic contexts to minimize scheduling overhead */
static inline bool z_erofs_in_atomic(void)
{
if (IS_ENABLED(CONFIG_PREEMPTION) && rcu_preempt_depth())
return true;
if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
return true;
return !preemptible();
}
static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io,
int bios)
@@ -1457,7 +1448,7 @@ static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io,
if (atomic_add_return(bios, &io->pending_bios))
return;
if (z_erofs_in_atomic()) {
if (bio_in_atomic()) {
/* See `sync_decompress` in sysfs-fs-erofs for more details */
if (sbi->sync_decompress == EROFS_SYNC_DECOMPRESS_AUTO)
sbi->sync_decompress = EROFS_SYNC_DECOMPRESS_FORCE_ON;
fs/fs-writeback.c
+1 -1
View File
@@ -25,7 +25,7 @@
#include <linux/pagemap.h>
#include <linux/kthread.h>
#include <linux/writeback.h>
#include <linux/blkdev.h>
#include <linux/blk_plug.h>
#include <linux/backing-dev.h>
#include <linux/tracepoint.h>
#include <linux/device.h>
fs/iomap/direct-io.c
+4 -1
View File
@@ -359,6 +359,7 @@ static ssize_t iomap_dio_bio_iter_one(struct iomap_iter *iter,
iomap_max_bio_size(&iter->iomap), alignment);
else
ret = bio_iov_iter_get_pages(bio, dio->submit.iter,
bdev_dma_alignment(bio->bi_bdev),
alignment - 1);
if (unlikely(ret))
goto out_put_bio;
@@ -1033,7 +1034,9 @@ ssize_t __iomap_dio_read_simple(struct kiocb *iocb, struct iov_iter *iter,
bio->bi_iter.bi_sector = iomap_sector(&iomi->iomap, iomi->pos);
bio->bi_ioprio = iocb->ki_ioprio;
ret = bio_iov_iter_get_pages(bio, iter, alignment - 1);
ret = bio_iov_iter_get_pages(bio, iter,
bdev_dma_alignment(bio->bi_bdev),
alignment - 1);
if (unlikely(ret))
goto out_bio_put;
fs/iomap/ioend.c
+3 -2
View File
@@ -239,8 +239,6 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
if (wpc->iomap.flags & IOMAP_F_SHARED)
ioend_flags |= IOMAP_IOEND_SHARED;
if (folio_test_dropbehind(folio))
ioend_flags |= IOMAP_IOEND_DONTCACHE;
if (pos == wpc->iomap.offset && (wpc->iomap.flags & IOMAP_F_BOUNDARY))
ioend_flags |= IOMAP_IOEND_BOUNDARY;
@@ -257,6 +255,9 @@ new_ioend:
if (!bio_add_folio(&ioend->io_bio, folio, map_len, poff))
goto new_ioend;
if (folio_test_dropbehind(folio))
bio_set_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);
/*
* Clamp io_offset and io_size to the incore EOF so that ondisk
* file size updates in the ioend completion are byte-accurate.

Some files were not shown because too many files have changed in this diff Show More