This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
Pull dmaengine updates from Vinod Koul:
"Core:
- New API to combine configuration and preparation and users
New hardware support:
- Mediatek MT8189 SoC uart dma support
Updates:
- Designware dma driver flatten desc structures and simplify code,
interrupt-path groundwork changes, first part of PCI EP DMA support
- Updates to zynqmp_dma with runtime PM and device removal
improvments
- Xilinx dma optimizations for AXIDMA and MCDMA channel management"
* tag 'dmaengine-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine: (73 commits)
dmaengine: dw-edma: Mark emulated IRQ as level-triggered
dmaengine: idxd: assign all engines to group 0 in IAA defaults
dmaengine: qcom_hidma: remove conditional return with no effect
dmaengine: qcom-bam-dma: fix autosuspend cleanup during removal
dmaengine: fsl-edma: tracing: no ptr dereference during log output
dmaengine: dw-edma: Program endpoint function numbers
dmaengine: dw-edma-pcie: Add chip flags to match data
dmaengine: dw-edma-pcie: Handle optional data blocks
dmaengine: dw-edma-pcie: Factor out descriptor block address lookup
dmaengine: dw-edma-pcie: Add register offset match flag
dmaengine: dw-edma-pcie: Add platform ops to match data
dmaengine: dw-edma-pcie: Rename vsec_data to dma_data
dmaengine: dw-edma-pcie: Add capability match data
dmaengine: dw-edma-pcie: Track non-LL mode in DMA data
dmaengine: dw-edma: Add partial channel ownership mode
dmaengine: dw-edma: Initialize IRQ data before requesting IRQs
dmaengine: dw-edma: Add core quiesce operations
dmaengine: dw-edma: Add per-channel interrupt routing control
dmaengine: dw-edma: Factor out HDMA interrupt setup helper
dmaengine: dw-edma: Defer channel IRQ handling to workqueue
...
Pull PCI updates from Bjorn Helgaas:
"Resource management:
- Add hotplug reservation only once (not at each level of the
hierarchy) so bridge windows don't grow more than necessary (Ilpo
Järvinen)
Driver binding:
- Rework device matching so device ID lifetime only needs to cover
the probe path since dynamic IDs can be removed at any time (Gary
Guo)
Error handling:
- Update mappings of AER errors to agent & layer and log them for
each individual error when multiple errors detected (Lukas Wunner)
- Log Error Source only once, not twice in separate messages (Lukas
Wunner)
- Emit TLP Log only for unmasked errors (Lukas Wunner)
- Support Advisory Non-Fatal Errors (Lukas Wunner)
- Allow DPC on all Downstream Ports, not just Root Ports, when OS
controls AER (Darshit Shah)
ASPM:
- Program the same ASPM Control values for every function of
multi-function devices, as recommended by the PCIe spec (Krishna
Chaitanya Chundru)
- Avoid L0s for Realtek RTS525A, where it causes an AER interrupt
storm (Max Lee)
- Avoid ASPM L0s, L1, and L1 PM Substates based on 'aspm-no-l0s',
'aspm-no-l1' [1], and 'aspm-no-l1ss' DT properties (Krishna
Chaitanya Chundru)
Power management:
- Allow D3 for native hotplug-capable Root Ports on non-x86 platforms
(we avoid D3 for these ports on x86 because some old platforms
didn't validate it) (Manivannan Sadhasivam)
- Allow portdrv to claim Ports even if they don't support services
(AER, PME, DPC, hotplug, etc) so it can do power management (Brian
Norris)
Power control:
- Add support for PCIe WAKE# interrupt when described via DT (Krishna
Chaitanya Chundru)
- For the TC9563 PCIe switch:
- Take a reference on the I2C adapter to avoid uninterruptible
hang when unloading an I2C module while in-use (Johan Hovold)
- Update DT binding and driver to restrict Tx Amplitude, DFE and
N_FTS to USP, DSP1 and DSP2 (Manivannan Sadhasivam)
- Power off only external-facing ports (DSP1, DSP2), leaving USP
and DSP3 (aka VDSP) powered up (Manivannan Sadhasivam)
- Move integrated MAC Endpoint out of the list of internal ports
and configure it separately (Manivannan Sadhasivam)
Virtualization:
- Add ACS quirk for Pericom PI7C9X2G608 switches (Tim Harvey)
- Fix a long-standing bug in the Intel PCH Root Port MPC ACS quirk
that didn't update the intended INTEL_MPC_REG_IRBNCE bit because it
used a 16-bit config write when a 32-bit write was intended
(Mohamad Raizudeen)
Procfs:
- Avoid spurious runtime PM wakeup on config space accesses that are
outside config space and fail before reaching PCI (Krzysztof
Wilczyński)
- Warn on user-space writes to kernel-exclusive config space regions,
as we already do for sysfs (Krzysztof Wilczyński)
- Check credentials of opener, not reader, for config space reads, as
we already do for sysfs (Krzysztof Wilczyński)
Sysfs:
- In pci_write_legacy_io(), avoid out-of-bounds reads from the user
buffer and fix incorrect ioport write data (1-byte writes on
little-endian powerpc, 2- and 4-byte writes on big-endian powerpc)
(Krzysztof Wilczyński)
- In pci_read_legacy_io(), fix incorrect ioport read data for 2- and
4-byte reads on big-endian powerpc (Krzysztof Wilczyński)
- Fix I/O port accessor argument order in Alpha pci_legacy_write()
(Krzysztof Wilczyński)
- Avoid spurious runtime PM wakeup on config space accesses that are
outside config space and fail before reaching PCI (Krzysztof
Wilczyński)
- Return -EINVAL, not -ENODEV, for mmap of I/O BAR that fails because
the arch doesn't support it, as we do for procfs (Krzysztof
Wilczyński)
- Check for LOCKDOWN_PCI_ACCESS for legacy_io and legacy_mem, as we
do for other config space accessors (Krzysztof Wilczyński)
Peer-to-peer DMA:
- Add Nvidia Vera Rubin to list of platforms that support P2PDMA
(Leon Romanovsky)
Endpoint framework:
- Check doorbell SUCCESS bit in pci_endpoint_test to avoid treating
some failures as successes (Niklas Cassel)
- Fail doorbell test when the trigger IRQ is missed (Niklas Cassel)
New native PCIe controller drivers:
- Add DT binding and driver for NVIDIA Tegra264 (Thierry Reding)
Native PCIe controllers:
- Use common wait time definitions for PCIe link monitoring instead
of defining driver-private duplicates (Thierry Reding)
Generic host bridge driver:
- Fix NULL pointer dereference that caused enumeration failures on
32-bit CAM systems (Steffen Persvold)
Amlogic Meson PCIe controller driver:
- Correct the PERST# GPIO state so it remains asserted until power
and REFCLK become stable to fix enumeration failure (Ronald
Claveau)
ASPEED PCIe controller driver:
- Switch to irq_domain_create_linear() so we can obsolete
irq_domain_add_linear() (Jiri Slaby)
Cadence PCIe controller driver:
- Add MODULE_DEVICE_TABLE to generate module aliases for OF-based
module autoloading (Pengpeng Hou)
- Add debugfs 'ltssm_status' file for LGA- and HPA-based Cadence
controllers (Hans Zhang)
- Support up to x4 (not x2) lanes for J200 (Takuma Fujiwara)
- Fix host/endpoint dependencies for cadence-plat driver to fix link
error when cadence-plat is built-in but the host or endpoint driver
is modular (Aksh Garg)
Freescale i.MX6 PCIe controller driver:
- Add imx6 intr/aer/pme interrupt lines for i.MX95 (Richard Zhu)
- Remove PERST# checking from pci_host_common_parse_port() so callers
can decide whether to fall back to legacy DT binding with PERST# in
the host bridge (Sherry Sun)
- Fix build issues when PCI_PWRCTRL_GENERIC or PCI_HOST_COMMON is a
module (Arnd Bergmann)
- Create pwrctrl devices only once by doing it from imx_pcie_probe()
instead of imx_pcie_host_init(), which is used during both probe
and resume (Sherry Sun)
- Use 'dw_pcie_rp->skip_pwrctrl_off' to avoid powering off devices
during suspend to preserve wakeup capability (Sherry Sun)
- Add runtime PM support for i.MX95 to allow dynamic power management
when the link is idle (Richard Zhu)
Intel VMD host bridge driver:
- Support device ID 0x28C1 and assume that BIOS has already
enumerated the hierarchy below VMD and stored bus range info for OS
to use (Nirmal Patel)
- Add support for VMCONFIG BUS_RESTRICT_CFG=3, which makes it
possible to enumerate downstream devices on Intel Arrow Lake-HX
systems and probably others (Ali Alaei)
- Pay attention to _OSC negotiation for VMD hierarchy only when
running on bare metal, not when running in a VM (Nirmal Patel)
- Add Nova Lake (NVL) and Dunlow (DNL) Device IDs (Szymon Durawa)
MediaTek PCIe controller driver:
- Add support for PCIe controller in EcoNet EN7528 and EN751221 SoCs
(Caleb James DeLisle)
MediaTek PCIe Gen3 controller driver:
- Add mediatek-gen3 'memory-region' for restricted DMA buffer
(Chen-Yu Tsai)
NVIDIA Tegra264 PCIe controller driver:
- Distinguish Tegra264 C0 PCIe controller for internal GPU from C1-C5
controllers so the unit address matches the first 'reg' entry
(Thierry Reding)
- Add Tegra264 Root Port stanzas to prepare for generic WAKE#
handling (Thierry Reding)
Qualcomm PCIe controller driver:
- Add IPQ9650 compatible with global interrupt (Kathiravan
Thirumoorthy)
- Add IPQ5210 compatible with IPQ9574 fallback (Varadarajan
Narayanan)
- Add DT binding and driver support for Hawi SoC (Matthew Leung)
- Skip PERST# GPIOs provided by downstream PCIe devices, which should
be handled by drivers of those devices (Manivannan Sadhasivam)
- Stop advertising Attention Button Present (no Qcom SoCs support
Attention Buttons) so pciehp can use Presence Detect Changed events
(Qiang Yu)
Renesas R-Car PCIe controller driver:
- Add rcar-gen4-pci-host optional 'msi-parent' for GIT ITS (Marek
Vasut)
- When MSI is enabled but iMSI-RX is not used, configure AXIINTC to
allow GIT ITS to handle MSI (Marek Vasut)
- Refactor GIC600 implementation to make it easier to add platforms
that only support 32-bit addressing (Marek Vasut)
- Add Renesas R-Car Gen4 S4/V4H/V4M to the list of GIC600
integrations that only support 32-bit addressing (Marek Vasut)
Renesas RZ/G3S PCIe controller driver:
- Add DT binding and driver support for RZ/V2H(P) SoC, which contains
two PCIe controllers, configured either as a single x4 link or two
independent x2 link controllers (Lad Prabhakar)
SpacemiT K1 PCIe controller driver:
- Add missing MODULE_DEVICE_TABLE() to generate module alias info for
OF-based module autoloading (Pengpeng Hou)
StarFive PCIe controller driver:
- Fix resource leaks on error paths in host_init() (Ali Tariq)
- Fix runtime PM handling and teardown ordering to avoid register
access while power or clocks are disabled (Ali Tariq)
- Check for runtime PM resume failure to avoid register access while
power or clocks are disabled (Ali Tariq)
Synopsys DesignWare PCIe controller driver:
- Add LECARC PMU IDs to the DWC RAS/DES VSEC list so it can take
advantage of the existing debugfs support for silicon debug, error
injection, and event counters (Brett Zhou)
- Factor pcie_valid_speed() and pci_bus_speed2lnkctl2() out of bwctrl
so they can be shared by the DWC core (Hans Zhang)
- Flush MSI writes from endpoint before unmapping the iATU, as we
already do for MSI-X writes (Niklas Cassel)
- Unmap MSI iATU window before mapping MSI-X window, to avoid a
subsequent MSI write using a disabled aperture and losing the
interrupt (Niklas Cassel)
- Change endpoint .pre_init() and .init() callbacks to return errors
and handle them (Marek Vasut)
UltraRISC PCIe controller driver:
- Add 'core', 'dbi', and 'aux' clocks to DT binding and manage them
in the driver (Jia Wang)
- Use module_platform_driver() since this may be built as a module,
though not removable because IRQs can't be safely disposed (Jia
Wang)
MicroSemi Switchtec management driver:
- Add Microchip PCI1008 device ID and include it in NTB DMA alias
quirk (Logan Gunthorpe)
Miscellaneous:
- Document how to write PCI Host Controller drivers (Manivannan
Sadhasivam)
- Fix typos in documentation (D'Orus Tsitera)
- Use %pe format specifier to print error pointers so we get symbolic
errname when available (Krzysztof Wilczyński)"
* tag 'pci-v7.3-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci: (124 commits)
PCI: vmd: Add Nova Lake (NVL) and Dunlow (DNL) Device IDs
PCI: tegra264: Add Tegra264 support
dt-bindings: PCI: tegra264: Switch to PCIe Root Port bindings
dt-bindings: PCI: tegra264: Strictly distinguish C0 from C1-C5
PCI/AER: Support Advisory Non-Fatal Errors
PCI: Fix 32-bit config write in Intel PCH Root Port MPC ACS quirk
PCI: dwc: Handle return value from endpoint .pre_init callback
PCI: dwc: Handle return value from endpoint .init callback
PCI: dwc: Add PCI ID for LECARC PCIe PMU
PCI/ASPM: Mask ASPM states based on Devicetree properties
PCI/ASPM: Disable/restore ASPM on every function for multi-function devices
Documentation: PCI: Document how to write PCI Host Controller drivers
PCI/ASPM: Use pcie_capability_clear_and_set_word() for ASPM disable/restore
PCI: Add support for PCIe WAKE# interrupt
PCI: Allow D3 for native hotplug-capable Root Ports on non-x86 platforms
dt-bindings: PCI: Correct white-space style
PCI/ASPM: Avoid L0s for Realtek RTS525A
PCI: ultrarisc: Use module_platform_driver()
PCI: ultrarisc: Get and enable DP1000 PCIe controller clocks
dt-bindings: PCI: ultrarisc: Add required DP1000 PCIe clocks
...
The interrupt-emulation virtual IRQ uses handle_level_irq(), but the IRQ
descriptor has not been marked with IRQ_LEVEL.
The interrupt emulation is somewhat unusual: the eDMA interrupt handlers
dispatch the virtual IRQ for every edma_int[] interrupt because software
cannot reliably tell an interrupt-emulation event from one caused solely
by DONE/ABORT status. If an interrupt arrives before the doorbell
handler is registered for the virtual IRQ, the IRQ core marks it
pending. When the IRQ is later started, check_irq_resend() treats it as
non-level and replays the pending interrupt, causing the newly
registered handler to run for a stale event.
Mark the virtual IRQ with IRQ_LEVEL so the stale pending state is
cleared without being replayed. This was observed in pci_endpoint_test
as two doorbell handler calls when the DMA-variant test ran before
DOORBELL_TEST.
Fixes: d9d5e1bdd1 ("dmaengine: dw-edma: Add virtual IRQ for interrupt-emulation doorbells")
Cc: stable@vger.kernel.org
Reported-by: Niklas Cassel <cassel@kernel.org>
Closes: https://lore.kernel.org/r/ampndLtU32ODmncX@ryzen
Tested-by: Niklas Cassel <cassel@kernel.org>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Link: https://patch.msgid.link/20260730160701.3550710-1-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The IAA device defaults only assigned engine 0 to group 0, leaving
engines 1 through max_engines-1 unassigned (group_id = -1). This means
that by default only a single engine processed descriptors, limiting
throughput to one engine's capacity.
Assign all available engines to group 0 so that the full hardware
parallelism is used out of the box without requiring manual
accel-config setup.
Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Link: https://patch.msgid.link/20260805-iaa-crypto-fixes-zswap-v2-1-55c02694f499@intel.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
bam_dma_probe() calls pm_runtime_use_autosuspend(), but
bam_dma_remove() does not call the matching
pm_runtime_dont_use_autosuspend() when removing the device.
If the autosuspend delay is set to a negative value while autosuspend
is enabled, the runtime PM core increments usage_count to prevent
runtime suspend. Without calling pm_runtime_dont_use_autosuspend()
during teardown, this reference is not dropped and usage_count remains
unbalanced.
Add the missing pm_runtime_dont_use_autosuspend() call before forcing
the device into runtime suspend during removal.
This issue was found by manual code inspection.
Fixes: 7d2545599f ("dmaengine: qcom-bam-dma: Add pm_runtime support")
Signed-off-by: Guangshuo Li <lgs201920130244@gmail.com>
Reviewed-by: Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
Link: https://patch.msgid.link/20260808110642.2770355-1-lgs201920130244@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The fsl edma events store a pointer to a struct fsl_edma_engine in the
ringbuffer and dereference it when a log entry is printed. At this time,
the pointer may no longer be valid.
Event injection can be used to trigger a crash:
$ cd /sys/kernel/tracing
$ echo 'value = 0' > events/fsl_edma/edma_writeb/inject
$ cat trace
The log output needs only edma->membase. Add a membase field at the end
of the event and use the new field for log output. Keep the existing
fields for backward compatibility.
Fixes: 11102d0c34 ("dmaengine: fsl-edma: add trace event support")
Cc: stable@vger.kernel.org
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Martin Kaiser <martin@kaiser.cx>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260718130024.341243-1-martin@kaiser.cx
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Pull dmaengine fixes from Vinod Koul:
- switchtec fix for register programming
- sun6i descriptor reclaim fix
- Intel idxd fixes for double free in error and setup failure
- Qualcomm bam dma command element fix
* tag 'dmaengine-fix-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/vkoul/dmaengine:
dmaengine: qcom: bam_dma: Fix command element mask field for BAM v1.6.0+
dmaengine: idxd: fix fdev setup failure cleanup in idxd_cdev_open()
dmaengine: idxd: fix double free of wq, engine, and group structs
dmaengine: sun6i-dma: Fix reclaim descriptors while terminating DMA
dmaengine: switchtec-dma: fix FIELD_GET misuse when programming SE threshold
The eDMA/HDMA transfers the driver issues carry a requester function
number in their TLPs, but nothing ever programs it: eDMA v0 leaves the
FUNC_NUM field of the channel control word zero and HDMA leaves the
per-channel func_num register at its reset value, so every transfer is
attributed to function 0. That is invisible in single-function setups,
but once the DMA block serves a non-zero endpoint function, its
requests must carry that function's number for the host to attribute
and translate them correctly.
Record the function number in the chip data (PCI_FUNC() of the probing
device for dw-edma-pcie) and program it per channel.
Endpoint-local chip instances keep func_no at 0, so transfers issued by
the endpoint-side driver remain PF0-attributed. Delegated channels are
programmed by the host-side dw-edma-pcie instance when it takes over the
channel, using that instance's PCI_FUNC().
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-15-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Skip data block BAR mapping and debug output when a channel has no data
block size. This lets future providers describe channels that only need
descriptor memory exposed.
No functional change intended for existing Synopsys EDDA and
AMD (Xilinx) MDB/CPM6 devices. Their static channel descriptions still
provide data block sizes where data block windows are used. A zero-sized
data block now means "not present" for future metadata providers.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-13-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add an optional physical address override to struct dw_edma_block and
use a helper to compute descriptor block addresses.
No functional change intended. Existing Synopsys EDDA and AMD (Xilinx)
MDB/CPM6 block descriptors leave the override unset, so the helper still
returns the same values as before.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-12-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add a match-data flag for devices whose DMA register block starts at an
offset inside the mapped BAR. Existing Synopsys EDDA and AMD (Xilinx)
MDB/CPM6 matches keep using the BAR mapping base directly.
The offsets handled here come from static, trusted match data. Later
metadata-derived offsets are validated when the endpoint DMA metadata
path parses them at that trust boundary.
No functional change intended.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-11-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dw_edma_pcie_probe() now obtains DMA layout data through device-specific
capability callbacks, not only from PCIe Vendor-Specific Extended
Capabilities. Rename the local data copy from vsec_data to dma_data
before adding endpoint DMA BAR metadata discovery, which does not rely
on VSEC.
No functional change intended.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-9-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Move device-specific capability parsing behind per-device match data.
The existing probe path mixes two decisions: which static template a PCI
ID uses, and which device-specific capability parser adjusts that
template. Split those decisions so device-specific discovery can be
added through match data instead of adding more vendor checks to
dw_edma_pcie_probe().
No functional change is intended for the existing Synopsys EDDA and
AMD (Xilinx) MDB/CPM6 matches. They still copy the same static template
data and run the same capability parsing logic before BAR mapping. The
AMD (Xilinx) MDB/CPM6 entries also keep using endpoint memory physical
addresses for descriptor windows through a new match-data flag.
Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-8-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The dw-edma-pcie driver copies static template data into a mutable
dw_edma_pcie_data instance before applying capability-derived updates.
Keep the derived non-LL mode in that copy as well, instead of only
tracking it in a local variable in dw_edma_pcie_probe().
This prepares for keeping capability parsing behind match data without a
separate non-LL output parameter.
No functional change intended.
Suggested-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-7-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
A DesignWare eDMA instance may represent only a subset of channels that
is also initialized by another OS instance, such as an endpoint-side OS.
Add a partial ownership flag for instances that must preserve
controller-wide state owned by that peer.
In partial ownership mode, dw-edma skips the initial core reset and uses
the limited quiesce path in probe() and remove() instead of the full
core-off path. The flag also makes the driver validate the ownership
granularity required by each register layout before registering
channels.
Probe fails if the limited quiesce cannot stop its resources; remove
reports the error after completing the remaining driver teardown.
Partial instances also skip interrupt-emulation doorbell allocation: the
emulated doorbell is a controller-level resource, and a partial owner
must not claim it on behalf of the whole block.
For EDMA_MF_EDMA_UNROLL and EDMA_MF_HDMA_COMPAT, the driver programs
per-direction registers, such as DMA_{WRITE,READ}_INT_MASK_OFF and
DMA_{WRITE,READ}_INT_CLEAR_OFF. These register layouts have at most
EDMA_MAX_{WR,RD}_CH channels per direction, so the capped hardware
channel count still represents the whole direction. A partial instance
can therefore expose write or read channels only if it owns every
channel in that direction; otherwise two OS instances could update the
same direction-wide registers without a shared locking protocol.
In contrast, HDMA native uses per-channel registers, so it can be owned
at channel granularity.
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-6-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Add core operations that quiesce only the resources represented by a
dw-edma instance, separate from the existing full controller off path.
For unrolled eDMA, quiesce masks the direction, disables ENGINE_EN,
waits for it to read back clear, then clears pending interrupt status.
If ENGINE_EN does not clear, return the timeout to the caller.
HDMA compatibility mode does not implement ENGINE_EN, so quiesce clears
CHi_PWR_EN for each represented channel instead. Both register maps
share interrupt control per direction, so quiescing one channel
quiesces the whole direction. Callers must own that direction and stop
the peer from programming it first.
HDMA native has per-channel registers and can quiesce the represented
channel directly.
No caller is added yet, so this is a no-functional-change preparation
for delegated channel reclaim and partial-owned remove paths.
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-4-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
DesignWare eDMA can signal completion locally through edma_int[] and
remotely through IMWr/MSI. When channels are delegated to a remote
frontend, the local endpoint side and the remote host side must not both
service the same DONE/ABORT status.
Add channel interrupt routing state and initialize it from the controller
instance configuration. Update the eDMA and HDMA native paths so
linked-list interrupt generation, HDMA non-linked-list interrupt enables,
and DONE/ABORT masking follow the selected mode. For HDMA native
non-linked-list channels, keep the local stop/abort enables set so status
is latched. In remote mode, also enable remote signaling and mask the local
interrupt pins.
Keep the existing dw-edma-pcie host-side instances in remote interrupt
routing mode so their IMWr/MSI completion model remains unchanged after
local routing becomes the zero value.
Note:
- The routing mode describes where a channel should report completion.
It does not by itself say whether this dw-edma instance owns the
interrupt status. A local instance must ignore remote-only channels,
and a remote instance must ignore local-only channels, even if such
interrupts are unexpectedly delivered. Otherwise the non-owner side
could steal the interrupt from the owner by clearing shared DONE/ABORT
status.
Cc: Devendra K Verma <devendra.verma@amd.com>
Suggested-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-3-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The HDMA linked-list and non-linked-list start paths both program the
stop/abort interrupt setup register using the same local/remote enable
policy. Only the interrupt-mask handling differs by transfer mode.
Factor the common setup into dw_hdma_v0_core_int_setup() before adding
per-channel interrupt routing support. No functional change intended.
Suggested-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260721062815.4117887-2-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
On some SoCs (e.g. R-Car S4) the endpoint-side eDMA raises a single
fixed SPI that is hardwired to CPU0 and covers every read and write
channel. Handling channel events directly in that hard IRQ context
serializes the completion processing of all channels on one CPU:
descriptor recycling and refill, client callbacks (the vchan tasklet
runs on the scheduling CPU) and the doorbell writes all funnel through
CPU0, while the handler additionally spins on each channel's vc.lock.
Especially under heavy multichannel load, this contention becomes a
performance bottleneck.
Keep the hard IRQ handler minimal: clear the status, dispatch channel
events, and defer per-channel processing to work items. A work item per
channel preserves ordering while allowing different channels to run in
parallel on any CPU.
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-10-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The v0 interrupt handler reads the interrupt status register twice per
invocation, once through the DONE accessor and once through the ABORT
accessor, although both fields live in the same 32-bit register. On
remote setups (dw-edma-pcie) each read is a non-posted round trip across
the PCIe link costing on the order of a microsecond, and with one
completion interrupt per element the duplicate adds up. As an example,
profiling the R-Car S4 remote path put the handler at ~7us per
invocation, dominated by such reads.
Read the register once and derive the DONE and ABORT views from the
snapshot. No abort is lost to this because the pass only clears status
bits it observed, so an abort raised after the snapshot keeps its status
and its own interrupt delivery brings it to the next pass. An abort on
an observed channel cannot race the clear either. Software can restart
the halted channel only after abort() runs, and abort() is called after
dw_edma_v0_core_clear_abort_int().
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-9-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dw_edma_pcie enables the PCI device with pcim_enable_device(), so IRQ
vectors allocated by pci_alloc_irq_vectors() are released by
pcim_msi_release() on device release. The driver should not call
pci_free_irq_vectors() manually.
Drop the redundant remove-time cleanup and rely on the managed PCI
device lifetime instead, as documented by commit 03e4905402 ("PCI/MSI:
Clarify pci_free_irq_vectors() usage for managed devices").
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-8-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
terminate_all() can finish immediately when the channel is unconfigured,
paused, idle, or already stopped in hardware. A pending PAUSE request can
survive these paths and block issue_pending() even after termination.
Clear the request whenever termination leaves the channel idle. A running
channel keeps its STOP request until the interrupt handler consumes it.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-7-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
pause() and resume() read and update channel state without holding vc.lock,
while the interrupt handlers update the same state under it. Take the same
lock around those state checks so that request, status, and configured stay
consistent.
For example, pause() can observe EDMA_ST_BUSY right before the interrupt
handler completes the final descriptor and moves the channel to
EDMA_ST_IDLE, and then record EDMA_REQ_PAUSE on an already idle channel. No
further interrupt will acknowledge the request, and since issue_pending()
requires EDMA_REQ_NONE, the channel is wedged for good: terminate_all()
leaves the stale request behind, so even reconfiguring the channel does not
recover it.
issue_pending() already runs under vc.lock, but it tests configured before
taking it. Move that test under the lock as well, so configured, request,
and status are evaluated as one channel-state snapshot.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-6-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
If PAUSE is requested while the final burst of a descriptor is in
flight, the DONE interrupt takes the PAUSE path without checking whether
the descriptor has been depleted. The depleted descriptor remains on the
issued list and the channel enters EDMA_ST_PAUSE.
On resume, dw_edma_start_transfer() can select that depleted descriptor
again even though no burst remains, leaving the channel in an invalid
busy state.
Check for descriptor completion before acknowledging PAUSE. If there is
no work to start on resume, leave the channel idle. Also ignore DONE
interrupts while the channel is paused so a stale or repeated interrupt
cannot change its state or start queued work.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Cc: stable@vger.kernel.org
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-5-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dw_edma_abort_interrupt() drops vc.lock before changing request and
status. issue_pending() can acquire the lock in that small window,
observe the old busy state, and skip starting queued descriptors. Then
the abort handler overwrites the channel status as idle, leaving the new
descriptors stranded for good.
Keep descriptor completion and the state transition in the same critical
section.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-4-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The DMA Engine client documentation says in the "Terminate APIs" section
of Documentation/driver-api/dmaengine/client.rst:
"No callback functions will be called for any incomplete transfers."
dw-edma instead calls vchan_cookie_complete() when a deferred STOP reaches
the interrupt handler. This schedules a callback for the active descriptor
and leaves other issued or submitted descriptors queued. A late callback
after dmaengine_terminate_sync() can dereference client state that has
already been freed, while leftover descriptors may later restart into
reused buffers or leak.
Move all issued and submitted descriptors to the terminated list whenever
termination completes. For a pending STOP, do this from both the DONE and
ABORT paths. Complete their cookies in order without scheduling callbacks.
A STOP can remain pending until the running transfer raises an
interrupt. Make device_synchronize() wait for such a pending STOP to
complete before releasing terminated descriptors. Reuse it from
free_chan_resources(), then release the remaining virt-dma resources.
Sleep instead of busy-polling while waiting, and warn if the existing
timeout expires.
Fixes: e63d79d1ff ("dmaengine: Add Synopsys eDMA IP core driver")
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Signed-off-by: Koichiro Den <den@valinux.co.jp>
Link: https://patch.msgid.link/20260717180639.2643243-3-den@valinux.co.jp
Signed-off-by: Vinod Koul <vkoul@kernel.org>
'struct dma_descriptor_metadata_ops' in not modified in these drivers.
Constifying these structures moves some data to a read-only section, so
increases overall security, especially when the structure holds some
function pointers.
On a x86_64, with allmodconfig, as an example:
Before:
======
text data bss dec hex filename
120635 21584 64 142283 22bcb drivers/dma/xilinx/xilinx_dma.o
After:
=====
text data bss dec hex filename
120699 21520 64 142283 22bcb drivers/dma/xilinx/xilinx_dma.o
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Reviewed-by: Sai Sree Kartheek Adivi <s-adivi@ti.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>
Link: https://patch.msgid.link/b0a22171f3ed68e156a2fa84383e99c23ec6b2ff.1784037977.git.christophe.jaillet@wanadoo.fr
Signed-off-by: Vinod Koul <vkoul@kernel.org>
dma_request_chan() assumes both @dev and @name are valid, but neither is
checked before use.
dev is dereferenced immediately via dev_fwnode(), which accesses
dev->of_node or dev->fwnode without checking for NULL. Likewise, if name is
NULL and the OF/ACPI lookup does not succeed, the legacy filter-map path
eventually passes it to strcmp(), resulting in a NULL pointer dereference.
These are caller bugs rather than normal lookup failures, so add a
WARN_ON() at function entry to catch invalid arguments early during
development instead of crashing later.
No functional change for valid callers.
Signed-off-by: bui duc phuc <phucduc.bui@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260716052758.23465-1-phucduc.bui@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Replace the open-coded platform_get_resource() plus devm_ioremap()
sequence with devm_platform_ioremap_resource(), which fetches the
resource, requests the region and maps it in one call. Switch the error
check to IS_ERR()/PTR_ERR() and drop the now-unused struct resource
pointer.
The raideng node has a single reg region (0x320000, 0x10000); the
job-queue/ring children are separate OF devices probed independently, so
the region reservation added by devm_ioremap_resource() is exclusive and
does not introduce overlap failures.
Assisted-by: opencode:hy3-free
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260716202949.677290-5-rosenp@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The fsl_re_ctrl and fsl_re_chan_cfg structures describe memory-mapped
RAID Engine registers accessed only via ioread32be()/iowrite32be(), yet
the pointers to them (re_regs in struct fsl_re_drv_private, and jrregs
in struct fsl_re_chan) were not __iomem-qualified, so sparse emitted
"different address spaces" warnings for every register access.
Store both MMIO bases as a plain void __iomem * and derive jrregs with
void __iomem * arithmetic from re_regs, rather than carrying typed
register struct pointers through the driver. Each function that touches
the registers introduces a local typed pointer (struct fsl_re_ctrl
__iomem *ctrl) and uses ->field, which is the idiomatic kernel pattern
and keeps the registers' __iomem qualification intact.
Reported-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/oe-kbuild-all/202008111749.yy85rFMD%25lkp@intel.com/
Assisted-by: opencode:hy3-free
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260716202949.677290-4-rosenp@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The final-frame bit is now passed as the "final" argument of
fill_cfd_frame() (as fsl_re_prep_dma_memcpy already did) and set in CPU
order before the single cpu_to_be32() store, replacing the previous
read-modify-write of the __be32 efrl32 field.
Assisted-by: opencode:hy3-free
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260716202949.677290-3-rosenp@gmail.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
As per 'Designware Cores PCI Express Controller Databook',
Section 7.1 - Overview, HDMA supports 64 Read and 64 Write
channels. Current controller driver supports up to 8 read and
write channels only. In order to utilize all the channels the
controller driver need to have the channel related structs
and variables as per the number of channels supported by IP.
Following changes are made to enable 64 Read / 64 Write
channel support:
o Defined HDMA specific macros to reflect the channel count.
o The count of ll_regions and dt_regions in dw_edma_chip and
dw_edma_pcie_data shall be in accordance to number of read
and write channels.
o In dw_edma_probe() configure the channels as per the channels
of the IP used.
o Changed mask types to u64 for higher channel counts.
Signed-off-by: Devendra K Verma <devendra.verma@amd.com>
Reviewed-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260713064854.4065262-1-devverma@amd.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The current descriptor layout is:
struct dw_edma_desc *desc
└─ chunk list
└─ burst[]
Creating a DMA descriptor requires at least two kzalloc() calls because
each chunk is allocated as a linked-list node. Since the number of bursts
is already known when the descriptor is created, this linked-list layer is
unnecessary.
Move the burst array directly into struct dw_edma_desc and remove the
struct dw_edma_chunk layer entirely.
Use start_burst and done_burst to track the current bursts, which current
are in the DMA link list.
Tested-by: Koichiro Den <den@valinux.co.jp>
Tested-By: Devendra Verma <devendra.verma@amd.com>
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260713-edma_ll-v7-10-6fb7498c901e@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
The current descriptor layout is:
struct dw_edma_desc *desc
└─ chunk list
└─ burst list
Creating a DMA descriptor requires at least three kzalloc() calls because
each burst is allocated as a linked-list node. Since the number of bursts
is already known when the descriptor is created, a linked list is not
necessary.
Allocate a burst array when creating each chunk to simplify the code and
eliminate one kzalloc() call.
Tested-by: Koichiro Den <den@valinux.co.jp>
Tested-By: Devendra Verma <devendra.verma@amd.com>
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260713-edma_ll-v7-9-6fb7498c901e@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Introduce four new callbacks to fill link list entries in preparation for
replacing dw_(edma|hdma)_v0_core_start().
Filling link list entries is expected to become more complex, and without
this abstraction both eDMA and HDMA paths would need to duplicate the same
logic. Add fill-entry callbacks so the code can be shared cleanly between
eDMA and HDMA implementations.
Tested-by: Koichiro Den <den@valinux.co.jp>
Tested-By: Devendra Verma <devendra.verma@amd.com>
Signed-off-by: Frank Li <Frank.Li@nxp.com>
Link: https://patch.msgid.link/20260713-edma_ll-v7-6-6fb7498c901e@nxp.com
Signed-off-by: Vinod Koul <vkoul@kernel.org>
Kees Cook