This is another run of the Coccinelle script for converting kmalloc()
family of allocations to kmalloc_obj() via the existing rules in
scripts/coccinelle/api/kmalloc_objs.cocci
This catches both the set of kmalloc() uses added since the first
kmalloc_obj() conversions in v7.0 and adds a large group missed in the
first pass due to Coccinelle not interacting well with the cleanup.h
scoped_...() family of macros[1]. I worked around this with spatch's
"--macro-file" argument to a file with all the scoped_...() macros mapped
to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control
flow indicator I could find.
Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc,
riscv, and s390 with no new warnings.
Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1]
Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2]
Signed-off-by: Kees Cook <kees+treewide@kernel.org>
Pull fuse updates from Miklos Szeredi:
- Improve performance of the io-uring transport by introducing buffer
pools and zero-copy (Joanne)
- Fix lots of bugs (Baokun Li)
- Fix io-uring initialization issues (Joanne, Bernd)
- More prep work for large folios (Joanne)
- Don't limit buffered read to 128k (Jim Harris)
- Fix zeroing of page end (dirtied with mmap) on file size extension
(Jimmy Zuber)
- Improve performance in certain cases with wake_up_sync() when queuing
request (Xuewen Yan)
- Misc fixes and cleanups (Xuewen Yan)
* tag 'fuse-update-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse: (35 commits)
fuse: zero the partial EOF page when extending a file
io_uring: Add missing include for ITER_SOURCE and ITER_DEST
fuse: Fix the condition to enable over-io-uring
fuse: invalidate the correct range after O_APPEND direct write
selftests/fuse: test post-EOF page zeroing when a file is extended
fuse: wake one waiter per freed slot when raising max_background
fuse: use min_not_zero() in fuse_init_server_timeout()
fuse: copy request headers via a stack buffer for io-uring
fuse: give wakeup hints to the scheduler for synchronous requests
fuse: check for NULL root inode in fuse_fill_super_submount
fuse: reject a duplicate fd= mount option
cuse: wait for pending RCU callbacks on module exit
fuse: fix invalidate lock leak on open O_TRUNC DAX failure
fuse: fix invalidate lock leak on setattr writeback failure
fuse: wait for FR_FINISHED on abort_on_kill to prevent use-after-free
fuse: make dentry_tree_work static
docs: fuse: document io-uring buffer pool and zero-copy uapi
fuse: add zero-copy over io-uring
fuse: support registered buffer pools in io-uring
fuse: add io-uring buffer pools
...
Extending a fuse file past a non-page-aligned EOF does not zero the tail of
the old last page. When that page is cached and has been mmap-dirtied beyond
the old EOF, the now in-bounds tail is served to later reads as stale data
rather than zeros, which violates POSIX file-extension semantics.
Some file systems get this zeroing automatically at writeback time
(block_write_full_folio() / iomap_writeback_handle_eof() zero the tail of the
folio straddling i_size). A non-writeback caching fuse file system uses neither
path, so it has to zero the tail itself from the size-extending paths, like
XFS (xfs_file_write_zero_eof()) and ext4 (ext4_block_zero_eof()) do.
Call truncate_pagecache_range() over the newly-exposed range up front from the
three paths that extend a file, before the new size is published:
- a buffered write whose position is past the old EOF (fuse_perform_write());
- a size-extending setattr/truncate (fuse_do_setattr());
- a size-extending fallocate (fuse_file_fallocate()).
This unmaps the stale mappings and zeroes the partial tail of the old EOF
folio, so a later read returns zeros. Truncating [old EOF, write start) before
a buffered write keeps the dropped range disjoint from the written data, so a
write that lands inside the old EOF folio is preserved.
writeback_cache connections are unaffected, as their writes go through
iomap_file_buffered_write(), which zeroes post-EOF folios. The bug is
observable on a non-writeback_cache server that returns FOPEN_KEEP_CACHE on
writable files (without FOPEN_DIRECT_IO), and is caught by the new
write_extend_eof fuse selftest.
Signed-off-by: Jimmy Zuber <jamz@amazon.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
The existing condition in fuse_uring_cmd() is there only to avoid
disabling io-uring for connections that already run with it, missing
was a condition to refuse any IORING_OP_URING_CMD if the
connection/channel didn't get enabled because of missing FUSE_INIT
reply flag FUSE_OVER_IO_URING. Without the reply flag the barrier in
fuse_uring_ready() doesn't work and IO could already be going on and
cause deadlock states (at a minimum one between fch->bg_lock and
queue->lock).
The change itself is trivial, but brings behavior change,
FUSE_OVER_IO_URING has to be set in the FUSE_INIT_REPLY by fuse servers
to accept any IORING_OP_URING_CMD. Libfuse does that and the only
non-libfuse implementation I found (fractal-fuse) also does it.
Qemu patches for fuse-io-uring are not merged yet, as far as I know.
Moved up is the smp_load_acquire(&fch->initialized) check, as a
fuse-server implementation might try to setup io-uring before FUSE_INIT
is processed and might have gotten -EOPNOTSUPP instead of -EAGAIN.
Also fixed is a stale comment that explains the handling of the
FUSE_OVER_IO_URING flag in early RFC versions.
If there should be a report from any library or application we
probably need to revert this commit.
Fixes: 3393ff964e ("fuse: block request allocation until io-uring init is complete")
Signed-off-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_direct_write_iter() captures pos before generic_write_checks(),
which moves ki_pos to EOF for O_APPEND writes:
fuse_direct_write_iter()
{
pos = iocb->ki_pos; /* 0 (user-supplied) */
generic_write_checks(); /* ki_pos -> EOF */
fuse_direct_io(); /* writes at EOF, correct */
invalidate(pos, pos + res); /* [0, res) -- wrong */
}
The post-write invalidation targets a stale range instead of the
actual written range at EOF.
This can cause data inconsistency when the file size is not
page-aligned. The tail page straddling EOF has a valid portion
before EOF that concurrent readers can fault back in during the
DIO write window:
Tail page (file size X not page-aligned):
page_start X (EOF) page_end
|--- valid data ----|-- stale --|
CPU0 (O_APPEND DIO writer) CPU1 (buffered reader)
-------------------------- ----------------------
invalidate [X, X+len)
tail page evicted
FUSE_WRITE in flight ...
read [page_start, X)
tail page re-faulted
[X, page_end) = stale
FUSE_WRITE completes
i_size = X + len
invalidate [0, len) <- WRONG
tail page still cached
read [X, X+len)
hits stale tail page
returns old data
Fix by reading pos back from iocb->ki_pos after generic_write_checks(),
as generic_file_direct_write() does.
Also fix a typo in the comment ("may have" -> "may have competed").
Fixes: 2b0408d028 ("fuse: invalidate page cache after DIO and async DIO writes")
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_get_req() parks background allocations on fch->blocked_waitq via
wait_event_state_exclusive(), so each wakeup releases exactly one
waiter. fuse_chan_max_background_set() clears fch->blocked when the
new limit exceeds num_background, but the accompanying wake_up()
releases a single waiter regardless of how many slots just became
available. Raising max_background from 10 to 100 therefore admits one
request instead of ninety.
The remaining waiters are not permanently stranded — the "else if
(!fch->blocked)" branch in fuse_request_end() wakes one more per
completion — but that only helps while requests keep completing.
Consider a fixed pool of threads doing readahead or async direct I/O
with the quota exhausted: every thread is either in flight or parked,
and each completion wakes one waiter while freeing one slot, a net
change of zero. num_background oscillates around the old limit and
the added quota is never taken up.
Waking one waiter per freed slot also preserves submission order:
once fch->blocked is clear, new callers of fuse_get_req() skip the
waitqueue entirely, overtaking waiters that parked before the limit
was raised.
Use wake_up_nr() with the number of slots that just became available.
Since the wakeup is guarded by !fch->blocked, num_background is
strictly below max_background, so the count is at least 1 and never
degenerates into wake_up_all().
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
Reviewed-By: Horst Birthelmer <hbirthelmer@ddn.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_init_server_timeout() limits timeout to fuse_max_req_timeout with
the same logic as min_not_zero(), and returns early exactly when the
computed timeout would be zero.
So use min_not_zero() instead and return when the computed timeout is
zero.
No functional change.
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Reviewed-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
The fuse-io-uring transport copies req->in.h out to the ring in
fuse_uring_copy_to_ring() and req->out.h back in fuse_uring_commit().
Both headers live inside the fuse_request slab object, whose cache
(fuse_req_cachep) is created without a usercopy whitelist, so copying
them directly to/from userspace trips CONFIG_HARDENED_USERCOPY and
panics:
usercopy: Kernel memory exposure attempt detected from SLUB object
'fuse_request' (offset 56, size 40)!
kernel BUG at mm/usercopy.c:102!
Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI
RIP: 0010:usercopy_abort (mm/usercopy.c:90)
Call Trace:
__check_heap_object (mm/slub.c:8268)
__check_object_size (mm/usercopy.c:197 mm/usercopy.c:258 mm/usercopy.c:223)
copy_header_to_ring (fs/fuse/dev_uring.c:618)
fuse_uring_prepare_send (fs/fuse/dev_uring.c:776 fs/fuse/dev_uring.c:785)
fuse_uring_send_in_task (fs/fuse/dev_uring.c:1306)
tctx_task_work_run (io_uring/tw.c:96)
task_work_run (kernel/task_work.c:233)
io_run_task_work (io_uring/tw.h:84)
io_cqring_wait (io_uring/wait.c:278)
__do_sys_io_uring_enter (io_uring/io_uring.c:2685)
entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121)
Bounce both headers through an on-stack copy so the usercopy touches
stack memory, not the slab object.
Fixes: c090c8abae ("fuse: Add io-uring sqe commit and fetch support")
Cc: stable@vger.kernel.org
Reported-by: Weiming Shi <bestswngs@gmail.com>
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Xiang Mei <xmei5@asu.edu>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Reviewed-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
When a synchronous FUSE request is sent, the in-kernel client queues it
on fiq->pending and wakes the userspace daemon sleeping in
fuse_dev_do_read()->wait_event_interruptible_exclusive(fiq->waitq, ...).
The client then blocks in request_wait_answer() waiting for the reply,
so the waker is about to go to sleep: this is exactly the pattern that
WF_SYNC is meant to optimise.
As Peter Zijlstra explained in the earlier discussion [1],
WF_SYNC is a hint that the waker is about to sleep and the waker and
wakee share data, so stacking the woken thread on the current CPU
is beneficial for cache locality instead of searching for an idle one.
Add a wake_up_sync() wrapper for task on the synchronous request path.
Performance:
On an Android big.LITTLE device where the FUSE daemon (MediaProvider)
runs as a background service on the little cores while foreground
applications run on the big cores, the synchronous wakeup hint lets
the scheduler pull the daemon thread onto the big core that is
issuing the request, where the request data is cache-hot. Measured
by qixiaoyu [2] on a 2000-picture zip decompression to /sdcard:
------------------------------------------
| Default | patched | Improvement |
------------------------------------------
| 13.0 s | 7.0 s | 46% |
------------------------------------------
Server thread wall duration: 3583 ms -> 1276 ms
Server runs on big core: 5% -> 79%
The original 4K-file copy/compress/decompress workload [1] on the
same kind of device showed a ~28% improvement (13.8s -> 9.9s).
Note: Miklos reported [2] that on his test box he could not observe
an actual migration from wake_up_interruptible_sync(); the benefit
appears to be most visible on asymmetric topologies (big.LITTLE,
where the daemon normally lives on a little core) and on workloads
dominated by small synchronous requests. No regression was
reported on the symmetric- SMP test setups tried.
The earlier version of this change [1] added a `bool sync` argument
to all three hooks of `struct fuse_iqueue_ops` and threaded it
through virtio_fs as well. Miklos questioned the interface churn,
and the patch has been stalled since.
Re-work it so the exported interface is left alone. The hint is
carried in a new FR_SYNC_WAKEUP bit of the existing `fuse_req->flags`
bitfield (an `unsigned long`, so no layout change):
- __fuse_request_send() sets the flag before fuse_send_one().
- fuse_dev_queue_req() consumes it with test_and_clear_bit() and
forwards the result to fuse_dev_wake_and_unlock(), which then
picks wake_up_sync() or wake_up().
- The forget, interrupt and resend paths pass `false` explicitly,
preserving their original wake_up() behaviour.
Only /dev/fuse ever wakes fiq->waitq; virtio_fs and fuse_uring
dispatch through their own transport and never call wake_up(), so
threading `sync` through their ops would just add an unused argument.
test_and_clear_bit() makes the flag a one-shot hint that cannot leak
into a future requeue, and no extra cleanup is needed in
fuse_request_end()/fuse_put_request().
[1] https://lore.kernel.org/lkml/1638780405-38026-1-git-send-email-quic_pragalla@quicinc.com/
[2] https://lore.kernel.org/lkml/20221222093407.GA1141@mi-HP-ProDesk-680-G4-MT/
This work is based on "Pradeep P V K <quic_pragalla@quicinc.com>" and
"Pavankumar Kondeti <quic_pkondeti@quicinc.com>"
Assisted-by: TRAE:GLM-5.2
Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_iget() can return NULL when its inode allocation fails, but
fuse_fill_super_submount() passed the result straight to get_fuse_inode()
and decremented fi->nlookup without checking it:
root = fuse_iget(sb, parent_fi->nodeid, ...);
fi = get_fuse_inode(root);
fi->nlookup--;
Inside fuse_iget() the inode allocation can fail and return NULL. The
submount root takes the iget5_locked() path, whose alloc_inode() can fail
under memory pressure (the auto-submount branch can fail the same way in
new_inode() or fuse_alloc_submount_lookup()):
inode = iget5_locked(sb, nodeid, fuse_inode_eq, fuse_inode_set,
&nodeid);
if (!inode)
return NULL;
A NULL root makes get_fuse_inode() a container_of() on NULL and the
nlookup decrement a write to a bogus address, oopsing the mount. With
CONFIG_KASAN the following null pointer dereference is reported when the
root inode allocation of an auto-submount fails (e.g. under memory
pressure):
==================================================================
BUG: KASAN: null-ptr-deref in fuse_get_tree_submount+0x656/0x8b0
Read of size 8 at addr 00000000000002b0 by task ls/942
CPU: 0 PID: 942 Comm: ls Tainted: G W 6.6 #15
Call Trace:
<TASK>
fuse_get_tree_submount+0x656/0x8b0
vfs_get_tree+0x48/0x140
fc_mount+0x13/0x50
fuse_dentry_automount+0x7a/0xb0
__traverse_mounts+0xca/0x330
step_into+0x339/0xac0
path_lookupat+0xc5/0x2f0
filename_lookup+0x163/0x2a0
vfs_statx+0xd5/0x200
do_statx+0x83/0xd0
__x64_sys_statx+0xa0/0xc0
do_syscall_64+0x37/0x90
entry_SYSCALL_64_after_hwframe+0x78/0xe2
</TASK>
==================================================================
Return -ENOMEM instead; the caller tears down the partially built
superblock on error, matching the other error returns in this
function.
Fixes: 1866d779d5 ("fuse: Allow fuse_fill_super_common() for submounts")
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_opt_fd() stored the fuse device in ctx->fud and bumped its refcount
unconditionally:
ctx->fud = fuse_dev_grab(file);
If fd= is given twice (two fsconfig FSCONFIG_SET_FD calls), the second
call overwrites ctx->fud and grabs the new device, while the reference
taken on the first device is never released - a permanent refcount leak
that pins the first fuse_dev until reboot.
Reject a second fd= outright. ctx is zeroed on allocation, so a non-NULL
ctx->fud reliably means the option was already processed.
Fixes: d42eb23b2e ("fuse: don't require /dev/fuse fd to be kept open during mount")
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Since commit 053fc4f755 ("fuse: fix UAF in rcu pathwalks"),
fuse_conn_put() frees the fuse_conn through call_rcu() rather than
synchronously. For cuse, fc->release is cuse_fc_release(), which
lives in the cuse module. If the module is removed before the RCU
grace period ends, the callback jumps into freed module memory:
userspace / module unload | RCU softirq
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
close(/dev/cuse) |
cuse_channel_release() |
fuse_dev_release() |
fuse_conn_put(fch->conn) |
call_rcu(delayed_release) ------+---> callback queued
|
rmmod cuse |
cuse_exit() |
cuse_channel_destroy() |
... |
return |
|
<module text freed> |
| rcu_do_batch()
| delayed_release()
| fc->release()
| -> cuse_fc_release()
| ^^^ freed text!
The freed module text is unmapped by vfree(), so the jump into the
stale callback triggers a page-fault Oops. If the virtual address
is subsequently reused, the callback could execute unrelated code
(undefined behaviour).
Fix this by calling rcu_barrier() in cuse_exit() so that any pending
fuse_conn release callback completes before the module is removed.
Fixes: 053fc4f755 ("fuse: fix UAF in rcu pathwalks")
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_open() takes filemap_invalidate_lock() for a DAX truncate
(dax_truncate = true) and releases it before the out_inode_unlock
label. But when fuse_dax_break_layouts() fails, the goto
out_inode_unlock skips the unlock and leaks the rwsem, so any later
fault or truncate on the file stalls on the stale lock.
fuse_dax_break_layouts() can fail with -ERESTARTSYS when a signal
interrupts the wait for busy DAX pages to drain:
open("file", O_RDWR | O_TRUNC)
└─ fuse_open()
├─ filemap_invalidate_lock() # dax_truncate
└─ fuse_dax_break_layouts()
└─ dax_break_layout()
└─ wait_page_idle() # TASK_INTERRUPTIBLE
└─ fuse_wait_dax_page() # unlock, schedule, re-lock
└─ signal → -ERESTARTSYS
goto out_inode_unlock # <- lock leaked
Fix this by moving filemap_invalidate_unlock() below the label so
that all error paths release the lock, and rename the label to
out_unlock as it now covers more than just the inode lock.
Fixes: 2fdbb8dd01 ("fuse: fix deadlock between atomic O_TRUNC and page invalidation")
Cc: stable@vger.kernel.org # v6.0+
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_do_setattr() takes filemap_invalidate_lock() for a DAX truncate
(fault_blocked = true) and releases it at the out:/error: labels. But
when a writeback flush is also needed, a write_inode_now() failure
returns directly and leaks the lock, so any later fault or truncate on
the file stalls on the stale rwsem.
For example, truncate(2) on a setuid file reaches fuse_do_setattr()
with both ATTR_SIZE and ATTR_MODE set:
truncate(2)
└─ do_truncate()
├─ dentry_needs_remove_privs() # S_ISUID
└─ notify_change() # KILL_SUID -> ATTR_MODE
└─ fuse_setattr() # no killpriv:
│ # ia_valid |= ATTR_MODE
└─ fuse_do_setattr()
├─ filemap_invalidate_lock() # IS_DAX && is_truncate
└─ write_inode_now() # is_wb && ATTR_MODE
└─ if (err) # e.g. daemon -> -EIO
return err # <- lock leaked
Fix this by adding an unlock label that releases the lock before
returning the error, and use it for the fuse_dax_break_layouts()
failure path as well.
Fixes: 6ae330cad6 ("virtiofs: serialize truncate/punch_hole and dax fault path")
Cc: stable@vger.kernel.org # v5.10+
Signed-off-by: Baokun Li <libaokun@linux.alibaba.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Pull misc vfs updates from Christian Brauner:
"Bigger cleanups:
- The lockref dead-count handling is tidied up.
The open-coded check for a count below zero as the dead marker
relies on information the caller should not have.
- make put_mnt_ns() leave mounts connected. Destroying a mount
namespace disconnected its mounts from their mount points. So a
file descriptor still open on the parent of a mount point could be
used to peek under it.
Locked mounts were already kept connected to prevent exactly that.
But a mount is only locked when its tree is copied across a user
namespace boundary. So a mount namespace set up by a privileged
component had no locked mounts and its mounts were disconnected.
Passing UMOUNT_CONNECTED keeps every mount connected and prevents
that bug.
- vfs_prepare_mode() passes S_IFDIR for directories. I meant to fix
that ago but didn't get to it. So now someone finally did it.
This kills the exception where the mode could be 0 when a directory
was created whereas every other creation operation passed it
explicitly already.
- move long delayed work for ufs, jffs2, hfsplus, hfs and affs from
the per-cpu system_long_wq to the new unbound system_dfl_long_wq.
None of that work relies on per-cpu state and the work item is
enqueued with queue_delayed_work() whose timer is global anyway. So
it may as well benefit from scheduler task placement.
Smaller fixes and cleanups:
- unlock_buffer() and journal_end_buffer_io_sync() use
clear_and_wake_up_bit()
- the pipe page pools are unified into a single per-pipe pool and the
extra wake_up(rd_wait) is limited to EPOLLET consumers
- eventpoll now computes its timer slack lazily in ep_poll()
- shrink_dcache_for_umount() keeps making progress on busy roots
- excess xarray nodes are freed in clear_inode()
- romfs detects hard link cycles
- the user path of nested backing files is fixed
- pidfd holds exec_update_lock around the namespace ioctl
- non-memcg-aware nr_cached_objects is skipped during memcg slab
shrink
- iomap_write_iter() always returns status
- mangle_path() is renamed to seq_mangle_path()
- inode timestamp accessors are annotated
- new regression test for pipe->poll_usage.
- a few documentation, kernel-doc and selftest fixes"
* tag 'vfs-7.3-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (67 commits)
selftests/namespaces: Fix racy pipe handshake in timens and pidns_separate
selftests/epoll: add a regression test for pipe->poll_usage
pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers
pidfd: hold exec_update_lock around namespace ioctl
fs: fix user path of nested backing files
fs: remove stale inode_insert5() kernel-doc parameter
fs: fix switch/case indentation in sysfs() syscall
fs: document semantics of kstat::{uid,gid} fields
dcache: keep shrink_dcache_for_umount() making progress on busy roots
seq_file: rename mangle_path to seq_mangle_path
nstree: add/fix struct ns_id_req kernel-doc member fields
dcache: use lockref routines for dead count checks
lockref: tidy up dead count handling
initramfs: fix typo in reserve_initrd_mem comment
fs/pipe: unify the page pools into a single per-pipe pool
fs: annotate inode timestamp accessors
eventpoll: compute timer slack lazily in ep_poll()
selftests/filesystems: add mntns cleanup test
put_mnt_ns(): leave mounts connected
affs: Move long delayed work on system_dfl_long_wq
...
Pull vfs lookup updates from Christian Brauner:
"This refactors lookup_open() and adds vfs_lookup_open() for nfsd.
mnt_want_write() and parent locking are moved into lookup_open()
itself.
audit_inode_child() is also now called in lookup_open() on failure.
That is the calling convention in vfs_create() and vfs_mkdir(), but
lookup_open() made no such call when atomic_open() should have created
a file and did not. And neither did the regular ->create() path fwiw.
This also contains work to remove the unneeded excl argument from the
->create() inode op"
* tag 'vfs-7.3-rc1.lookup' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs:
fs/namei.c: fix coding style in atomic_open() and lookup_open()
fs/namei.c: fix kerneldoc of atomic_open() and vfs_lookup_open()
fs/namei.c: update stale comments in lookup_open()
Remove excl arg to ->create inode_operation
fs/namei.c: update kerneldoc of atomic_open()
vfs: call audit_inode_child() in lookup_open() on failure
vfs: move create error && negative dentry case in lookup_open() up
VFS: add vfs_lookup_open() for nfsd
VFS: move delegated_inode retry loop into lookup_open()
VFS: move mnt_want_write() and locking into lookup_open()
Pull iomap updates from Christian Brauner:
"The bulk of this is the conversion of iomap to a single ->iomap_next()
callback and thus finishing the move to an iterator model.
Every iomap operation drove its iteration through a struct iomap_ops
holding ->iomap_begin() and ->iomap_end(). iomap_iter() only ever sees
those as pointers. That means every step of every iteration is an
indirect call.
This collapses both into one ->iomap_next() callback that finishes the
previous mapping and produces the next one. This lets callers inline
the iteration loop and pass its ->iomap_next() as a compile time
constant. That means the compiler can turn it into a direct and hence
inlineable call.
This also allows future callers to express custom logic to drive the
iteration forward better. xfs, btrfs, ext4, ext2, erofs, f2fs, gfs2,
hpfs, fuse, exfat, zonefs, ntfs, ntfs3 and the block device mapping
are all converted. No functional changes are intended.
This also adds a simple direct I/O path for small reads. On Gen5 NVMe
the __iomap_dio_rw() dominates 4K random reads. The same single-core
io_uring poll mode workload reaches ~3.2M IOPS against the raw block
device but only ~1.92M through ext4 or XFS.
__iomap_dio_rw(), iomap_iter(), iomap_dio_bio_iter() and kfree() were
at the top of the profile. The new path is very lightweight if no
special behavior is requested. The bio comes from a dedicated bioset
and laid out so the whole request is a single cacheline aligned
allocation. Completion runs inline.
That takes ext4 from 1.92M to 2.19M IOPS in the original workload. fio
shows around:
- 4% at libaio queue depths of 64 and up
- around 5% for io_uring
- up to 10% for io_uring poll mode at depth 256
on both ext4 and xfs.
A few other patches:
- iomap_folio_mark_uptodate() lets a filesystem that writes into the
page cache outside the iomap read and write paths keep iomap's
internal uptodate bitmap in sync, which fuse needs for
server-pushed notify stores before it can enable large folios;
- two fixes for iomap_bio_read_folio_range_sync(): a potential crash
when device integrity behavior is changed and a missing
bio_uninit().
- a folio batch release fix on iomap callback failures
- FGP_NOFS is dropped from iomap_get_folio()
- documentation fix"
* tag 'vfs-7.3-rc1.iomap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (29 commits)
iomap: iomap_bio_read_folio_range_sync is missing a call to bio_uninit
iomap: don't free integrity payload that doesn't exist
docs: fix grammatical error in iomap docs
exfat: convert iomap ops to ->iomap_next()
fuse: convert iomap ops to ->iomap_next()
hpfs: convert iomap ops to ->iomap_next()
gfs2: convert iomap ops to ->iomap_next()
f2fs: convert iomap ops to ->iomap_next()
block: convert iomap ops to ->iomap_next()
ext2: convert iomap ops to ->iomap_next()
zonefs: convert iomap ops to ->iomap_next()
erofs: convert iomap ops to ->iomap_next()
ext4: convert iomap ops to ->iomap_next()
ntfs: convert iomap ops to ->iomap_next()
ntfs3: convert iomap ops to ->iomap_next()
btrfs: convert iomap ops to ->iomap_next()
xfs: convert iomap ops to ->iomap_next()
iomap: add ->iomap_next()
iomap: use GFP_NOWAIT when application for iomap_dio_simple allocations
iomap: decouple simple direct I/O reads from iomap_dio_rw
...
The abort_on_kill path in request_wait_answer() calls fuse_abort_conn()
and returns without waiting for FR_FINISHED. If fuse_dev_do_write() is
concurrently processing the same request (FR_LOCKED set), the caller
frees req->args while it is still being accessed, causing a
use-after-free.
Fix this by jumping to the existing wait_event(FR_FINISHED) instead of
returning early. The wait will not hang because fuse_abort_conn()
ensures all requests are ended.
Reported-by: syzbot+d6540a3fa1626e11360d@syzkaller.appspotmail.com
Fixes: 204aa22a68 ("fuse: abort on fatal signal during sync init")
Cc: stable@vger.kernel.org
Assisted-by: Claude:claude-opus-4-6
Signed-off-by: Rochan Avlur <rochan.avlur@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
The dentry_tree_work is not exported, so make it static
to remove the followign sparse warning:
fs/fuse/dir.c:37:21: warning: symbol 'dentry_tree_work' was not declared. Should it be static?
Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Implement zero-copy in fuse io-uring to eliminate memory copies between
the application, kernel, and server for read/write operations. The
server can directly access client pages or page cache folios without
copying data through an intermediary buffer. When a fuse request arrives,
the kernel registers the relevant pages into a sparse slot in the
server's io_uring registered buffer table. The server can then operate
on these pages directly using io-uring fixed buffer operations (eg
read_fixed/write_fixed) and the kernel unregisters these pages when the
request completes. Non-page-backed args (eg op out headers) will go
through the payload buffer as normal. The server can specify which open
files should have their reads/writes go through zero-copy, by setting
the FOPEN_IO_URING_ZERO_COPY flag when servicing opens.
This requires CAP_SYS_ADMIN and bufpools. This is gated behind
CAP_SYS_ADMIN because zero-copy allows the server direct access to the
client's underlying pages, rather than operating on an intermediary
buffer that the contents of the client's pages were copied into or on
page cache folios.
The request flow for the zero-copy direct-io write path (client writes
data, server reads it) is as follows:
=======================================================================
| Kernel | FUSE server
| |
| "write(fd, buf, 1MB)" |
| |
| >sys_write() |
| >fuse_file_write_iter() |
| >fuse_send_one() |
| [req->args->in_pages = true] |
| [folios hold client write data] |
| |
| >fuse_uring_copy_to_ring() |
| >copy_header_to_ring(IN_OUT) |
| [memcpy fuse_in_header] |
| >copy_header_to_ring(OP) |
| [memcpy write_in header] |
| |
| >fuse_uring_args_to_ring() |
| >setup_fuse_copy_state() |
| [skip_folio_copy = true] |
| |
| >fuse_uring_set_up_zero_copy() |
| [folio_get for each client folio] |
| [build bio_vec array from folios] |
| >io_buffer_register_bvec() |
| [register pages at
ent->zero_copy_index] |
| [ent->zero_copied = true] |
| |
| >fuse_copy_args() |
| [skip_folio_copy => return 0 |
| for page arg, skip data copy] |
| |
| >copy_header_to_ring(RING_ENT) |
| [memcpy ent_in_out] |
| >io_uring_cmd_done() |
| |
| | [CQE received]
| |
| | [issue io_uring READ at
| | ent->zero_copy_index]
| | [reads directly from
| |client's pages (ZERO_COPY)]
| |
| | [write data to backing
| | store]
| | [submit COMMIT AND FETCH]
| |
| >fuse_uring_commit_fetch() |
| >fuse_uring_commit() |
| >fuse_uring_copy_from_ring() |
| >fuse_uring_req_end() |
| >io_buffer_unregister(ent->zero_copy_index) |
| [unregister pages from index] |
| >fuse_zero_copy_release() |
| [folio_put for each folio] |
| [ent->zero_copied = false] |
| >fuse_request_end() |
| [wake up client] |
The zero-copy read path is analogous.
Some requests may have both page-backed args and non-page-backed args.
For these requests, the page-backed args are zero-copied while the
non-page-backed args are copied to the buffer selected from the buffer
pool:
zero-copy: pages registered via io_buffer_register_bvec()
non-page-backed: copied to payload buffer via fuse_copy_args()
For a request whose payload is zero-copied, the
registration/unregistration path looks like:
register: fuse_uring_set_up_zero_copy()
folio_get() for each folio
io_buffer_register_bvec(ent->zero_copy_index)
unregister: fuse_uring_req_end()
io_buffer_unregister(ent->zero_copy_index)
-> fuse_zero_copy_release() callback
folio_put() for each folio
Please note that on abort for in-flight zero-copied requests that have
been sent to userspace, the registered bvec slot remains occupied and
its folios remain pinned until the io-uring ring is destroyed, at which
point io-uring unregisters all buffers and the fuse_zero_copy_release()
callback drops the folio references. Unregistering at teardown would
require operating on the ring context directly, whose validity is hard
to ascertain; this is deemed not worth the complexity for the abort
race, since everything is freed when the ring is torn down.
The throughput improvement from zero-copy depends on how much of the
per-request latency is spent on data copying vs backing I/O. The gain
comes from eliminating the payload-buffer memcpy, but accessing the
zero-copied pages requires the server to issue the read/write as an
IORING_OP_READ/WRITE_FIXED operation. The benefit is largest when the
mempcy is a meaningful fraction of per-request latency while backing i/o
is still noticable enough that the extra io-uring op's overhead doesn't
dominate.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered-buf zero-copy (zc vs base)
direct read ~5.1 GB/s ~5.4 GB/s ~8.9 GB/s (+75%)
direct write ~3.4 GB/s ~4.8 GB/s ~5.1 GB/s (+50%)
Reads end up higher than writes because the backing store reads faster
than it writes (the baseline shows the same read>write gap, and the raw
device does too). On a device-bound NVMe (~2 GB/s reads) the read gain
shrinks to ~10-16% (and no measurable gains for writes), as backing I/O
rather than the eliminated copy dominates latency.
The benefit overall scales with how much of the
per-request latency is the data copy versus backing I/O.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Allow servers to use a buffer pool that is also registered through
io-uring. When the server registers a buffer pool with io-uring, the
pages backing the pool are pinned upfront. This eliminates the overhead
of pinning/unpinning user pages and translating virtual addresses per
i/o request. This also allows servers to use the same registered memory
for subsequent backing store I/O (eg read_fixed/write_fixed), keeping
data in the same pinned pages without additional pinning or mapping
overhead required.
To use this, the server needs to set the FUSE_URING_REGISTERED_BUFPOOL
flag when adding a bufpool through the FUSE_IO_URING_CMD_ADD_BUFPOOL
cmd. For every sqe submitted (including the one for adding the bufpool),
it should set sqe->uring_cmd_flags to include IORING_URING_CMD_FIXED,
and pass in the index where the registered bufpool resides to
sqe->buf_index.
Benchmarked with passthrough_hp (--nopassthrough, q_depth=8) on a
2-socket Intel Xeon Gold 6138 (40 cores / 80 threads), using fio (sync
engine, bs=1M, O_DIRECT, numjobs=2, 30s run + 10s ramp, 3 runs) where
direct-I/O throughput is against a RAM-backed (tmpfs) source (backing
I/O is not the bottleneck):
baseline registered buffers
direct read ~5.1 GB/s ~5.4 GB/s (+~5%)
direct write ~3.4 GB/s ~4.8 GB/s (+~45%)
Registered buffers bring up the write path speed up closer to speed of
reads. There isn't much improvement for reads because it is already fast
enough where it's at the copy-bound ceiling (surpassing that requires
doing zero-copy). On a device-bound NVMe though, the differences are
within noise, as backing I/O dominates per-request latency.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Right now, ents and buffers are tightly coupled in fuse io-uring where
each entry has its own dedicated payload buffer, requiring N buffers for
N entries where each buffer must be large enough to accomodate the
maximum payload size. This is suboptimal as most request types (lookup,
open, release, getattr, etc) require vastly less bytes than the maximum
payload size and some requests (unlink, rmdir, fsync, flush, etc) do not
require payload buffers at all.
Instead of requiring a 1:1 coupling between ents and payload buffers,
allow the server to pass in a buffer pool (a contiguous chunk of memory)
that the kernel will use as it wishes for servicing ents/requests.
Entries only reserve a "buffer" from the pool while actively processing
a request that requires a payload buffer. This decoupling and letting
the kernel delegate memory from the pool for requests allows the kernel to
optimize memory usage and reduces the memory usage requirements needed
to use fuse-over-io-uring.
A pool is registered per queue with the new
FUSE_IO_URING_CMD_ADD_BUFPOOL command. The server passes the pool's base
address and length in fuse_uring_cmd_req.bufpool.{uaddr,len}.
Internally, the kernel splits the region into buffers of
ring->max_payload_sz bytes each (nr_bufs = pool len / max_payload_sz). A
queue commits to a payload mode on first use: registering an entry that
carries its own payload selects the legacy per-entry mode, while
ADD_BUFPOOL selects pool mode. The two are mutually exclusive, so
ADD_BUFPOOL must be issued before any payload-carrying entries are
registered on that queue. The queue must have been created before the
bufpool is added, through the FUSE_IO_URING_CMD_ADD_QUEUE command.
The kernel tracks free buffers with a bitmap (a set bit marks a free
buffer). On dispatch, a request that needs a payload claims a free
buffer (find_first_bit + clear). A request that needs none claims
nothing. The buffer's byte offset within the pool is reported to the
server in the new fuse_uring_ent_in_out.offset field so that the server
can locate the payload. On completion the buffer is returned to the pool
or reused directly if the next request on that entry also has a payload.
The FUSE_HAS_IO_URING_BUFPOOL flag advertises kernel support to the
server for bufpools.
Buffer pool request flow
~~~~~~~~~~~~~~~~~~~~~~~~
| Kernel | FUSE daemon
| |
| [request arrives] |
| [claim a free pool buffer] |
| >fuse_uring_select_buffer() |
| [copy headers to ring] |
| [copy payload to buffer] |
| [report buffer offset in ent_in_out] |
| >io_uring_cmd_done() |
| | [read headers]
| | [read/write payload at offset]
| | [process request]
| | >io_uring_submit()
| | COMMIT_AND_FETCH
| >fuse_uring_commit_fetch() |
| [copy reply from ring] |
| [return buffer to the pool] |
| >fuse_uring_recycle_buffer() |
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse-over-io-uring queues are currently created lazily, as a side effect
of the first FUSE_IO_URING_CMD_REGISTER command for a given qid. This
ties queue creation to entry registration.
Add a FUSE_IO_URING_CMD_ADD_QUEUE command so a server can create a queue
explicitly, decoupling queue setup from entry registration. This is
additionally a prerequisite for FUSE_IO_URING_CMD_ADD_BUFPOOL, which
attaches a buffer pool to an existing queue and therefore needs the
queue to have been created first.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Currently, the connection's fuse_ring is created lazily on the first
FUSE_IO_URING_CMD_REGISTER command. A server registers entries from one
thread per queue (one per CPU) and those threads issue their first
REGISTER command concurrently. They then race to create the single
per-connection fuse_ring, which required open-coded handling in
fuse_uring_create() to detect and protect against concurrent creations.
Decouple fuse_ring creation from ent registration and move it to
FUSE_INIT reply processing after a server has negotiated and set
FUSE_OVER_IO_URING. The ring is published before the connection is
marked initialized. fuse_uring_register() no longer creates the ring and
it instead uses the ring set up at init time.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Convert fuse iomap_ops to the new ->iomap_next() callback. Each callback is
generated with the DEFINE_IOMAP_ITER_NEXT()/DEFINE_IOMAP_ITER_NEXT_END()
macros, which wrap the iomap_iter_next() helper to finish the previous
mapping if needed and produce the next one. No functional changes are
intended.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Link: https://patch.msgid.link/20260729192737.3190206-19-joannelkoong@gmail.com
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
When fuse enables large folios, a large folio will be backed by
iomap_folio_state that keeps track of uptodate and dirty state in an
internal bitmap.
Fuse writethrough and notify store paths currently set folio uptodate
state with folio_mark_uptodate(), which touches only the folio-level
flag, but on an iomap-backed folio, that leaves the uptodate bitmap out
of sync.
Use the iomap_folio_mark_uptodate() helper to update both the folio
uptodate state and the iomap uptodate bitmap.
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Link: https://patch.msgid.link/20260707220450.1200943-4-joannelkoong@gmail.com
Acked-by: Miklos Szeredi <mszeredi@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
In the writethrough path (fuse_send_write_pages()), if the write to the
server failed or was a short write, the uptodate flag on the folios are
cleared.
As explained by Matthew in [1], this is dangerous because the folio may
be mapped into userspace. The mm code has the invariant that a
non-uptodate folio must never be visible to userspace (to avoid
potentially leaking confidental information to userspace) and has checks
in place for this that if violated can bring down the whole machine.
Practically speaking, the effect of this change for the fuse
writethrough error path is that if an application does a write and then
the server fails to persist the data or only services a short write, the
page cache folio keeps the data the application wrote instead of being
reverted to the server's contents on the next read. The failure is still
reported to the application synchronously through the short count /
error return of the write() syscall. Folios that were only partially
written are unaffected since they were never marked uptodate in the
first place (fuse_fill_write_page() only marks a folio as uptodate if
the whole folio was written to).
[1] https://lore.kernel.org/linux-fsdevel/ajtPMgO65FA1TXhi@casper.infradead.org/
Suggested-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Link: https://patch.msgid.link/20260707220450.1200943-2-joannelkoong@gmail.com
Acked-by: Miklos Szeredi <mszeredi@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
The only time that 'false' is passed as the 'excl' arg to the ->create
inode_operation is in lookup_open() when ->atomic_open is not provided
by the parent directory.
*all* directory inode_operations which do not have ->atomic_open
completely ignore the 'excl' arg.
Therefore we don't need the 'excl' arg. Those few ->create operations
which pay attention to the arg are only ever called with a value of
'true'.
We remove that arg and change all ->create operations to behave as those
thhe arg were 'true'.
Signed-off-by: NeilBrown <neil@brown.name>
Link: https://patch.msgid.link/178290671516.27465.15984496764174914338@noble.neil.brown.name
Reviewed-by: Jori Koolstra <jkoolstra@xs4all.nl>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
A FUSE server that advertises a large max_pages and max_write (e.g.
max_pages=256, max_write=1MB) cannot currently obtain matching
FUSE_READ request sizes from the kernel. Buffered sequential writes
arrive at the server at the negotiated max_write size, but a large
buffered read() is split into several smaller FUSE_READ requests.
For a buffered read, filemap_get_pages() -> page_cache_sync_ra() sizes
the read against ractl_max_pages():
max_pages = ractl->ra->ra_pages;
if (req_size > max_pages && bdi->io_pages > max_pages)
max_pages = min(req_size, bdi->io_pages);
fuse leaves bdi->io_pages at the default VM_READAHEAD_PAGES (128KB), so
a 1MB read() (req_size = 256 pages) is clamped to the readahead window
(128KB, or 256KB for POSIX_FADV_SEQUENTIAL), producing four 256KB
FUSE_READ round-trips instead of one.
Set bdi->io_pages to fc->max_pages after feature negotiation. As the
code above shows, io_pages only raises the limit when the request size
already exceeds the readahead window, so it enlarges explicitly
requested reads without enlarging the speculative readahead window.
This avoids increasing speculative page-cache readahead on behalf of
an unprivileged server. NFS does the same, setting io_pages from
rpages while leaving ra_pages at the default.
fc->max_pages is already bounded by fc->max_pages_limit (and, for
virtio-fs, by the virtqueue descriptor count), so io_pages inherits
the same bound.
Suggested-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Jim Harris <jim.harris@nvidia.com>
Assisted-by: Cursor:claude-opus-4.8
Reviewed-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_uring_create_queue() initializes a fuse_ring_queue and then
publishes the pointer into ring->queues[qid] with WRITE_ONCE() under the
fch->lock. There are several readers that may concurrently be fetching
that pointer locklessly and then deferencing it.
WRITE_ONCE() doesn't ensure ordering of the queue's field
initialization before the ring->queues[qid] pointer assignment. The
queue must be published with smp_store_release() so the field
initialization is guaranteed to happen before.
Readers in paths where the read may happen concurrently with the store
need to use READ_ONCE() because any race involving a plain access is
undefined.
Fixes: 24fe962c86 ("fuse: {io-uring} Handle SQEs - register commands")
Cc: stable@vger.kernel.org
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_chan_set_initialized() sets values for the connection state and
then sets fch->initialized to true, but lockless readers read
fch->initialized and if true, go to read the connection state values,
without using any barriers.
There are a few instances where this happens (fuse_uring_cmd() before
dispatching register / commit-and-fetch cmds, fuse_dev_do_wriite() for
handling notify retrieves, etc).
To make this as simple as possible, use release/acquire semantics for
writing/reading fch->initialized. Add the missing read barriers.
This is not marked for stable as these are not realistically reachable
on a well-behaved server, and buggy/malicious servers who trigger this
path fail benignly rather than crash or deadlock the kernel.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
fuse_block_alloc() reads fch->initialized and then fch->io_uring.
fch->io_uring is set before fch->initialized, ordered by the smp_wmb()
in fuse_chan_set_intialized(), but fuse_block_alloc() has no matching
read barrier between the two loads.
This may lead a CPU to observe fch->initialized=1 but fch->io_uring=0,
and skip the check that blocks request allocation until the io-uring
queues are ready. This can reintroduce the lock-order inversion deadlock
that commit 3393ff964e prevents.
Add an smp_rmb() barrier to pair with the smp_wmb() in
fuse_chan_set_initialized() to prevent this.
Fixes: 3393ff964e ("fuse: block request allocation until io-uring init is complete")
Cc: stable@vger.kernel.org
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
After commit f8fce75fed ("fuse: clear intr_entry in fuse_resend and
fuse_remove_pending_req") the WARN_ON(!list_empty(&req->intr_entry)) in
fuse_request_free() still triggers due to the following race:
In request_wait_answer()
if (test_bit(FR_SENT, &req->flags)) -> returns true
In fuse_chan_resend()
clear_bit(FR_SENT, &req->flags)
In request_wait_answer()
queue_interrupt(req)
Fix by:
- move clearing FR_SENT inside fpq->lock
- move setting FR_PENDING inside fiq->lock
- recheck FR_SENT after acquiring fiq->lock in fuse_dev_queue_interrupt()
Reported-by: zdi-disclosures@trendmicro.com
Fixes: f8fce75fed ("fuse: clear intr_entry in fuse_resend and fuse_remove_pending_req")
Cc: stable@vger.kernel.org # 6.9
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
When fuse enables large folios, a large folio will be backed by
iomap_folio_state that keeps track of uptodate and dirty state in an
internal bitmap.
Fuse writethrough and notify store paths currently set folio uptodate
state with folio_mark_uptodate(), which touches only the folio-level
flag, but on an iomap-backed folio, that leaves the uptodate bitmap out
of sync.
Use the iomap_folio_mark_uptodate() helper to update both the folio
uptodate state and the iomap uptodate bitmap.
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
In the writethrough path (fuse_send_write_pages()), if the write to the
server failed or was a short write, the uptodate flag on the folios are
cleared.
As explained by Matthew in [1], this is dangerous because the folio may
be mapped into userspace. The mm code has the invariant that a
non-uptodate folio must never be visible to userspace (to avoid
potentially leaking confidental information to userspace) and has checks
in place for this that if violated can bring down the whole machine.
Practically speaking, the effect of this change for the fuse
writethrough error path is that if an application does a write and then
the server fails to persist the data or only services a short write, the
page cache folio keeps the data the application wrote instead of being
reverted to the server's contents on the next read. The failure is still
reported to the application synchronously through the short count /
error return of the write() syscall. Folios that were only partially
written are unaffected since they were never marked uptodate in the
first place (fuse_fill_write_page() only marks a folio as uptodate if
the whole folio was written to).
[1] https://lore.kernel.org/linux-fsdevel/ajtPMgO65FA1TXhi@casper.infradead.org/
Suggested-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
...in hope of removing d_time one day.
Fixes: 2396356a94 ("fuse: add more control over cache invalidation behaviour")
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Move the call to fuse_send_readpages from the iomap ->submit_read method
to the fuse readahead implementation.
fuse_read_folio() does not need to call fuse_send_readpages() because it
always does reads synchronously (the iomap->submit_read method for this
was a no-op since data->ia is always NULL for fuse_read_folio()).
This prepares for an iomap fix that will call ->submit_read after each
iomap.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260629121750.3392300-3-hch@lst.de
Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
There is a comment in vfs_prepare_mode() that says:
Note that it's currently valid for @type to be 0 if a directory is
created. Filesystems raise that flag individually and we need to check
whether each filesystem can deal with receiving S_IFDIR from the vfs
before we enforce a non-zero type.
It is safe to do this clean-up except that three filesystems (fuse,
cifs, and coda) forward the mkdir @mode unchanged to something outside
the kernel. Mask S_IFDIR back out in coda_mkdir(), fuse_mkdir() and
cifs_mkdir() so that what is sent outside the kernel is unchanged.
Their maintainers can drop the mask once they have confirmed it is safe.
Assisted-by: LLM
Signed-off-by: Jori Koolstra <jkoolstra@xs4all.nl>
Link: https://patch.msgid.link/20260630105400.68459-2-jkoolstra@xs4all.nl
Reviewed-by: NeilBrown <neil@brown.name>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Pull fuse updates from Miklos Szeredi:
- Fix lots of bugs, most from the late 6.x era, but some going back
to 2.6.x
- Add subsystems (io-uring, passthrough) and respective maintainers
(Bernd, Joanne and Amir)
- Separate transport and fs layers (Miklos)
- Don't block on cat /dev/fuse (Joanne)
- Perform some refactoring in fuse-uring (Joanne)
- Don't use bounce-buffer for READDIR reply in virtio-fs (Matthew Ochs)
- Clean up documentation (Randy)
- Improve tracing (Amir)
- Extend page cache invalidation after DIO (Cheng Ding)
- Invalidate readdir cache on epoch change (Jun Wu)
- Misc cleanups
* tag 'fuse-update-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse: (81 commits)
fuse-uring: clear ent->fuse_req in commit_fetch error path
fuse-uring: use named constants for io-uring iovec indices
fuse-uring: refactor setting up copy state for payload copying
fuse-uring: use enum types for header copying
fuse-uring: refactor io-uring header copying from ring
fuse-uring: refactor io-uring header copying to ring
fuse-uring: separate next request fetching from sending logic
fuse: invalidate readdir cache on epoch bump
virtio-fs: avoid double-free on failed queue setup
fuse: invalidate page cache after DIO and async DIO writes
fuse: set ff->flock only on success
fuse: clean up interrupt reading
fuse: remove stray newline in fuse_dev_do_read()
fuse: use READ_ONCE in fuse_chan_num_background()
fuse: dax: Move long delayed work on system_dfl_long_wq
fuse: add fuse_request_sent tracepoint
fuse: Add SPDX ID lines to some files
fuse: use QSTR() instead of QSTR_INIT() in fuse_get_dentry
fuse: convert page array allocation to kcalloc()
fuse: use current creds for backing files
...
fuse_uring_commit_fetch() error path called fuse_request_end(req) without
clearing ent->fuse_req when fuse_ring_ent_set_commit() fails. The
still-pending fuse_uring_send_in_task() task-work later dereferences the
dangling pointer through fuse_uring_prepare_send(), causing a
use-after-free.
End the request with fuse_uring_req_end(), which handles all conditions
already.
Annotation/edition by Bernd: The UAF should be fixed by other means already
and actually has to be avoided that way.
Just checking for ent->fuse_req == NULL in fuse_uring_send_in_task()
would be prone to race conditions, because if malicious userspace
would commit requests that have passed the NULL check, but are
in doing args copy, it would still trigger a use-after-free.
Setting ent->fuse_req = NULL in fuse_uring_commit_fetch() still
makes sense, though.
Reported-by: Shuvam Pandey <shuvampandey1@gmail.com>
Reported-by: Berkant Koc <me@berkoc.com>
Signed-off-by: Zhenghang Xiao <kipreyyy@gmail.com>
Signed-off-by: Bernd Schubert <bernd@bsbernd.com>
Reviewed-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Replace magic indices 0 and 1 for the iovec array with named constants
FUSE_URING_IOV_HEADERS and FUSE_URING_IOV_PAYLOAD. This makes the usages
self-documenting and prepares for buffer ring support which will also
reference these iovec slots by index.
Reviewed-by: Bernd Schubert <bernd@bsbernd.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Use enum types to identify which part of the header needs to be copied.
This improves the interface and will simplify both kernel-space and
user-space header addresses copying when buffer rings are added.
Reviewed-by: Bernd Schubert <bschubert@ddn.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Move header copying from ring logic into a new copy_header_from_ring()
function. This makes the copy_from_user() logic more clear and
centralizes error handling / rate-limited logging.
Reviewed-by: Bernd Schubert <bschubert@ddn.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Move header copying to ring logic into a new copy_header_to_ring()
function. This makes the copy_to_user() logic more clear and centralizes
error handling / rate-limited logging.
Reviewed-by: Bernd Schubert <bschubert@ddn.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Simplify the logic for fetching + sending off the next request.
This gets rid of fuse_uring_send_next_to_ring() which contained
duplicated logic from fuse_uring_send(). This decouples request fetching
from the send operation, which makes the control flow clearer and
reduces unnecessary parameter passing.
Reviewed-by: Bernd Schubert <bschubert@ddn.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Reviewed-by: Baokun Li <libaokun@linux.alibaba.com>
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
FUSE_NOTIFY_INC_EPOCH invalidates dentries, but does not invalidate cached
readdir results. A process with cwd inside a FUSE mount can therefore
observe stale readdir(".") output after an epoch bump.
Fix this by recording epoch in the readdir cache and checking it on reuse.
Minimal reproducer:
- mount a tiny FUSE fs with an empty root directory
- on opendir, enable fi->cache_readdir and fi->keep_cache
- chdir into the mount and call readdir(".") to populate readdir cache
- make the FUSE server report one file in the root directory
- send only FUSE_NOTIFY_INC_EPOCH
- call readdir(".") again; before this change it stays stale, after this
change it sees the new file
Fixes: 2396356a94 ("fuse: add more control over cache invalidation behaviour")
Signed-off-by: Jun Wu <quark@meta.com>
Reviewed-by: Joanne Koong <joannelkoong@gmail.com>
Reviewed-by: Luis Henriques <luis@igalia.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
virtio_fs_setup_vqs() allocates fs->vqs and fs->mq_map before calling
virtio_find_vqs(). If virtio_find_vqs() fails, the error path frees both
pointers and returns an error to virtio_fs_probe().
virtio_fs_probe() then drops the last kobject reference, and
virtio_fs_ktype_release() frees fs->vqs and fs->mq_map again. This leaves
dangling pointers in struct virtio_fs and can trigger a double-free during
probe failure cleanup.
Set fs->vqs and fs->mq_map to NULL immediately after kfree() in the
virtio_fs_setup_vqs() error path so that the later kobject release sees an
uninitialized state and kfree(NULL) becomes harmless.
This can be reproduced when a broken virtio-fs device advertises more
request queues than the transport actually provides. In that case
virtio_find_vqs() fails while setting up the extra queue, and the probe
path reaches the double-free cleanup sequence.
Signed-off-by: Yung-Tse Cheng <mes900903@gmail.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
This fixe does page cache invalidation after DIO and async DIO writes for
both O_DIRECT and FOPEN_DIRECT_IO cases.
Commit b359af8275 ("fuse: Invalidate the page cache after FOPEN_DIRECT_IO
write") fixed xfstests generic/209 for DIO writes in the FOPEN_DIRECT_IO
path. DIO writes without FOPEN_DIRECT_IO are already handled by
generic_file_direct_write().
However, async DIO writes (xfstests generic/451) remain unhandled.
After this fix:
- Async write with FUSE_ASYNC_DIO:
invalidate in fuse_aio_invalidate_worker()
- Otherwise (Sync or async write without FUSE_ASYNC_DIO):
- With FOPEN_DIRECT_IO:
invalidate in fuse_direct_write_iter()
- Without FOPEN_DIRECT_IO:
invalidate in generic_file_direct_write()
Workqueue is required for async write invalidation to prevent deadlock:
calling it directly in the I/O end routine (which is in fuse worker thread
context) can block on a folio lock held by a buffered I/O thread waiting
for the same fuse worker thread.
Co-developed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Cheng Ding <cding@ddn.com>
Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Kees Cook