Sandro/linux
1
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-09-18 22:09:30 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm

Browse Source 
Pull kvm updates from Paolo Bonzini:
 "ARM64:

   - Add support for 'slot' based PMU events, paired with new UAPI that
     compels the user to select a specific PMU implementation

   - Lazy save/restore of vCPU state for pKVM, along with various fixes
     and cleanups to the management of vCPU state between the untrusted
     host and pKVM hypervisor

   - Disable traps of EL1 registers for nested hypervisors when
     FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits
     are stateful in the EL1 counterpart

   - Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the
     scope of those instructions remains 'in host' (i.e. L1
     kernel/userspace)

   - Pile of fixes for the management of the VNCR pseudo-TLB, such as
     under-invalidations and races with concurrent TLBIs on other vCPUs

   - Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
     runtime-patched constant, allowing the same data to be shared with
     pKVM prior to dropping host privileges

   - Considerable pile of LLM-assisted fixes around the shop but mostly
     in the VGIC, our in-kernel generator of bugs (and sometimes
     interrupts)

  LoongArch:

   - Advertise already-supported capabilities

   - Some bug fixes about timer and MMIO

   - Some hardening about interrupt injection

   - Replace kvm_err() with kvm_pr_unimpl()

   - Add FPU/LSX/LASX test cases for selftests

  RISC-V:

   - Svadu/Zicfiss/Zicfilp FWFT support for Guest

   - Use try_cmpxchg for IMSIC MRIF RMW

   - More arch-specific tracepoints in KVM RISC-V

   - Eager page splitting when enabling dirty logging

   - Optimize hfence request handling for SMP Guests

   - Improve dirty log clearing by skipping zero bits in mask

   - Guard HFENCE range loops against overflow

   - CPU PM notifiers in KVM RISC-V for non-retentive idle states

   - Fix kernel-mode vector context save/restore for Guest

  s390:

   - Fixes for vfio-ap

   - Fixes for the gmap rework

   - Fixes for vsie

   - AI triggered fixes all over

   - diag9c tracing

   - code move preparation for the additional arm64 support

   - enable CONTEXT_ANALYSIS

  x86:

   - Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding
     regs.c (the kvm_cache_regs.h => regs.h is already applied) and
     msrs.{c,h}, and moving relevant code out of x86.c

   - Split kvm_mmu in three parts, respectively to describe the format
     of page tables, walking the guest page tables and building the page
     tables. Always use the same page table walker kvm->arch.gva_walk as
     the entry point to convert a guest's virtual address, where the
     previous code used two different kvm_mmu structs depending on
     whether the walk included nested EPT/NPT or not. Make page fault
     vmexits reuse the permission checking machinery that is used for
     guest page faults. This is both a cleanup and a baby step towards
     supporting XS/XU memory permissions

   - Document some of the "fun" gotchas with the APIC base when creating
     IRQCHIPs on x86

   - Remove a defunct masterclock update from kvm_xen_shared_info_init().
     It could result in incorrect kvmclock due to triggering an
     unnecessary switch to/from masterclock mode

   - Skip Xen runstate time updates if time has effectively gone
     backwards, so that the guest doesn't report 100% steal time for
     a very, very long time

   - Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM
     was updating the wrong sub-leaf, and upstream KVM will soon provide
     all the information needed by userspace to populate the CPUID field
     itself

   - Fix a bug where KVM would walk a newly created rmap without holding
     the rmap lock (or mmu_lock) during aging

   - Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs

   - Fix a variety of #DB priority bugs

   - Fix a class of races related to enabling Hyper-V emulation on a
     vCPU after the vCPU is visible to the rest of KVM

   - Use static calls for nested virtualization ops

   - Move more KVM-internal code out of x86's kvm_host.h

   - Enumerate support for a variety of Zhaoxin instructions that don't
     require explicit virtualization

   - Fix missing EFER validation bugs, including in the KVM_SET_SREGS*
     path

   - Harden kvm_vcpu_map() against double-mapping and thus leaking
     references

   - Misc fixes and cleanups, e.g. for largely benign syzkaller splats

  x86 (Intel):

   - Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used
     for IPI virtualization when the vCPU is freed, to fix a
     use-after-free where hardware will write to a freed vCPU's PID

   - Service local TLB flushes on a failed nested VM-Enter to fix a bug
     where KVM could miss a TLB on a future, successful VM-Enter with
     the same L2 VPID

   - Cap the maximum value shoved into the VMX Preemption Timer to
     workaround an erratum that affects all existing Intel CPUs that
     support CPUID 0x15

   - Fix VPID virtualization bugs where KVM would fail to flush hardware
     TLBs

   - Harden the TDX "populate" ioctls against bad input, and to prepare
     for supporting in-place private<=>shared conversion

  x86 (AMD):

   - Forcefully invalidate SNP VMSA pages if their backing guest_memfd
     page is zapped/invalidated, e.g. due to a PUNCH_HOLE in response to
     a Page-State Change request

   - Remove a dying VM from the GA Log notifier list before the VM is
     actually destroyed, to fix a potential use-after-free

   - While FOLL_WRITE was needed in the past to trigger CoW unsharing,
     nowadays FOLL_LONGTERM does that already even without FOLL_WRITE,
     and in fact, get_user_pages() actually disallows FOLL_WRITE
     together with FOLL_LONGTERM. So don't pass FOLL_WRITE when
     registering encrypted memory regions, i.e. when pinning SEV/SEV-ES
     guest memory, to fix a regression with file-backed memory
     introduced by KVM's (correct) usage of long-term pins

     (This was reviewed by mm maintainers; for more information, see
     commit ee1a586dd1 "KVM: SEV: Drop FOLL_WRITE for encrypted region
     registration")

   - Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled
     hosts to fix a data corruption issue due to the PSP driver
     assigning to-be-written pages to firmware (as required by the SNP
     specs)

   - Unconditionally intercept ICBEP so that KVM generates the correct
     guest RIP when handling an ICEBP-induced TASK_SWITCH #VMEXIT

   - Harden the SNP "populate" ioctls against bad input, and to prepare
     for supporting in-place private<=>shared conversion

  Generic:

   - Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's
     debugfs

   - Add a per-VM bitmap to track which vCPU IDs have been "claimed" but
     for which the vCPU isn't yet online, and use the bitmap to reject
     duplicate IDs before calling into arch code. This allows arch code
     to consume vcpu_id without having to worry about cross-vCPU
     clobbering (at least s390 and x86 have had related bugs)

   - Rework the so called "prepare" and "invalidate" guest_memfd hooks
     to prepare for in-place private<=>shared conversion, and clean up a
     few warts along the way

  Selftests:

   - Automatically allocate a full page for L2 guest stacks on x86
     instead of requiring test-specific L1 guest code to carve out a
     portion of the L1 stack for L2 usage, and to ensure the L2 stack
     also adheres to the x86-64 calling convention ABI

   - Add a selftest to verify {Guest,Host}-Only behavior in x86's
     mediated PMU

   - Clean up nested SVM's handling of GPRs on L2<=>L1 transitions,
     reuse the functionality for nested VMX, and drop the ucall hack
     that was fudging around the lack of GPR switching on nVMX

   - Add a stress test to verify KVM doesn't clobber/drop #PF state,
     e.g. CR2, across save/restore, including when L2 is active

   - Add a test to verify KVM_CREATE_VM accepts exactly what is reported
     by KVM_CAP_VM_TYPES

   - Misc selftests fixes and cleanups

   - Fix several issues with seeding the pRNG, and rework the pRNG APIs
     to that the pRNG can be sanely used in host code, not just guest
     code

   - Add an IRQ test to validate virtual IRQ deliverty for IRQs wired up
     via KVM_IRQFD + KVM_SET_GSI_ROUTING, with optional support for
     triggering IRQs via writes to an assigned VFIO device

   - Add syscall wrappers to assert success on a variety of pthreads and
     CPU affinity APIs

   - Set vCPU pthread affinity as early as possible to reduce contention
     issues that were surfaced by PREEMPT_LAZY, which result in runtimes
     of over a minute on large hosts, versus the expected ~5 seconds

   - Rework the PMU counters test to run each testcase using a single VM
     with many vCPUs for each sub-testcase, instead of using a unique VM
     for each sub-testcase. This cuts the runtime by ~20x

  Miscellaneous:

   - MAINTAINERS updates for vfio-ap, guest_memfd, kvm-x86. Mostly
     representing the status quo more accurately, but also... welcome
     David Hildenbrand as guest_memfd reviewer!"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (413 commits)
  KVM: arm64: Validate GICv5 timer PPIs before claiming ownership
  KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs
  KVM: arm64: vgic: Prevent speculative SPI array underflow
  KVM: arm64: vgic: Free gic_kvm_info on initialization failure
  KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'
  s390/vfio-ap: Fix NULL deref in status_show() during queue probe
  s390/vfio-ap: Fix hot-unplug skipped when last AP adapter or domain removed
  s390/vfio-ap: fix potential use of uninitialized apm_filtered bitmap
  s390/vfio-ap: Fix control domain removal in vfio_ap_mdev_cfg_remove
  s390/vfio-ap: Fix required lock not held during update of ap_matrix_mdev object
  s390/vfio-ap: Fix missing lock required to access list of ap_matrix_mdev objects
  s390/vfio-ap: Fix dereference matrix_mdev->kvm without checking for NULL
  s390/vfio-ap: Fix stale do_remove flag across iterations in vfio_ap_mdev_cfg_remove
  RISC-V: KVM: fix vcpu vector context handling for kernel-mode vector
  riscv: vector: allow non-preemptible kernel-mode vector with IRQs off
  riscv: vector: refactor riscv_v_start_kernel_context
  KVM: s390: gmap: Make prefix handling optional
  KVM: s390: gmap: Make CMMA optional
  KVM: s390: gmap: Make storage keys optional
  KVM: s390: Prepare gmap for a second KVM implementation
  ...
This commit is contained in:
Linus Torvalds
2026-08-25 11:48:04 -07:00
parent 70f5376dbd 76671054f9
commit 7bb6284aa7
296 changed files with 12846 additions and 8610 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/admin-guide/kernel-parameters.txt
+5 -2
View File
@@ -3081,7 +3081,7 @@ Kernel parameters
Default is 0 (don't ignore, but inject #GP)
kvm.eager_page_split=
[KVM,X86] Controls whether or not KVM will try to
[KVM,X86,RISCV] Controls whether or not KVM will try to
proactively split all huge pages during dirty logging.
Eager page splitting reduces interruptions to vCPU
execution by eliminating the write-protection faults
@@ -3101,7 +3101,10 @@ Kernel parameters
the KVM_CLEAR_DIRTY ioctl, and only for the pages being
cleared.
Eager page splitting is only supported when kvm.tdp_mmu=Y.
On x86, eager page splitting is only supported when
kvm.tdp_mmu=Y.
On RISCV, eager page splitting is supported by default.
Default is Y (on).
Documentation/virt/kvm/api.rst
+25 -1
View File
@@ -856,12 +856,21 @@ Writes the floating point state to the vcpu.
Creates an interrupt controller model in the kernel.
On x86, creates a virtual ioapic, a virtual PIC (two PICs, nested), and sets up
future vcpus to have a local APIC. IRQ routing for GSIs 0-15 is set to both
PIC and IOAPIC; GSI 16-23 only go to the IOAPIC.
PIC and IOAPIC; GSI 16-23 only go to the IOAPIC. This ioctl can only be
called before creating any vcpus.
On arm64, a GICv2 is created. Any other GIC versions require the usage of
KVM_CREATE_DEVICE, which also supports creating a GICv2. Using
KVM_CREATE_DEVICE is preferred over KVM_CREATE_IRQCHIP for GICv2.
On s390, a dummy irq routing table is created.
On x86, subsequent vcpu creation may install a private 4 KiB memory slot at the
default APIC base address (0xfee00000). User memory regions must not overlap
this address; doing so will cause vcpu creation to fail with ``EEXIST``, or the
memory region to be rejected if created after the vcpu. This occurs when
APIC access acceleration is enabled (APICv on Intel, AVIC on AMD), which is
the default on supported hardware. The same constraint applies when using
``KVM_CAP_SPLIT_IRQCHIP``.
Note that on s390 the KVM_CAP_S390_IRQCHIP vm capability needs to be enabled
before KVM_CREATE_IRQCHIP can be used.
@@ -3515,6 +3524,17 @@ Possible features:
Depends on KVM_CAP_ARM_PSCI_0_2.
- KVM_ARM_VCPU_PMU_V3: Emulate PMUv3 for the CPU.
Depends on KVM_CAP_ARM_PMU_V3.
- KVM_ARM_VCPU_PMU_V3_STRICT: Enable strict PMUv3 UAPI.
Requires KVM_ARM_VCPU_PMU_V3. Depends on KVM_CAP_ARM_PMU_V3_STRICT.
When enabled:
* Userspace must explicitly select a PMU implementation before
initializing the PMU or configuring a PMU event filter
* If the PMU implements FEAT_PMUv3p4, PMMIR_EL1.SLOTS provides the
hardware value of the underlying implementation
* Writes to PMCR_EL0.N via KVM_SET_ONE_REG are ignored
- KVM_ARM_VCPU_PTRAUTH_ADDRESS: Enables Address Pointer authentication
for arm64 only.
@@ -8010,6 +8030,10 @@ used in the IRQ routing table. The first args[0] MSI routes are reserved
for the IOAPIC pins. Whenever the LAPIC receives an EOI for these routes,
a KVM_EXIT_IOAPIC_EOI vmexit will be reported to userspace.
As with ``KVM_CREATE_IRQCHIP``, subsequent vcpu creation may install a private
memory slot at the APIC base address (0xfee00000) that must not overlap user
memory regions. See ``KVM_CREATE_IRQCHIP`` for details.
Fails if VCPU has already been created, or if the irqchip is already in the
kernel (i.e. KVM_CREATE_IRQCHIP has already been called).
Documentation/virt/kvm/devices/vcpu.rst
+9 -2
View File
@@ -53,8 +53,9 @@ Returns:
======= ======================================================
-EEXIST Interrupt number already used
-ENODEV PMUv3 not supported or GIC not initialized
-ENXIO PMUv3 not supported, missing VCPU feature or interrupt
number not set (non-GICv5 guests, only)
-ENXIO PMUv3 not supported, missing VCPU feature, missing
hardware PMU, or interrupt number not set (non-GICv5
guests, only)
-EBUSY PMUv3 already initialized
======= ======================================================
@@ -62,6 +63,9 @@ Request the initialization of the PMUv3. If using the PMUv3 with an in-kernel
virtual GIC implementation, this must be done after initializing the in-kernel
irqchip.
When the KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature is enabled this must be done
after selecting a hardware PMU.
1.3 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_FILTER
-----------------------------------------
@@ -108,6 +112,9 @@ hardware event. Filtering event 0x1E (CHAIN) has no effect either, as it
isn't strictly speaking an event. Filtering the cycle counter is possible
using event 0x11 (CPU_CYCLES).
When the KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature is enabled this must be done
after selecting a hardware PMU.
1.4 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_SET_PMU
------------------------------------------
MAINTAINERS
+20 -2
View File
@@ -14268,6 +14268,7 @@ F: kernel/umh.c
KERNEL VIRTUAL MACHINE (KVM)
M: Paolo Bonzini <pbonzini@redhat.com>
R: Sean Christopherson <seanjc@google.com>
L: kvm@vger.kernel.org
S: Supported
W: http://www.linux-kvm.org
@@ -14385,6 +14386,7 @@ L: kvm@vger.kernel.org
S: Supported
P: Documentation/process/maintainer-kvm-x86.rst
T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
T: git https://github.com/kvm-x86/linux.git
F: Documentation/process/maintainer-kvm-x86.rst
F: arch/x86/include/asm/kvm*
F: arch/x86/include/asm/svm.h
@@ -14635,6 +14637,16 @@ S: Maintained
F: Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml
F: drivers/video/backlight/ktz8866.c
KVM GUEST_MEMFD
M: Sean Christopherson <seanjc@google.com>
M: Paolo Bonzini <pbonzini@redhat.com>
R: David Hildenbrand <david@kernel.org>
L: kvm@vger.kernel.org
S: Supported
T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
T: git https://github.com/kvm-x86/linux.git
F: virt/kvm/guest_memfd.*
KVM PARAVIRT (KVM/paravirt)
M: Paolo Bonzini <pbonzini@redhat.com>
R: Vitaly Kuznetsov <vkuznets@redhat.com>
@@ -14659,6 +14671,7 @@ M: Paolo Bonzini <pbonzini@redhat.com>
L: kvm@vger.kernel.org
S: Supported
T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
T: git https://github.com/kvm-x86/linux.git
F: arch/x86/kvm/hyperv.*
F: arch/x86/kvm/kvm_onhyperv.*
F: arch/x86/kvm/svm/hyperv.*
@@ -14673,6 +14686,7 @@ M: Paolo Bonzini <pbonzini@redhat.com>
L: kvm@vger.kernel.org
S: Supported
T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
T: git https://github.com/kvm-x86/linux.git
F: arch/x86/kvm/xen.*
L3MDEV
@@ -24126,12 +24140,14 @@ S: Supported
F: drivers/s390/block/scm*
F: drivers/s390/cio/scm.c
S390 VFIO AP DRIVER
S390 VFIO-AP DRIVER
M: Tony Krowiak <akrowiak@linux.ibm.com>
M: Halil Pasic <pasic@linux.ibm.com>
M: Jason Herne <jjherne@linux.ibm.com>
L: linux-s390@vger.kernel.org
L: kvm@vger.kernel.org
S: Supported
T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git
F: Documentation/arch/s390/vfio-ap*
F: drivers/s390/crypto/vfio_ap*
@@ -24142,6 +24158,7 @@ R: Halil Pasic <pasic@linux.ibm.com>
L: linux-s390@vger.kernel.org
L: kvm@vger.kernel.org
S: Supported
T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git
F: Documentation/arch/s390/vfio-ccw.rst
F: drivers/s390/cio/vfio_ccw*
F: include/uapi/linux/vfio_ccw.h
@@ -24153,7 +24170,8 @@ R: Eric Farman <farman@linux.ibm.com>
L: linux-s390@vger.kernel.org
L: kvm@vger.kernel.org
S: Supported
F: arch/s390/kvm/pci*
T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git
F: arch/s390/kvm/s390/pci*
F: drivers/vfio/pci/vfio_pci_zdev.c
F: include/uapi/linux/vfio_zdev.h
arch/arm64/include/asm/cpufeature.h
+1
View File
@@ -968,6 +968,7 @@ struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id);
extern struct arm64_ftr_override id_aa64mmfr0_override;
extern struct arm64_ftr_override id_aa64mmfr1_override;
extern struct arm64_ftr_override id_aa64mmfr2_override;
extern struct arm64_ftr_override id_aa64mmfr4_override;
extern struct arm64_ftr_override id_aa64pfr0_override;
extern struct arm64_ftr_override id_aa64pfr1_override;
extern struct arm64_ftr_override id_aa64zfr0_override;
arch/arm64/include/asm/kvm_arm.h
+12 -15
View File
@@ -287,21 +287,6 @@
GENMASK(19, 18) | \
GENMASK(15, 0))
/*
* Polarity masks for HCRX_EL2, limited to the bits that we know about
* at this point in time. It doesn't mean that we actually *handle*
* them, but that at least those that are not advertised to a guest
* will be RES0 for that guest.
*/
#define __HCRX_EL2_MASK (BIT_ULL(6))
#define __HCRX_EL2_nMASK (GENMASK_ULL(24, 14) | \
GENMASK_ULL(11, 7) | \
GENMASK_ULL(5, 0))
#define __HCRX_EL2_RES0 ~(__HCRX_EL2_nMASK | __HCRX_EL2_MASK)
#define __HCRX_EL2_RES1 ~(__HCRX_EL2_nMASK | \
__HCRX_EL2_MASK | \
__HCRX_EL2_RES0)
/* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */
#define HPFAR_MASK (~UL(0xf))
/*
@@ -348,4 +333,16 @@
{ PSR_AA32_MODE_UND, "32-bit UND" }, \
{ PSR_AA32_MODE_SYS, "32-bit SYS" }
/*
* ARMv8 Reset Values
*/
#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
PSR_F_BIT | PSR_D_BIT)
#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
PSR_F_BIT | PSR_D_BIT)
#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
PSR_AA32_I_BIT | PSR_AA32_F_BIT)
#endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_asm.h
+2 -2
View File
@@ -113,6 +113,7 @@ enum __kvm_host_smccc_func {
__KVM_HOST_SMCCC_FUNC___pkvm_finalize_teardown_vm,
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load,
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put,
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_sync_state,
__KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid,
MARKER(__KVM_HOST_SMCCC_FUNC_MAX)
@@ -214,7 +215,6 @@ struct kvm_nvhe_init_params {
unsigned long hcr_el2;
unsigned long vttbr;
unsigned long vtcr;
unsigned long tmp;
};
/*
@@ -281,7 +281,7 @@ extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
extern void __kvm_adjust_pc(struct kvm_vcpu *vcpu);
extern u64 __vgic_v3_get_gic_config(void);
extern bool __vgic_v3_get_gic_config(void);
extern void __vgic_v3_init_lrs(void);
#define __KVM_EXTABLE(from, to) \
arch/arm64/include/asm/kvm_emulate.h
+106 -1
View File
@@ -266,6 +266,25 @@ static inline bool vserror_state_is_nested(struct kvm_vcpu *vcpu)
(__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_TMEA);
}
static inline bool kvm_has_nv2(struct kvm *kvm)
{
return (cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
kvm_has_feat(kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY));
}
static inline bool kvm_has_nv3(struct kvm *kvm)
{
return (cpus_have_final_cap(ARM64_HAS_NV3) &&
kvm_has_feat(kvm, ID_AA64MMFR4_EL1, NV_frac, NV3));
}
static inline bool is_nested_nv3_ctxt(struct kvm_vcpu *vcpu)
{
return (has_vhe() && kvm_has_nv3(vcpu->kvm) && is_nested_ctxt(vcpu) &&
(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_EL2_NV) &&
(__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_NVTGE));
}
/*
* The layout of SPSR for an AArch32 state is different when observed from an
* AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
@@ -506,6 +525,12 @@ static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu)
return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
}
/* In nVHE hyp code, registers are always in memory: use the raw accessors. */
#if defined(__KVM_NVHE_HYPERVISOR__)
#define vcpu_read_sys_reg(v, r) __vcpu_sys_reg(v, r)
#define vcpu_write_sys_reg(v, x, r) __vcpu_assign_sys_reg(v, r, x)
#endif
static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu)
{
if (vcpu_mode_is_32bit(vcpu)) {
@@ -617,7 +642,7 @@ static __always_inline void kvm_incr_pc(struct kvm_vcpu *vcpu)
*/
static inline u64 vcpu_sanitised_cptr_el2(const struct kvm_vcpu *vcpu)
{
u64 cptr = __vcpu_sys_reg(vcpu, CPTR_EL2);
u64 cptr = vcpu_read_sys_reg(vcpu, CPTR_EL2);
if (!vcpu_el2_e2h_is_set(vcpu))
cptr = translate_cptr_el2_to_cpacr_el1(cptr);
@@ -686,6 +711,86 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu *vcpu)
if (kvm_has_feat(kvm, ID_AA64ISAR1_EL1, LS64, LS64_V))
vcpu->arch.hcrx_el2 |= HCRX_EL2_EnASR;
/*
* NV3 is a host-specific extension, and we always use
* it when present and that the guest uses NV. It may
* be hidden from the guest though.
*/
if (cpus_have_final_cap(ARM64_HAS_NV3) &&
vcpu_has_nv(vcpu) && vcpu_el2_e2h_is_set(vcpu)) {
vcpu->arch.hcrx_el2 |= HCRX_EL2_NVTGE;
/*
* If the guest is NV2-capable, then we need to see
* all the TLBIs, as configured in HCR_EL2.
* Otherwise, relax the TLBI traps to only TGE=0.
*/
if (!kvm_has_nv2(vcpu->kvm)) {
vcpu->arch.hcrx_el2 |= (HCRX_EL2_NVnTTLB |
HCRX_EL2_NVnTTLBIS);
if (kvm_has_feat(kvm, ID_AA64ISAR0_EL1, TLB, OS))
vcpu->arch.hcrx_el2 |= HCRX_EL2_NVnTTLBOS;
}
}
}
}
/* Reset a vcpu's core registers. */
static inline void kvm_reset_vcpu_core(struct kvm_vcpu *vcpu)
{
u32 pstate;
if (vcpu_el1_is_32bit(vcpu))
pstate = VCPU_RESET_PSTATE_SVC;
else if (vcpu_has_nv(vcpu))
pstate = VCPU_RESET_PSTATE_EL2;
else
pstate = VCPU_RESET_PSTATE_EL1;
/* Reset core registers */
memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu)));
memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs));
vcpu->arch.ctxt.spsr_abt = 0;
vcpu->arch.ctxt.spsr_und = 0;
vcpu->arch.ctxt.spsr_irq = 0;
vcpu->arch.ctxt.spsr_fiq = 0;
vcpu_gp_regs(vcpu)->pstate = pstate;
}
/* PSCI reset handling for a vcpu. */
static inline void kvm_reset_vcpu_psci(struct kvm_vcpu *vcpu,
struct vcpu_reset_state *reset_state)
{
unsigned long target_pc = reset_state->pc;
/* Gracefully handle Thumb2 entry point */
if (vcpu_mode_is_32bit(vcpu) && (target_pc & 1)) {
target_pc &= ~1UL;
vcpu_set_thumb(vcpu);
}
/* Propagate caller endianness */
if (reset_state->be)
kvm_vcpu_set_be(vcpu);
*vcpu_pc(vcpu) = target_pc;
/*
* We may come from a state where either a PC update was
* pending (SMC call resulting in PC being increpented to
* skip the SMC) or a pending exception. Make sure we get
* rid of all that, as this cannot be valid out of reset.
*
* Note that clearing the exception mask also clears PC
* updates, but that's an implementation detail, and we
* really want to make it explicit.
*/
vcpu_clear_flag(vcpu, PENDING_EXCEPTION);
vcpu_clear_flag(vcpu, EXCEPT_MASK);
vcpu_clear_flag(vcpu, INCREMENT_PC);
vcpu_set_reg(vcpu, 0, reset_state->r0);
}
#endif /* __ARM64_KVM_EMULATE_H__ */
arch/arm64/include/asm/kvm_host.h
+10 -4
View File
@@ -39,7 +39,7 @@
#define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS
#define KVM_VCPU_MAX_FEATURES 9
#define KVM_VCPU_MAX_FEATURES 10
#define KVM_VCPU_VALID_FEATURES (BIT(KVM_VCPU_MAX_FEATURES) - 1)
#define KVM_REQ_SLEEP \
@@ -387,6 +387,9 @@ struct kvm_arch {
/* Maximum number of counters for the guest */
u8 nr_pmu_counters;
/* PMMIR_EL1.SLOTS value exposed to the guest. */
u8 pmmir_slots;
/* Hypercall features firmware registers' descriptor */
struct kvm_smccc_features smccc_feat;
struct maple_tree smccc_filter;
@@ -411,8 +414,8 @@ struct kvm_arch {
/* Masks for VNCR-backed and general EL2 sysregs */
struct kvm_sysreg_masks *sysreg_masks;
/* Count the number of VNCR_EL2 currently mapped */
atomic_t vncr_map_count;
/* Count the number of VNCR_EL2 TLBs */
atomic_t vncr_tlb_count;
/*
* For an untrusted host VM, 'pkvm.handle' is used to lookup
@@ -543,6 +546,7 @@ enum vcpu_sysreg {
MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */
CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */
ZCR_EL2, /* SVE Control Register (EL2) */
HCR_EL2, /* Hypervisor Control Register */
/* Any VNCR-capable reg goes after this point */
MARKER(__VNCR_START__),
@@ -571,7 +575,7 @@ enum vcpu_sysreg {
VNCR(TFSR_EL1), /* Tag Fault Status Register (EL1) */
VNCR(VPIDR_EL2),/* Virtualization Processor ID Register */
VNCR(VMPIDR_EL2),/* Virtualization Multiprocessor ID Register */
VNCR(HCR_EL2), /* Hypervisor Configuration Register */
VNCR(NVHCR_EL2),/* NV Hypervisor Configuration Register */
VNCR(HSTR_EL2), /* Hypervisor System Trap Register */
VNCR(VTTBR_EL2),/* Virtualization Translation Table Base Register */
VNCR(VTCR_EL2), /* Virtualization Translation Control Register */
@@ -1051,6 +1055,8 @@ struct kvm_vcpu_arch {
#define INCREMENT_PC __vcpu_single_flag(iflags, BIT(1))
/* Target EL/MODE (not a single flag, but let's abuse the macro) */
#define EXCEPT_MASK __vcpu_single_flag(iflags, GENMASK(3, 1))
/* Host-set: the hyp flushes the non-protected vCPU state in on entry */
#define PKVM_HOST_STATE_DIRTY __vcpu_single_flag(iflags, BIT(4))
/* Helpers to encode exceptions with minimum fuss */
#define __EXCEPT_MASK_VAL unpack_vcpu_flag(EXCEPT_MASK)
arch/arm64/include/asm/kvm_nested.h
+20
View File
@@ -291,6 +291,13 @@ static inline u64 decode_range_tlbi(u64 val, u64 *range, u16 *asid)
base = (val & GENMASK(36, 0)) << shift;
/*
* We only deal with at most 48bit VA/IPA, so 48 is where we
* sign-extend from. Should we support FEAT_L{VP}A* at some point,
* this will need to be revisited.
*/
base = (u64)sign_extend64(base, 48);
if (asid)
*asid = FIELD_GET(TLBIR_ASID_MASK, val);
@@ -298,6 +305,12 @@ static inline u64 decode_range_tlbi(u64 val, u64 *range, u16 *asid)
num = FIELD_GET(GENMASK(43, 39), val);
*range = __TLBI_RANGE_PAGES(num, scale) << shift;
/* Cap the range to the correct half of the address space */
if (!(base & BIT(48)))
*range = min(*range, (BIT(48) - base));
else
*range = min(*range, ~base + 1);
return base;
}
@@ -388,6 +401,8 @@ struct s1_walk_result {
bool failed;
};
#define S1_MMU_DISABLED (-127)
static inline void fail_s1_walk(struct s1_walk_result *wr, u8 fst, bool s1ptw)
{
wr->fst = fst;
@@ -396,6 +411,11 @@ static inline void fail_s1_walk(struct s1_walk_result *wr, u8 fst, bool s1ptw)
wr->failed = true;
}
static inline bool s1_walk_translated(struct s1_walk_result *wr)
{
return wr->level != S1_MMU_DISABLED;
}
int __kvm_translate_va(struct kvm_vcpu *vcpu, struct s1_walk_info *wi,
struct s1_walk_result *wr, u64 va);
int __kvm_find_s1_desc_level(struct kvm_vcpu *vcpu, u64 va, u64 ipa,
arch/arm64/include/asm/kvm_pkvm.h
+7 -1
View File
@@ -45,6 +45,9 @@ static inline bool kvm_pkvm_ext_allowed(struct kvm *kvm, long ext)
return true;
case KVM_CAP_ARM_MTE:
return false;
case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
case KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES:
return false;
default:
return !kvm || !kvm_vm_is_protected(kvm);
}
@@ -195,7 +198,10 @@ struct pkvm_mapping {
struct rb_node node;
u64 gfn;
u64 pfn;
u64 nr_pages;
struct {
u64 nr_pages:48;
u64 nc:1;
};
u64 __subtree_last; /* Internal member for interval tree */
};
arch/arm64/include/asm/vncr_mapping.h
+1 -1
View File
@@ -11,7 +11,7 @@
#define VNCR_VTCR_EL2 0x040
#define VNCR_VMPIDR_EL2 0x050
#define VNCR_CNTVOFF_EL2 0x060
#define VNCR_HCR_EL2 0x078
#define VNCR_NVHCR_EL2 0x078
#define VNCR_HSTR_EL2 0x080
#define VNCR_VPIDR_EL2 0x088
#define VNCR_TPIDR_EL2 0x090
arch/arm64/include/uapi/asm/kvm.h
+1
View File
@@ -106,6 +106,7 @@ struct kvm_regs {
#define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */
#define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */
#define KVM_ARM_VCPU_HAS_EL2_E2H0 8 /* Limit NV support to E2H RES0 */
#define KVM_ARM_VCPU_PMU_V3_STRICT 9 /* No default PMU creation */
struct kvm_vcpu_init {
__u32 target;
arch/arm64/kernel/asm-offsets.c
-1
View File
@@ -124,7 +124,6 @@ int main(void)
DEFINE(NVHE_INIT_HCR_EL2, offsetof(struct kvm_nvhe_init_params, hcr_el2));
DEFINE(NVHE_INIT_VTTBR, offsetof(struct kvm_nvhe_init_params, vttbr));
DEFINE(NVHE_INIT_VTCR, offsetof(struct kvm_nvhe_init_params, vtcr));
DEFINE(NVHE_INIT_TMP, offsetof(struct kvm_nvhe_init_params, tmp));
#endif
#ifdef CONFIG_CPU_PM
DEFINE(CPU_CTX_SP, offsetof(struct cpu_suspend_ctx, sp));
arch/arm64/kernel/cpu_errata.c
+15 -2
View File
@@ -272,7 +272,7 @@ has_neoverse_n1_erratum_1542419(const struct arm64_cpu_capabilities *entry,
return is_midr_in_range(&range) && has_dic;
}
static const struct midr_range impdef_pmuv3_cpus[] = {
static const struct midr_range apple_cpus[] = {
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM),
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM),
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM_PRO),
@@ -301,7 +301,14 @@ static bool has_impdef_pmuv3(const struct arm64_cpu_capabilities *entry, int sco
if (pmuver != ID_AA64DFR0_EL1_PMUVer_IMP_DEF)
return false;
return is_midr_in_range_list(impdef_pmuv3_cpus);
return is_midr_in_range_list(apple_cpus);
}
static bool has_broken_gic_v3_seis(const struct arm64_cpu_capabilities *entry, int scope)
{
return (is_kernel_in_hyp_mode() &&
is_midr_in_range_list(apple_cpus) &&
(read_sysreg_s(SYS_ICH_VTR_EL2) & ICH_VTR_EL2_SEIS));
}
static void cpu_enable_impdef_pmuv3_traps(const struct arm64_cpu_capabilities *__unused)
@@ -1017,6 +1024,12 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
.matches = has_impdef_pmuv3,
.cpu_enable = cpu_enable_impdef_pmuv3_traps,
},
{
.desc = "Known broken GICv3 SEIS implementation",
.capability = ARM64_WORKAROUND_GICv3_BROKEN_SEIS,
.type = ARM64_CPUCAP_SYSTEM_FEATURE,
.matches = has_broken_gic_v3_seis,
},
{
}
};
arch/arm64/kernel/cpufeature.c
+17 -1
View File
@@ -785,6 +785,7 @@ static const struct arm64_ftr_bits ftr_raz[] = {
struct arm64_ftr_override __read_mostly id_aa64mmfr0_override;
struct arm64_ftr_override __read_mostly id_aa64mmfr1_override;
struct arm64_ftr_override __read_mostly id_aa64mmfr2_override;
struct arm64_ftr_override __read_mostly id_aa64mmfr4_override;
struct arm64_ftr_override __read_mostly id_aa64pfr0_override;
struct arm64_ftr_override __read_mostly id_aa64pfr1_override;
struct arm64_ftr_override __read_mostly id_aa64zfr0_override;
@@ -858,7 +859,8 @@ static const struct __ftr_reg_entry {
ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2,
&id_aa64mmfr2_override),
ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3),
ARM64_FTR_REG(SYS_ID_AA64MMFR4_EL1, ftr_id_aa64mmfr4),
ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR4_EL1, ftr_id_aa64mmfr4,
&id_aa64mmfr4_override),
/* Op1 = 0, CRn = 10, CRm = 4 */
ARM64_FTR_REG(SYS_MPAMIDR_EL1, ftr_mpamidr),
@@ -2614,6 +2616,20 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
{ /* Sentinel */ }
},
},
{
.desc = "FEAT_NV2p1",
.capability = ARM64_HAS_NV2P1,
.type = ARM64_CPUCAP_SYSTEM_FEATURE,
.matches = has_cpuid_feature,
ARM64_CPUID_FIELDS(ID_AA64MMFR4_EL1, NV_frac, NV2P1)
},
{
.desc = "FEAT_NV3",
.capability = ARM64_HAS_NV3,
.type = ARM64_CPUCAP_SYSTEM_FEATURE,
.matches = has_cpuid_feature,
ARM64_CPUID_FIELDS(ID_AA64MMFR4_EL1, NV_frac, NV3)
},
{
.capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE,
.type = ARM64_CPUCAP_SYSTEM_FEATURE,
arch/arm64/kernel/image-vars.h
+2
View File
@@ -51,6 +51,7 @@ PI_EXPORT_SYM(id_aa64isar2_override);
PI_EXPORT_SYM(id_aa64mmfr0_override);
PI_EXPORT_SYM(id_aa64mmfr1_override);
PI_EXPORT_SYM(id_aa64mmfr2_override);
PI_EXPORT_SYM(id_aa64mmfr4_override);
PI_EXPORT_SYM(id_aa64pfr0_override);
PI_EXPORT_SYM(id_aa64pfr1_override);
PI_EXPORT_SYM(id_aa64smfr0_override);
@@ -92,6 +93,7 @@ KVM_NVHE_ALIAS(spectre_bhb_patch_wa3);
KVM_NVHE_ALIAS(spectre_bhb_patch_clearbhb);
KVM_NVHE_ALIAS(alt_cb_patch_nops);
KVM_NVHE_ALIAS(kvm_compute_ich_hcr_trap_bits);
KVM_NVHE_ALIAS(kvm_patch_ich_vtr_el2);
/* Global kernel state accessed by nVHE hyp code. */
KVM_NVHE_ALIAS(kvm_vgic_global_state);
arch/arm64/kernel/pi/idreg-override.c
+10
View File
@@ -107,6 +107,15 @@ static const struct ftr_set_desc mmfr2 __prel64_initconst = {
},
};
static const struct ftr_set_desc mmfr4 __prel64_initconst = {
.name = "id_aa64mmfr4",
.override = &id_aa64mmfr4_override,
.fields = {
FIELD("nv_frac", ID_AA64MMFR4_EL1_NV_frac_SHIFT, NULL),
{}
},
};
static bool __init pfr0_sve_filter(u64 val)
{
/*
@@ -221,6 +230,7 @@ PREL64(const struct ftr_set_desc, reg) regs[] __prel64_initconst = {
{ &mmfr0 },
{ &mmfr1 },
{ &mmfr2 },
{ &mmfr4 },
{ &pfr0 },
{ &pfr1 },
{ &isar1 },
arch/arm64/kvm/arch_timer.c
+11 -4
View File
@@ -876,8 +876,14 @@ static void timer_set_traps(struct kvm_vcpu *vcpu, struct timer_map *map)
assign_clear_set_bit(tvt02, CNTHCTL_EL1NVVCT, clr, set);
assign_clear_set_bit(tpt02, CNTHCTL_EL1NVPCT, clr, set);
/* This only happens on VHE, so use the CNTHCTL_EL2 accessor. */
sysreg_clear_set(cnthctl_el2, clr, set);
/*
* This only happens on VHE, so use the CNTHCTL_EL2 accessor, unless
* we are sure CNTKCTL_EL1 is completely stateful with FEAT_NV2p1.
*/
if (!cpus_have_final_cap(ARM64_HAS_NV2P1))
sysreg_clear_set(cnthctl_el2, clr, set);
else
sysreg_clear_set(cntkctl_el1, clr, set);
}
void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu)
@@ -1529,13 +1535,14 @@ static bool timer_irqs_are_valid(struct kvm_vcpu *vcpu)
ctx = vcpu_get_timer(vcpu, i);
irq = timer_irq(ctx);
if (kvm_vgic_set_owner(vcpu, irq, ctx))
break;
/* With GICv5, the default PPI is what you get -- nothing else */
if (vgic_is_v5(vcpu->kvm) && irq != get_vgic_ppi(vcpu->kvm, default_ppi[i]))
break;
if (kvm_vgic_set_owner(vcpu, irq, ctx))
break;
/*
* We know by construction that we only have PPIs, so all values
* are less than 32 for non-GICv5 VGICs. On GICv5, they are
arch/arm64/kvm/arm.c
+22 -4
View File
@@ -465,6 +465,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
r = get_num_wrps();
break;
case KVM_CAP_ARM_PMU_V3:
case KVM_CAP_ARM_PMU_V3_STRICT:
r = kvm_supports_guest_pmuv3();
break;
case KVM_CAP_ARM_INJECT_SERROR_ESR:
@@ -748,6 +749,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
if (is_protected_kvm_enabled()) {
kvm_call_hyp(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3);
kvm_call_hyp_nvhe(__pkvm_vcpu_put);
/* __pkvm_vcpu_put implies a sync of the state */
if (!kvm_vm_is_protected(vcpu->kvm))
vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
}
kvm_vcpu_put_debug(vcpu);
@@ -979,6 +984,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
return ret;
if (is_protected_kvm_enabled()) {
/* Start with the vcpu in a dirty state */
if (!kvm_vm_is_protected(vcpu->kvm))
vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
ret = pkvm_create_hyp_vm(kvm);
if (ret)
return ret;
@@ -1576,8 +1584,10 @@ static unsigned long system_supported_vcpu_features(void)
if (!cpus_have_final_cap(ARM64_HAS_32BIT_EL1))
clear_bit(KVM_ARM_VCPU_EL1_32BIT, &features);
if (!kvm_supports_guest_pmuv3())
if (!kvm_supports_guest_pmuv3()) {
clear_bit(KVM_ARM_VCPU_PMU_V3, &features);
clear_bit(KVM_ARM_VCPU_PMU_V3_STRICT, &features);
}
if (!system_supports_sve())
clear_bit(KVM_ARM_VCPU_SVE, &features);
@@ -1618,6 +1628,11 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,
test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features))
return -EINVAL;
/* Strict PMUv3 UAPI requires PMUv3. */
if (test_bit(KVM_ARM_VCPU_PMU_V3_STRICT, &features) &&
!test_bit(KVM_ARM_VCPU_PMU_V3, &features))
return -EINVAL;
if (!test_bit(KVM_ARM_VCPU_EL1_32BIT, &features))
return 0;
@@ -1647,10 +1662,13 @@ static int kvm_setup_vcpu(struct kvm_vcpu *vcpu)
int ret = 0;
/*
* When the vCPU has a PMU, but no PMU is set for the guest
* yet, set the default one.
* When the vCPU has a PMU, but no PMU is set for the guest yet, set
* the default one. If KVM_ARM_VCPU_PMU_V3_STRICT is set, no default
* PMU is created, and userspace must select a PMU via
* KVM_ARM_VCPU_PMU_V3_SET_PMU.
*/
if (kvm_vcpu_has_pmu(vcpu) && !kvm->arch.arm_pmu)
if (kvm_vcpu_has_pmu(vcpu) && !kvm->arch.arm_pmu &&
!kvm_vcpu_has_pmuv3_strict(vcpu))
ret = kvm_arm_set_default_pmu(kvm);
/* Prepare for nested if required */
arch/arm64/kvm/at.c
-2
View File
@@ -11,8 +11,6 @@
#include <asm/kvm_mmu.h>
#include <asm/lsui.h>
#define S1_MMU_DISABLED (-127)
static int get_ia_size(struct s1_walk_info *wi)
{
return 64 - wi->txsz;
arch/arm64/kvm/config.c
+25 -2
View File
@@ -225,6 +225,7 @@ struct reg_feat_map_desc {
#define FEAT_HCX ID_AA64MMFR1_EL1, HCX, IMP
#define FEAT_S2PIE ID_AA64MMFR3_EL1, S2PIE, IMP
#define FEAT_GCIE ID_AA64PFR2_EL1, GCIE, IMP
#define FEAT_NV3 ID_AA64MMFR4_EL1, NV_frac, NV3
static bool not_feat_aa64el3(struct kvm *kvm)
{
@@ -904,6 +905,12 @@ static const DECLARE_FEAT_MAP_FGT(hdfgwtr2_desc, hdfgwtr2_masks,
static const struct reg_bits_to_feat_map hcrx_feat_map[] = {
NEEDS_FEAT(HCRX_EL2_NVTGE |
HCRX_EL2_NVnTTLB |
HCRX_EL2_NVnTTLBIS |
HCRX_EL2_NVnTTLBOS,
FEAT_NV3),
NEEDS_FEAT(HCRX_EL2_SRMASKEn, FEAT_SRMASK),
NEEDS_FEAT(HCRX_EL2_PACMEn, feat_pauth_lr),
NEEDS_FEAT(HCRX_EL2_EnFPM, FEAT_FPMR),
NEEDS_FEAT(HCRX_EL2_GCSEn, FEAT_GCS),
@@ -930,10 +937,12 @@ static const struct reg_bits_to_feat_map hcrx_feat_map[] = {
NEEDS_FEAT(HCRX_EL2_EnASR, FEAT_LS64_V),
NEEDS_FEAT(HCRX_EL2_EnALS, FEAT_LS64),
NEEDS_FEAT(HCRX_EL2_EnAS0, FEAT_LS64_ACCDATA),
FORCE_RES0(HCRX_EL2_RES0),
FORCE_RES1(HCRX_EL2_RES1),
};
static const DECLARE_FEAT_MAP(hcrx_desc, __HCRX_EL2,
static const DECLARE_FEAT_MAP(hcrx_desc, HCRX_EL2,
hcrx_feat_map, FEAT_HCX);
static const struct reg_bits_to_feat_map hcr_feat_map[] = {
@@ -1010,6 +1019,9 @@ static const struct reg_bits_to_feat_map hcr_feat_map[] = {
static const DECLARE_FEAT_MAP(hcr_desc, HCR_EL2,
hcr_feat_map, FEAT_AA64EL2);
static const DECLARE_FEAT_MAP(nvhcr_desc, NVHCR_EL2,
hcr_feat_map, FEAT_NV3);
static const struct reg_bits_to_feat_map sctlr2_feat_map[] = {
NEEDS_FEAT(SCTLR2_EL1_NMEA |
SCTLR2_EL1_EASE,
@@ -1384,6 +1396,7 @@ void __init check_feature_map(void)
check_reg_desc(&hdfgwtr2_desc);
check_reg_desc(&hcrx_desc);
check_reg_desc(&hcr_desc);
check_reg_desc(&nvhcr_desc);
check_reg_desc(&sctlr2_desc);
check_reg_desc(&tcr2_el2_desc);
check_reg_desc(&sctlr_el1_desc);
@@ -1579,11 +1592,21 @@ struct resx get_reg_fixed_bits(struct kvm *kvm, enum vcpu_sysreg reg)
break;
case HCRX_EL2:
resx = compute_reg_resx_bits(kvm, &hcrx_desc, 0, 0);
resx.res1 |= __HCRX_EL2_RES1;
break;
case HCR_EL2:
resx = compute_reg_resx_bits(kvm, &hcr_desc, 0, 0);
break;
case NVHCR_EL2:
/*
* Only apply sanitisation if we do have FEAT_NV3.
* Otherwise, the register aliases with HCR_EL2 in VNCR,
* and we're better off relying on data transfers between
* NVHCR_EL2 and HCR_EL2 to sanitise things.
*/
resx = (kvm_has_nv3(kvm) ?
compute_reg_resx_bits(kvm, &nvhcr_desc, 0, 0) :
(typeof(resx)){});
break;
case SCTLR2_EL1:
case SCTLR2_EL2:
resx = compute_reg_resx_bits(kvm, &sctlr2_desc, 0, 0);
arch/arm64/kvm/emulate-nested.c
+13 -5
View File
@@ -136,6 +136,8 @@ enum cgt_group_id {
CGT_CPTR_TTA,
CGT_MDCR_HPMN,
CGT_HCR_NV_HCRX_nNVTGE,
/* Must be last */
__NR_CGT_GROUP_IDS__
};
@@ -588,6 +590,15 @@ static enum trap_behaviour check_mdcr_hpmn(struct kvm_vcpu *vcpu)
return BEHAVE_HANDLE_LOCALLY;
}
static enum trap_behaviour check_hcr_nv_hcrx_nnvtge(struct kvm_vcpu *vcpu)
{
if ((__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_EL2_NV) &&
!(__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_NVTGE))
return BEHAVE_FORWARD_RW;
return BEHAVE_HANDLE_LOCALLY;
}
#define CCC(id, fn) \
[id - __COMPLEX_CONDITIONS__] = fn
@@ -598,6 +609,7 @@ static const complex_condition_check ccc[] = {
CCC(CGT_CNTHCTL_EL1NVVCT, check_cnthctl_el1nvvct),
CCC(CGT_CPTR_TTA, check_cptr_tta),
CCC(CGT_MDCR_HPMN, check_mdcr_hpmn),
CCC(CGT_HCR_NV_HCRX_nNVTGE, check_hcr_nv_hcrx_nnvtge),
};
/*
@@ -853,6 +865,7 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = {
SR_TRAP(SYS_SCTLR2_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_HCR_EL2,
SYS_HCRX_EL2, CGT_HCR_NV),
SR_TRAP(SYS_NVHCR_EL2, CGT_HCR_NV_HCRX_nNVTGE),
SR_TRAP(SYS_SMPRIMAP_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SMCR_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_TTBR0_EL2,
@@ -2320,7 +2333,6 @@ int __init populate_nv_trap_config(void)
BUILD_BUG_ON(__NR_CGT_GROUP_IDS__ > BIT(TC_CGT_BITS));
BUILD_BUG_ON(__NR_FGT_GROUP_IDS__ > BIT(TC_FGT_BITS));
BUILD_BUG_ON(__NR_FG_FILTER_IDS__ > BIT(TC_FGF_BITS));
BUILD_BUG_ON(__HCRX_EL2_MASK & __HCRX_EL2_nMASK);
for (int i = 0; i < ARRAY_SIZE(encoding_to_cgt); i++) {
const struct encoding_to_trap_config *cgt = &encoding_to_cgt[i];
@@ -2346,10 +2358,6 @@ int __init populate_nv_trap_config(void)
}
}
if (__HCRX_EL2_RES0 != HCRX_EL2_RES0)
kvm_info("Sanitised HCR_EL2_RES0 = %016llx, expecting %016llx\n",
__HCRX_EL2_RES0, HCRX_EL2_RES0);
kvm_info("nv: %ld coarse grained trap handlers\n",
ARRAY_SIZE(encoding_to_cgt));
arch/arm64/kvm/handle_exit.c
+36 -4
View File
@@ -486,9 +486,32 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index)
}
}
static void handle_exit_pkvm_state(struct kvm_vcpu *vcpu, int exception_index)
{
int exception_code = ARM_EXCEPTION_CODE(exception_index);
if (!is_protected_kvm_enabled() || kvm_vm_is_protected(vcpu->kvm))
return;
/*
* Sync the context back when the host will read (trap) or write
* (SError) it. Preempt-off here, so the loaded hyp vCPU is stable.
*/
if (exception_code == ARM_EXCEPTION_TRAP ||
exception_code == ARM_EXCEPTION_EL1_SERROR ||
ARM_SERROR_PENDING(exception_index)) {
kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state);
vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY);
} else {
vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY);
}
}
/* For exit types that need handling before we can be preempted */
void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
{
handle_exit_pkvm_state(vcpu, exception_index);
if (ARM_SERROR_PENDING(exception_index)) {
if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) {
u64 disr = kvm_vcpu_get_disr(vcpu);
@@ -507,10 +530,20 @@ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index)
kvm_handle_guest_serror(vcpu, kvm_vcpu_get_esr(vcpu));
}
static bool nvhe_hyp_panic_host_s2_disabled(void)
{
return !is_protected_kvm_enabled() ||
IS_ENABLED(CONFIG_PKVM_DISABLE_STAGE2_ON_PANIC);
}
static void print_nvhe_hyp_panic(const char *name, u64 panic_addr)
{
kvm_err("nVHE hyp %s at: [<%016llx>] %pB!\n", name, panic_addr,
(void *)(panic_addr + kaslr_offset()));
/* Kallsyms might not be mapped in the host stage-2 */
if (nvhe_hyp_panic_host_s2_disabled())
kvm_err("nVHE hyp %s at: [<%016llx>] %pB!\n", name, panic_addr,
(void *)(panic_addr + kaslr_offset()));
else
kvm_err("nVHE hyp %s at: %016llx!\n", name, panic_addr);
}
static void kvm_nvhe_report_cfi_failure(u64 panic_addr)
@@ -538,8 +571,7 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr,
unsigned int line = 0;
/* All hyp bugs, including warnings, are treated as fatal. */
if (!is_protected_kvm_enabled() ||
IS_ENABLED(CONFIG_PKVM_DISABLE_STAGE2_ON_PANIC)) {
if (nvhe_hyp_panic_host_s2_disabled()) {
struct bug_entry *bug = find_bug(elr_in_kimg);
if (bug)
arch/arm64/kvm/hyp/exception.c
+9 -25
View File
@@ -20,22 +20,6 @@
#error Hypervisor code only!
#endif
static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
{
if (has_vhe())
return vcpu_read_sys_reg(vcpu, reg);
return __vcpu_sys_reg(vcpu, reg);
}
static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
{
if (has_vhe())
vcpu_write_sys_reg(vcpu, val, reg);
else
__vcpu_assign_sys_reg(vcpu, reg, val);
}
static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
u64 val)
{
@@ -101,14 +85,14 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
switch (target_mode) {
case PSR_MODE_EL1h:
vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL1);
sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
vbar = vcpu_read_sys_reg(vcpu, VBAR_EL1);
sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1);
vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
break;
case PSR_MODE_EL2h:
vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
vbar = vcpu_read_sys_reg(vcpu, VBAR_EL2);
sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL2);
vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
break;
default:
/* Don't do that */
@@ -185,7 +169,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
*/
static unsigned long get_except32_cpsr(struct kvm_vcpu *vcpu, u32 mode)
{
u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
u32 sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1);
unsigned long old, new;
old = *vcpu_cpsr(vcpu);
@@ -281,7 +265,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
{
unsigned long spsr = *vcpu_cpsr(vcpu);
bool is_thumb = (spsr & PSR_AA32_T_BIT);
u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
u32 sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1);
u32 return_address;
*vcpu_cpsr(vcpu) = get_except32_cpsr(vcpu, mode);
@@ -305,7 +289,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset)
if (sctlr & (1 << 13))
vect_offset += 0xffff0000;
else /* always have security exceptions */
vect_offset += __vcpu_read_sys_reg(vcpu, VBAR_EL1);
vect_offset += vcpu_read_sys_reg(vcpu, VBAR_EL1);
*vcpu_pc(vcpu) = vect_offset;
}
arch/arm64/kvm/hyp/include/hyp/switch.h
+24 -18
View File
@@ -108,9 +108,10 @@ static inline void __activate_cptr_traps_vhe(struct kvm_vcpu *vcpu)
* The architecture is a bit crap (what a surprise): an EL2 guest
* writing to CPTR_EL2 via CPACR_EL1 can't set any of TCPAC or TTA,
* as they are RES0 in the guest's view. To work around it, trap the
* sucker using the very same bit it can't set...
* sucker using the very same bit it can't set. FEAT_NV2p1 fixes it.
*/
if (vcpu_el2_e2h_is_set(vcpu) && is_hyp_ctxt(vcpu))
if (!cpus_have_final_cap(ARM64_HAS_NV2P1) &&
vcpu_el2_e2h_is_set(vcpu) && is_hyp_ctxt(vcpu))
val |= CPTR_EL2_TCPAC;
/*
@@ -325,6 +326,24 @@ static inline void __deactivate_traps_mpam(void)
write_sysreg_s(MPAMHCR_HOST_FLAGS, SYS_MPAMHCR_EL2);
}
/*
* Just like for HCR_EL2, we can't let the guest mess with some of the
* basics we rely on in HCRX_EL2. However, the major difference is that
* HCRX_EL2 only affects EL1, and never EL2 (sudden outburst of sanity, I
* guess). So it is always the guest inflicting it on its own guestx.
*
* Things we don't want to let the guest control are:
*
* - TMEA: That's for us to decide how an SEA is routed, not the guest.
*
* - PTTWI: Similarly, it is for us to decide whether Reduced Coherency for
* the PTW is a thing. It really isn't.
*
* - EnIDCP128: We don't allow IMPDEF sysregs -- full stop.
*/
#define NV_HCRX_GUEST_EXCLUDE (HCRX_EL2_TMEA | HCRX_EL2_PTTWI | \
HCRX_EL2_EnIDCP128)
static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *hctxt = host_data_ptr(host_ctxt);
@@ -350,8 +369,8 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
u64 hcrx = vcpu->arch.hcrx_el2;
if (is_nested_ctxt(vcpu)) {
u64 val = __vcpu_sys_reg(vcpu, HCRX_EL2);
hcrx |= val & __HCRX_EL2_MASK;
hcrx &= ~(~val & __HCRX_EL2_nMASK);
hcrx |= (val & ~NV_HCRX_GUEST_EXCLUDE);
hcrx &= ~(~val & ~NV_HCRX_GUEST_EXCLUDE);
}
ctxt_sys_reg(hctxt, HCRX_EL2) = read_sysreg_s(SYS_HCRX_EL2);
@@ -706,22 +725,9 @@ static inline bool handle_tx2_tvm(struct kvm_vcpu *vcpu)
return true;
}
/* Open-coded version of timer_get_offset() to allow for kern_hyp_va() */
static inline u64 hyp_timer_get_offset(struct arch_timer_context *ctxt)
{
u64 offset = 0;
if (ctxt->offset.vm_offset)
offset += *kern_hyp_va(ctxt->offset.vm_offset);
if (ctxt->offset.vcpu_offset)
offset += *kern_hyp_va(ctxt->offset.vcpu_offset);
return offset;
}
static inline u64 compute_counter_value(struct arch_timer_context *ctxt)
{
return arch_timer_read_cntpct_el0() - hyp_timer_get_offset(ctxt);
return arch_timer_read_cntpct_el0() - timer_get_offset(ctxt);
}
static bool kvm_handle_cntxct(struct kvm_vcpu *vcpu)
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
+11
View File
@@ -172,6 +172,10 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
if (ctxt_has_sctlr2(ctxt))
ctxt_sys_reg(ctxt, SCTLR2_EL1) = read_sysreg_el1(SYS_SCTLR2);
/* Retrieve L2's HCR_EL2, and save it for future use */
if (is_nested_nv3_ctxt(ctxt_to_vcpu(ctxt)))
ctxt_sys_reg(ctxt, NVHCR_EL2) = read_sysreg_s(SYS_NVHCR_EL2);
}
static inline void __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
@@ -285,6 +289,13 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt,
if (ctxt_has_sctlr2(ctxt))
write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR2_EL1), SYS_SCTLR2);
/*
* Publish the L2 view of HCR_EL2 to the HW if L1 is using NV3.
* Otherwise, the data is already in place in the L1's own VNCR.
*/
if (is_nested_nv3_ctxt(ctxt_to_vcpu(ctxt)))
write_sysreg_s(ctxt_sys_reg(ctxt, NVHCR_EL2), SYS_NVHCR_EL2);
}
/* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */
arch/arm64/kvm/hyp/include/nvhe/clock.h
+4 -4
View File
@@ -6,11 +6,11 @@
#include <asm/kvm_hyp.h>
#ifdef CONFIG_NVHE_EL2_TRACING
void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc);
u64 trace_clock(void);
void trace_hyp_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc);
u64 trace_hyp_clock(void);
#else
static inline void
trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) { }
static inline u64 trace_clock(void) { return 0; }
trace_hyp_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) { }
static inline u64 trace_hyp_clock(void) { return 0; }
#endif
#endif
arch/arm64/kvm/hyp/nvhe/clock.c
+2 -2
View File
@@ -30,7 +30,7 @@ static u64 __clock_mult_uint128(u64 cyc, u32 mult, u32 shift)
}
/* Does not guarantee no reader on the modified bank. */
void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc)
void trace_hyp_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc)
{
struct clock_data *clock = &trace_clock_data;
u64 bank = clock->cur ^ 1;
@@ -48,7 +48,7 @@ void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc)
}
/* Use untrusted host data */
u64 trace_clock(void)
u64 trace_hyp_clock(void)
{
struct clock_data *clock = &trace_clock_data;
u64 bank = smp_load_acquire(&clock->cur);
arch/arm64/kvm/hyp/nvhe/hyp-init.S
+7 -4
View File
@@ -106,16 +106,19 @@ SYM_CODE_START_LOCAL(___kvm_hyp_init)
and x2, x1, x2
cbz x2, 1f
// hVHE: Replay the EL2 setup to account for the E2H bit
// TPIDR_EL2 is used to preserve x0 across the macro maze...
/*
* hVHE: Replay the EL2 setup to account for the E2H bit
* TPIDR_EL2 and FAR_EL2 are used to preserve x0 and LR across
* the macro maze...
*/
isb
msr tpidr_el2, x0
str lr, [x0, #NVHE_INIT_TMP]
msr far_el2, lr
bl __kvm_init_el2_state
mrs lr, far_el2
mrs x0, tpidr_el2
ldr lr, [x0, #NVHE_INIT_TMP]
1:
ldr x1, [x0, #NVHE_INIT_TPIDR_EL2]
arch/arm64/kvm/hyp/nvhe/hyp-main.c
+233 -38
View File
@@ -7,6 +7,8 @@
#include <hyp/adjust_pc.h>
#include <hyp/switch.h>
#include <linux/irqchip/arm-gic-v3.h>
#include <asm/pgtable-types.h>
#include <asm/kvm_asm.h>
#include <asm/kvm_emulate.h>
@@ -102,16 +104,103 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu)
*host_data_ptr(fp_owner) = FP_STATE_HOST_OWNED;
}
static void flush_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;
unsigned int used_lrs, i;
host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
used_lrs = host_cpu_if->used_lrs;
used_lrs = min(used_lrs, hyp_gicv3_nr_lr);
hyp_cpu_if->vgic_hcr = host_cpu_if->vgic_hcr;
/* Should be a one-off */
hyp_cpu_if->vgic_sre = (ICC_SRE_EL1_DIB |
ICC_SRE_EL1_DFB |
ICC_SRE_EL1_SRE);
hyp_cpu_if->used_lrs = used_lrs;
for (i = 0; i < used_lrs; i++)
hyp_cpu_if->vgic_lr[i] = host_cpu_if->vgic_lr[i];
}
static void sync_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if;
unsigned int i;
host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr;
host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;
for (i = 0; i < hyp_cpu_if->used_lrs; i++)
host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
}
static void __copy_vcpu_state(const struct kvm_vcpu *from_vcpu,
struct kvm_vcpu *to_vcpu)
{
int i;
to_vcpu->arch.ctxt.regs = from_vcpu->arch.ctxt.regs;
to_vcpu->arch.ctxt.spsr_abt = from_vcpu->arch.ctxt.spsr_abt;
to_vcpu->arch.ctxt.spsr_und = from_vcpu->arch.ctxt.spsr_und;
to_vcpu->arch.ctxt.spsr_irq = from_vcpu->arch.ctxt.spsr_irq;
to_vcpu->arch.ctxt.spsr_fiq = from_vcpu->arch.ctxt.spsr_fiq;
to_vcpu->arch.ctxt.fp_regs = from_vcpu->arch.ctxt.fp_regs;
/*
* Copy the sysregs, but don't mess with the timer state which
* is directly handled by EL1 and is expected to be preserved.
* enum vcpu_sysreg is sparse: VNCR-mapped registers take values
* derived from their VNCR page offset, so the timer registers do
* not form a contiguous numeric range and must be skipped by name.
*/
for (i = 1; i < NR_SYS_REGS; i++) {
switch (i) {
case CNTVOFF_EL2:
case CNTV_CVAL_EL0:
case CNTV_CTL_EL0:
case CNTP_CVAL_EL0:
case CNTP_CTL_EL0:
continue;
}
to_vcpu->arch.ctxt.sys_regs[i] = from_vcpu->arch.ctxt.sys_regs[i];
}
}
static void sync_hyp_vcpu_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
__copy_vcpu_state(&hyp_vcpu->vcpu, hyp_vcpu->host_vcpu);
}
static void flush_hyp_vcpu_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
__copy_vcpu_state(hyp_vcpu->host_vcpu, &hyp_vcpu->vcpu);
}
static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
hyp_vcpu->vcpu.arch.debug_owner = host_vcpu->arch.debug_owner;
if (kvm_guest_owns_debug_regs(&hyp_vcpu->vcpu))
if (kvm_guest_owns_debug_regs(&hyp_vcpu->vcpu)) {
hyp_vcpu->vcpu.arch.vcpu_debug_state = host_vcpu->arch.vcpu_debug_state;
else if (kvm_host_owns_debug_regs(&hyp_vcpu->vcpu))
} else if (kvm_host_owns_debug_regs(&hyp_vcpu->vcpu)) {
hyp_vcpu->vcpu.arch.external_debug_state = host_vcpu->arch.external_debug_state;
/*
* The world switch loads MDSCR_EL1 from external_mdscr_el1
* (ctxt_mdscr_el1()).
*/
hyp_vcpu->vcpu.arch.external_mdscr_el1 = host_vcpu->arch.external_mdscr_el1;
}
}
static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu)
@@ -131,7 +220,17 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
fpsimd_sve_flush();
flush_debug_state(hyp_vcpu);
hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt;
/*
* If we deal with a non-protected guest and the state is potentially
* dirty (from a host perspective), copy the state back into the hyp
* vcpu.
*/
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
if (vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY))
flush_hyp_vcpu_state(hyp_vcpu);
} else {
hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt;
}
/* __hyp_running_vcpu must be NULL in a guest context. */
hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu = NULL;
@@ -150,13 +249,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2;
hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3 = host_vcpu->arch.vgic_cpu.vgic_v3;
/* Bound used_lrs by the number of implemented list registers. */
hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs =
min_t(unsigned int,
hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs,
hyp_gicv3_nr_lr);
flush_hyp_vgic_state(hyp_vcpu);
hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid;
}
@@ -164,25 +257,26 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu)
{
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
struct vgic_v3_cpu_if *hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
struct vgic_v3_cpu_if *host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
unsigned int i;
fpsimd_sve_sync(&hyp_vcpu->vcpu);
sync_debug_state(hyp_vcpu);
host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt;
host_vcpu->arch.hcr_el2 = hyp_vcpu->vcpu.arch.hcr_el2;
if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt;
} else {
/*
* PC feeds trace_kvm_exit(), PSTATE.SS the host software-step
* machine, and both run before the next on-demand ctxt sync.
*/
host_vcpu->arch.ctxt.regs.pc = hyp_vcpu->vcpu.arch.ctxt.regs.pc;
host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate;
}
host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault;
host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags;
host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr;
host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;
for (i = 0; i < hyp_cpu_if->used_lrs; ++i)
host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i];
sync_hyp_vgic_state(hyp_vcpu);
}
static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt)
@@ -210,18 +304,78 @@ static void handle___pkvm_vcpu_put(struct kvm_cpu_context *host_ctxt)
{
struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
if (hyp_vcpu)
if (hyp_vcpu) {
struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu;
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu) &&
!vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY)) {
sync_hyp_vcpu_state(hyp_vcpu);
}
pkvm_put_hyp_vcpu(hyp_vcpu);
}
}
static void handle___pkvm_vcpu_sync_state(struct kvm_cpu_context *host_ctxt)
{
struct pkvm_hyp_vcpu *hyp_vcpu;
hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
if (!hyp_vcpu || pkvm_hyp_vcpu_is_protected(hyp_vcpu))
return;
sync_hyp_vcpu_state(hyp_vcpu);
}
static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg,
struct pkvm_hyp_vcpu **hyp_vcpup)
{
struct kvm_vcpu *host_vcpu = kern_hyp_va(arg);
struct pkvm_hyp_vcpu *hyp_vcpu = NULL;
if (unlikely(is_protected_kvm_enabled())) {
hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
if (!hyp_vcpu || hyp_vcpu->host_vcpu != host_vcpu) {
hyp_vcpu = NULL;
host_vcpu = NULL;
}
}
*hyp_vcpup = hyp_vcpu;
return host_vcpu;
}
#define get_host_hyp_vcpus(ctxt, regnr, hyp_vcpup) \
({ \
DECLARE_REG(struct kvm_vcpu *, __vcpu, ctxt, regnr); \
__get_host_hyp_vcpus(__vcpu, hyp_vcpup); \
})
#define get_host_hyp_vcpus_from_vgic_v3_cpu_if(ctxt, regnr, hyp_vcpup) \
({ \
DECLARE_REG(struct vgic_v3_cpu_if *, cif, ctxt, regnr);\
struct kvm_vcpu *__vcpu = container_of(cif, \
struct kvm_vcpu, \
arch.vgic_cpu.vgic_v3); \
\
__get_host_hyp_vcpus(__vcpu, hyp_vcpup); \
})
static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
{
DECLARE_REG(struct kvm_vcpu *, host_vcpu, host_ctxt, 1);
struct pkvm_hyp_vcpu *hyp_vcpu;
struct kvm_vcpu *host_vcpu;
int ret;
if (unlikely(is_protected_kvm_enabled())) {
struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu();
host_vcpu = get_host_hyp_vcpus(host_ctxt, 1, &hyp_vcpu);
if (!host_vcpu) {
ret = -EINVAL;
goto out;
}
if (unlikely(hyp_vcpu)) {
/*
* KVM (and pKVM) doesn't support SME guests for now, and
* ensures that SME features aren't enabled in pstate when
@@ -233,23 +387,16 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
goto out;
}
if (!hyp_vcpu) {
ret = -EINVAL;
goto out;
}
flush_hyp_vcpu(hyp_vcpu);
ret = __kvm_vcpu_run(&hyp_vcpu->vcpu);
sync_hyp_vcpu(hyp_vcpu);
} else {
struct kvm_vcpu *vcpu = kern_hyp_va(host_vcpu);
/* The host is fully trusted, run its vCPU directly. */
fpsimd_lazy_switch_to_guest(vcpu);
ret = __kvm_vcpu_run(vcpu);
fpsimd_lazy_switch_to_host(vcpu);
fpsimd_lazy_switch_to_guest(host_vcpu);
ret = __kvm_vcpu_run(host_vcpu);
fpsimd_lazy_switch_to_host(host_vcpu);
}
out:
cpu_reg(host_ctxt, 1) = ret;
@@ -484,16 +631,63 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt)
static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
{
DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
struct pkvm_hyp_vcpu *hyp_vcpu;
struct kvm_vcpu *host_vcpu;
__vgic_v3_save_aprs(kern_hyp_va(cpu_if));
host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1,
&hyp_vcpu);
if (!host_vcpu)
return;
if (unlikely(hyp_vcpu)) {
struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if;
int i;
hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
__vgic_v3_save_aprs(hyp_cpu_if);
host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;
for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) {
host_cpu_if->vgic_ap0r[i] = hyp_cpu_if->vgic_ap0r[i];
host_cpu_if->vgic_ap1r[i] = hyp_cpu_if->vgic_ap1r[i];
}
} else {
__vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
}
}
static void handle___vgic_v3_restore_vmcr_aprs(struct kvm_cpu_context *host_ctxt)
{
DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
struct pkvm_hyp_vcpu *hyp_vcpu;
struct kvm_vcpu *host_vcpu;
__vgic_v3_restore_vmcr_aprs(kern_hyp_va(cpu_if));
host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1,
&hyp_vcpu);
if (!host_vcpu)
return;
if (unlikely(hyp_vcpu)) {
struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if;
int i;
hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3;
host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3;
hyp_cpu_if->vgic_vmcr = host_cpu_if->vgic_vmcr;
/* Should be a one-off */
hyp_cpu_if->vgic_sre = (ICC_SRE_EL1_DIB |
ICC_SRE_EL1_DFB |
ICC_SRE_EL1_SRE);
for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) {
hyp_cpu_if->vgic_ap0r[i] = host_cpu_if->vgic_ap0r[i];
hyp_cpu_if->vgic_ap1r[i] = host_cpu_if->vgic_ap1r[i];
}
__vgic_v3_restore_vmcr_aprs(hyp_cpu_if);
} else {
__vgic_v3_restore_vmcr_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3);
}
}
static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt)
@@ -761,6 +955,7 @@ static const hcall_t host_hcall[] = {
HANDLE_FUNC(__pkvm_finalize_teardown_vm),
HANDLE_FUNC(__pkvm_vcpu_load),
HANDLE_FUNC(__pkvm_vcpu_put),
HANDLE_FUNC(__pkvm_vcpu_sync_state),
HANDLE_FUNC(__pkvm_tlb_flush_vmid),
};
arch/arm64/kvm/hyp/nvhe/mem_protect.c
+7
View File
@@ -261,11 +261,18 @@ static void __apply_guest_page(void *va, size_t size,
static void clean_dcache_guest_page(void *va, size_t size)
{
/* See comment in __clean_dcache_guest_page() */
if (cpus_have_final_cap(ARM64_HAS_STAGE2_FWB))
return;
__apply_guest_page(va, size, __clean_dcache_guest_page);
}
static void invalidate_icache_guest_page(void *va, size_t size)
{
if (alternative_has_cap_unlikely(ARM64_HAS_CACHE_DIC))
return;
__apply_guest_page(va, size, __invalidate_icache_guest_page);
}
arch/arm64/kvm/hyp/nvhe/pkvm.c
+14 -1
View File
@@ -433,7 +433,6 @@ static void init_pkvm_hyp_vm(struct kvm *host_kvm, struct pkvm_hyp_vm *hyp_vm,
hyp_vm->host_kvm = host_kvm;
hyp_vm->kvm.created_vcpus = nr_vcpus;
hyp_vm->kvm.arch.pkvm.is_protected = READ_ONCE(host_kvm->arch.pkvm.is_protected);
hyp_vm->kvm.arch.pkvm.is_created = true;
hyp_vm->kvm.arch.flags = 0;
pkvm_init_features_from_host(hyp_vm, host_kvm);
@@ -529,6 +528,20 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu,
hyp_vcpu->vcpu.arch.cflags = READ_ONCE(host_vcpu->arch.cflags);
hyp_vcpu->vcpu.arch.mp_state.mp_state = KVM_MP_STATE_STOPPED;
if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) {
/*
* Timer offsets are pointing to the untrusted KVM copy,
* which is pinned in __pkvm_init_vm() for the VM life time.
* It is worth noting that hyp_vm->host_kvm points to an EL2
* linear map address and timer_get_offset() will use
* kern_hyp_va() which is safe as it is idempotent.
*/
vcpu_vtimer(&hyp_vcpu->vcpu)->offset.vm_offset =
&hyp_vm->host_kvm->arch.timer_data.voffset;
vcpu_ptimer(&hyp_vcpu->vcpu)->offset.vm_offset =
&hyp_vm->host_kvm->arch.timer_data.poffset;
}
ret = pkvm_vcpu_init_sysregs(hyp_vcpu);
if (ret)
goto done;
arch/arm64/kvm/hyp/nvhe/sys_regs.c
+3 -2
View File
@@ -257,10 +257,11 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr)
*vcpu_cpsr(vcpu) = read_sysreg_el2(SYS_SPSR);
/*
* Make sure we have the latest update to VBAR_EL1, as pKVM
* handles traps very early, before sysregs are resync'ed
* Sync VBAR_EL1 and SCTLR_EL1, both read by enter_exception64(),
* as pKVM handles traps before sysregs are resync'ed.
*/
__vcpu_assign_sys_reg(vcpu, VBAR_EL1, read_sysreg_el1(SYS_VBAR));
__vcpu_assign_sys_reg(vcpu, SCTLR_EL1, read_sysreg_el1(SYS_SCTLR));
kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC);
arch/arm64/kvm/hyp/nvhe/timer-sr.c
+5 -5
View File
@@ -45,11 +45,11 @@ void __timer_enable_traps(struct kvm_vcpu *vcpu)
/*
* Disallow physical timer access for the guest
* Physical counter access is allowed if no offset is enforced
* or running protected (we don't offset anything in this case).
* or running a protected VM (we don't offset anything in this case).
*/
clr = CNTHCTL_EL1PCEN;
if (is_protected_kvm_enabled() ||
!kern_hyp_va(vcpu->kvm)->arch.timer_data.poffset)
if (vcpu_is_protected(vcpu) ||
!timer_get_offset(vcpu_ptimer(vcpu)))
set |= CNTHCTL_EL1PCTEN;
else
clr |= CNTHCTL_EL1PCTEN;
@@ -61,9 +61,9 @@ void __timer_enable_traps(struct kvm_vcpu *vcpu)
/*
* Trap the virtual counter/timer if we have a broken cntvoff
* implementation.
* implementation and non zero offset as in timer_set_traps()
*/
if (has_broken_cntvoff())
if (has_broken_cntvoff() && timer_get_offset(vcpu_vtimer(vcpu)))
set |= CNTHCTL_EL1TVT | CNTHCTL_EL1TVCT;
sysreg_clear_set(cnthctl_el2, clr, set);
arch/arm64/kvm/hyp/nvhe/trace.c
+2 -2
View File
@@ -35,7 +35,7 @@ static bool hyp_trace_buffer_loaded(struct hyp_trace_buffer *trace_buffer)
void *tracing_reserve_entry(unsigned long length)
{
return simple_ring_buffer_reserve(this_cpu_ptr(trace_buffer.simple_rbs), length,
trace_clock());
trace_hyp_clock());
}
void tracing_commit_entry(void)
@@ -290,7 +290,7 @@ void __tracing_update_clock(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc)
}
/* ...we can now override the old one and swap. */
trace_clock_update(mult, shift, epoch_ns, epoch_cyc);
trace_hyp_clock_update(mult, shift, epoch_ns, epoch_cyc);
}
int __tracing_reset(unsigned int cpu)
arch/arm64/kvm/hyp/vgic-v3-sr.c
+13 -30
View File
@@ -16,9 +16,9 @@
#include "../../vgic/vgic.h"
#define vtr_to_max_lr_idx(v) ((v) & 0xf)
#define vtr_to_nr_pre_bits(v) ((((u32)(v) >> 26) & 7) + 1)
#define vtr_to_nr_apr_regs(v) (1 << (vtr_to_nr_pre_bits(v) - 5))
#define vtr_to_max_lr_idx(v) FIELD_GET(ICH_VTR_EL2_ListRegs, (v))
#define vtr_to_nr_pre_bits(v) (FIELD_GET(ICH_VTR_EL2_PREbits, (v)) + 1)
#define vtr_to_nr_apr_regs(v) BIT(vtr_to_nr_pre_bits(v) - 5)
u64 __gic_v3_get_lr(unsigned int lr)
{
@@ -367,7 +367,7 @@ void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if)
u64 val;
u32 nr_pre_bits;
val = read_gicreg(ICH_VTR_EL2);
val = vgic_ich_vtr();
nr_pre_bits = vtr_to_nr_pre_bits(val);
switch (nr_pre_bits) {
@@ -400,7 +400,7 @@ static void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if)
u64 val;
u32 nr_pre_bits;
val = read_gicreg(ICH_VTR_EL2);
val = vgic_ich_vtr();
nr_pre_bits = vtr_to_nr_pre_bits(val);
switch (nr_pre_bits) {
@@ -430,33 +430,19 @@ static void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if)
void __vgic_v3_init_lrs(void)
{
int max_lr_idx = vtr_to_max_lr_idx(read_gicreg(ICH_VTR_EL2));
int max_lr_idx = vtr_to_max_lr_idx(vgic_ich_vtr());
int i;
for (i = 0; i <= max_lr_idx; i++)
__gic_v3_set_lr(0, i);
}
/*
* Return the GIC CPU configuration:
* - [31:0] ICH_VTR_EL2
* - [62:32] RES0
* - [63] MMIO (GICv2) capable
*/
u64 __vgic_v3_get_gic_config(void)
/* Return true if GICv3 is MMIO (GICv2) capable, false otherwise */
bool __vgic_v3_get_gic_config(void)
{
u64 val, sre;
unsigned long flags = 0;
/*
* In compat mode, we cannot access ICC_SRE_EL1 at any EL
* other than EL1 itself; just return the
* ICH_VTR_EL2. ICC_IDR0_EL1 is only implemented on a GICv5
* system, so we first check if we have GICv5 support.
*/
if (cpus_have_final_cap(ARM64_HAS_GICV5_CPUIF))
return read_gicreg(ICH_VTR_EL2);
sre = read_gicreg(ICC_SRE_EL1);
/*
* To check whether we have a MMIO-based (GICv2 compatible)
@@ -497,10 +483,7 @@ u64 __vgic_v3_get_gic_config(void)
isb();
}
val = (val & ICC_SRE_EL1_SRE) ? 0 : (1ULL << 63);
val |= read_gicreg(ICH_VTR_EL2);
return val;
return !(val & ICC_SRE_EL1_SRE);
}
static void __vgic_v3_compat_mode_enable(void)
@@ -540,7 +523,7 @@ void __vgic_v3_restore_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if)
static int __vgic_v3_bpr_min(void)
{
/* See Pseudocode for VPriorityGroup */
return 8 - vtr_to_nr_pre_bits(read_gicreg(ICH_VTR_EL2));
return 8 - vtr_to_nr_pre_bits(vgic_ich_vtr());
}
static int __vgic_v3_get_group(struct kvm_vcpu *vcpu)
@@ -614,7 +597,7 @@ static int __vgic_v3_find_active_lr(struct kvm_vcpu *vcpu, int intid,
static int __vgic_v3_get_highest_active_priority(void)
{
u8 nr_apr_regs = vtr_to_nr_apr_regs(read_gicreg(ICH_VTR_EL2));
u8 nr_apr_regs = vtr_to_nr_apr_regs(vgic_ich_vtr());
u32 hap = 0;
int i;
@@ -707,7 +690,7 @@ static void __vgic_v3_set_active_priority(u8 pri, u32 vmcr, int grp)
static int __vgic_v3_clear_highest_active_priority(void)
{
u8 nr_apr_regs = vtr_to_nr_apr_regs(read_gicreg(ICH_VTR_EL2));
u8 nr_apr_regs = vtr_to_nr_apr_regs(vgic_ich_vtr());
u32 hap = 0;
int i;
@@ -1039,7 +1022,7 @@ static void __vgic_v3_read_ctlr(struct kvm_vcpu *vcpu, u32 vmcr, int rt)
{
u32 vtr, val;
vtr = read_gicreg(ICH_VTR_EL2);
vtr = vgic_ich_vtr();
/* PRIbits */
val = ((vtr >> 29) & 7) << ICC_CTLR_EL1_PRI_BITS_SHIFT;
/* IDbits */
arch/arm64/kvm/hyp/vhe/switch.c
+47 -15
View File
@@ -70,6 +70,12 @@ static u64 __compute_hcr(struct kvm_vcpu *vcpu)
if (!vcpu_el2_e2h_is_set(vcpu))
hcr |= HCR_NV1;
/* Publish the guest's view of HCR_EL2 to the HW */
if (cpus_have_final_cap(ARM64_HAS_NV3) && vcpu_el2_e2h_is_set(vcpu))
write_sysreg_s(__vcpu_sys_reg(vcpu, HCR_EL2), SYS_NVHCR_EL2);
else
__vcpu_assign_sys_reg(vcpu, NVHCR_EL2, __vcpu_sys_reg(vcpu, HCR_EL2));
/*
* Nothing in HCR_EL2 should impact running in hypervisor
* context, apart from bits we have defined as RESx (E2H,
@@ -339,18 +345,24 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code)
u64 esr = kvm_vcpu_get_esr(vcpu);
u64 spsr, elr, mode;
/* With NV3, the fast path is handled in HW */
if (cpus_have_final_cap(ARM64_HAS_NV3) && vcpu_el2_e2h_is_set(vcpu))
return false;
/*
* Going through the whole put/load motions is a waste of time
* if this is a VHE guest hypervisor returning to its own
* userspace, or the hypervisor performing a local exception
* return. No need to save/restore registers, no need to
* switch S2 MMU. Just do the canonical ERET.
* switch S2 MMU. Just do the canonical ERET unless we are in
* nested context.
*
* Unless the trap has to be forwarded further down the line,
* of course...
* Note that this is made possible because KVM itself never traps
* ERET when running an L2. The consequence is that any ERET trap is
* the result of HCR_EL2 or HFGITR_EL2 programming by L1 for its own
* guest, and the exception must be forwarded to L1.
*/
if ((__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) ||
(__vcpu_sys_reg(vcpu, HFGITR_EL2) & HFGITR_EL2_ERET))
if (is_nested_ctxt(vcpu))
return false;
spsr = read_sysreg_el1(SYS_SPSR);
@@ -424,11 +436,15 @@ static bool kvm_hyp_handle_tlbi_el2(struct kvm_vcpu *vcpu, u64 *exit_code)
return false;
/*
* If we have to check for any VNCR mapping being invalidated,
* go back to the slow path for further processing.
* If we have to check for any VNCR TLB being invalidated, go back
* to the slow path for further processing.
*
* The synchronisation betweem TLBI and walk is provided by the
* speculative increment of the TLB counter on walk, and the
* invalidation counter. Yes, this is fiddly.
*/
if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu) &&
atomic_read(&vcpu->kvm->arch.vncr_map_count))
atomic_read(&vcpu->kvm->arch.vncr_tlb_count))
return false;
__kvm_skip_instr(vcpu);
@@ -441,6 +457,9 @@ static bool kvm_hyp_handle_cpacr_el1(struct kvm_vcpu *vcpu, u64 *exit_code)
u64 esr = kvm_vcpu_get_esr(vcpu);
int rt;
if (cpus_have_final_cap(ARM64_HAS_NV2P1))
return false;
if (!is_hyp_ctxt(vcpu) || esr_sys64_to_sysreg(esr) != SYS_CPACR_EL1)
return false;
@@ -534,19 +553,17 @@ static const exit_handler_fn hyp_exit_handlers[] = {
[0x3F] = kvm_hyp_handle_impdef,
};
static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
static void fixup_nv_guest_exit(struct kvm_vcpu *vcpu)
{
synchronize_vcpu_pstate(vcpu);
/*
* If we were in HYP context on entry, adjust the PSTATE view
* so that the usual helpers work correctly. This enforces our
* invariant that the guest's HYP context status is preserved
* across a run.
*/
if (vcpu_has_nv(vcpu) &&
unlikely(host_data_test_flag(VCPU_IN_HYP_CONTEXT))) {
if (unlikely(host_data_test_flag(VCPU_IN_HYP_CONTEXT))) {
u64 mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
u64 hcr;
switch (mode) {
case PSR_MODE_EL1t:
@@ -559,11 +576,26 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
*vcpu_cpsr(vcpu) &= ~(PSR_MODE_MASK | PSR_MODE32_BIT);
*vcpu_cpsr(vcpu) |= mode;
/* Publish the latest HCR_EL2 to the emulation */
hcr = (cpus_have_final_cap(ARM64_HAS_NV3) &&
vcpu_el2_e2h_is_set(vcpu)) ?
read_sysreg_s(SYS_NVHCR_EL2) :
__vcpu_sys_reg(vcpu, NVHCR_EL2);
__vcpu_assign_sys_reg(vcpu, HCR_EL2, hcr);
}
/* Apply extreme paranoia! */
BUG_ON(vcpu_has_nv(vcpu) &&
!!host_data_test_flag(VCPU_IN_HYP_CONTEXT) != is_hyp_ctxt(vcpu));
BUG_ON(!!host_data_test_flag(VCPU_IN_HYP_CONTEXT) != is_hyp_ctxt(vcpu));
}
static bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
{
synchronize_vcpu_pstate(vcpu);
if (vcpu_has_nv(vcpu))
fixup_nv_guest_exit(vcpu);
return __fixup_guest_exit(vcpu, exit_code, hyp_exit_handlers);
}
arch/arm64/kvm/hyp/vhe/sysreg-sr.c
+15 -6
View File
@@ -42,10 +42,12 @@ static void __sysreg_save_vel2_state(struct kvm_vcpu *vcpu)
u64 val;
/*
* We don't save CPTR_EL2, as accesses to CPACR_EL1
* are always trapped, ensuring that the in-memory
* copy is always up-to-date. A small blessing...
* Without FEAT_NV2p1, we don't save CPTR_EL2, as accesses
* to CPACR_EL1 are always trapped, ensuring that the
* in-memory copy is always up-to-date. A small blessing...
*/
if (cpus_have_final_cap(ARM64_HAS_NV2P1))
__vcpu_assign_sys_reg(vcpu, CPTR_EL2, read_sysreg_el1(SYS_CPACR));
__vcpu_assign_sys_reg(vcpu, SCTLR_EL2, read_sysreg_el1(SYS_SCTLR));
__vcpu_assign_sys_reg(vcpu, TTBR0_EL2, read_sysreg_el1(SYS_TTBR0));
__vcpu_assign_sys_reg(vcpu, TTBR1_EL2, read_sysreg_el1(SYS_TTBR1));
@@ -67,11 +69,18 @@ static void __sysreg_save_vel2_state(struct kvm_vcpu *vcpu)
* The EL1 view of CNTKCTL_EL1 has a bunch of RES0 bits where
* the interesting CNTHCTL_EL2 bits live. So preserve these
* bits when reading back the guest-visible value.
*
* While NV2p1 fixes some of that, it makes CNTHCTL_EL2.ECV
* even more broken than it already was with NV2.
*/
val = read_sysreg_el1(SYS_CNTKCTL);
val &= CNTKCTL_VALID_BITS;
__vcpu_rmw_sys_reg(vcpu, CNTHCTL_EL2, &=, ~CNTKCTL_VALID_BITS);
__vcpu_rmw_sys_reg(vcpu, CNTHCTL_EL2, |=, val);
if (!cpus_have_final_cap(ARM64_HAS_NV2P1)) {
val &= CNTKCTL_VALID_BITS;
__vcpu_rmw_sys_reg(vcpu, CNTHCTL_EL2, &=, ~CNTKCTL_VALID_BITS);
__vcpu_rmw_sys_reg(vcpu, CNTHCTL_EL2, |=, val);
} else {
__vcpu_assign_sys_reg(vcpu, CNTHCTL_EL2, val);
}
}
__vcpu_assign_sys_reg(vcpu, SP_EL2, read_sysreg(sp_el1));
arch/arm64/kvm/mmu.c
+6 -3
View File
@@ -2113,11 +2113,14 @@ static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd)
* Permission faults just need to update the existing leaf entry,
* and so normally don't require allocations from the memcache. The
* only exception to this is when dirty logging is enabled at runtime
* and a write fault needs to collapse a block entry into a table.
* and a fault needs to collapse a block entry into a table.
* Under pKVM a permission fault can also collapse pages into a block,
* which needs a fresh mapping object, and the hypervisor requires the
* min-pages memcache even when the install allocates nothing.
*/
memcache = get_mmu_memcache(s2fd->vcpu);
if (!perm_fault || (memslot_is_logging(s2fd->memslot) &&
kvm_is_write_fault(s2fd->vcpu))) {
if (!perm_fault || memslot_is_logging(s2fd->memslot) ||
is_protected_kvm_enabled()) {
ret = topup_mmu_memcache(s2fd->vcpu, memcache);
if (ret)
return ret;
arch/arm64/kvm/nested.c
+135 -55
View File
@@ -16,6 +16,7 @@
#include <asm/sysreg.h>
#include "sys_regs.h"
#include "vgic/vgic.h"
struct vncr_tlb {
/* The guest's VNCR_EL2 */
@@ -27,7 +28,7 @@ struct vncr_tlb {
bool hpa_writable;
/* -1 when not mapped on a CPU */
int cpu;
atomic_t cpu;
/*
* true if the TLB is valid. Can only be changed with the
@@ -48,7 +49,7 @@ void kvm_init_nested(struct kvm *kvm)
{
kvm->arch.nested_mmus = NULL;
kvm->arch.nested_mmus_size = 0;
atomic_set(&kvm->arch.vncr_map_count, 0);
atomic_set(&kvm->arch.vncr_tlb_count, 0);
}
static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
@@ -506,7 +507,7 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
return ret;
}
static unsigned int ttl_to_size(u8 ttl)
static unsigned int __ttl_to_size(u8 ttl)
{
int level = ttl & 3;
int gran = (ttl >> 2) & 3;
@@ -562,10 +563,22 @@ static unsigned int ttl_to_size(u8 ttl)
return max_size;
}
static u8 pgshift_level_to_ttl(u16 shift, u8 level)
static unsigned int ttl_to_size(u8 ttl)
{
return __ttl_to_size(ttl) ?: SZ_1G;
}
static u8 pgshift_level_to_ttl(u16 shift, s8 level)
{
u8 ttl;
/*
* If we don't have a proper level, fallback to the maximum
* size.
*/
if (level < 0)
return 0;
switch(shift) {
case 12:
ttl = TLBI_TTL_TG_4K;
@@ -676,7 +689,11 @@ unsigned long compute_tlb_inval_range(struct kvm_s2_mmu *mmu, u64 val)
ttl = get_guest_mapping_ttl(mmu, addr);
}
max_size = ttl_to_size(ttl);
/*
* Don't use the default 1GB fallback, as we can adapt to the
* max mapping size we allow at S2.
*/
max_size = __ttl_to_size(ttl);
if (!max_size) {
/* Compute the maximum extent of the invalidation */
@@ -879,18 +896,41 @@ void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu)
}
}
/*
* Unmapping an L1 VNCR can happen concurrently without the mmu lock being
* effective (vcpu_put() vs TLBI handling). The atomic_xchg below ensures
* that only one CPU sets it to -1 while getting a valid CPU number back.
*/
static int unmap_l1_vncr(struct vncr_tlb *vt)
{
int cpu = atomic_xchg_relaxed(&vt->cpu, -1);
if (cpu != -1)
clear_fixmap(vncr_fixmap(cpu));
return cpu;
}
static void this_cpu_reset_vncr_fixmap(struct kvm_vcpu *vcpu)
{
if (!host_data_test_flag(L1_VNCR_MAPPED))
return;
BUG_ON(vcpu->arch.vncr_tlb->cpu != smp_processor_id());
BUG_ON(is_hyp_ctxt(vcpu));
clear_fixmap(vncr_fixmap(vcpu->arch.vncr_tlb->cpu));
vcpu->arch.vncr_tlb->cpu = -1;
/*
* Unconditionally unmap the local VNCR if we have lost the race
* against a concurrent TLBI. Otherwise we could end-up running
* another vcpu with VNCR still mapped if the TLBI thread is
* preempted between the exchange and the clear_fixmap().
*
* Note that we do not care about the TLBI nuking the fixmap behind
* the back of an running vcpu. This will only generate a fault and
* possibly a retranslation.
*/
if (unmap_l1_vncr(vcpu->arch.vncr_tlb) == -1)
clear_fixmap(vncr_fixmap(smp_processor_id()));
host_data_clear_flag(L1_VNCR_MAPPED);
atomic_dec(&vcpu->kvm->arch.vncr_map_count);
}
void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu)
@@ -978,11 +1018,26 @@ u16 get_asid_by_regime(struct kvm_vcpu *vcpu, enum trans_regime regime)
return asid;
}
static void invalidate_vncr(struct vncr_tlb *vt)
static void invalidate_vncr(struct kvm *kvm, struct vncr_tlb *vt)
{
BUG_ON(!vt->valid);
vt->valid = false;
if (vt->cpu != -1)
clear_fixmap(vncr_fixmap(vt->cpu));
unmap_l1_vncr(vt);
atomic_dec(&kvm->arch.vncr_tlb_count);
}
static bool vncr_tlb_intersects(struct vncr_tlb *vt, u64 addr,
u64 scope_start, u64 scope_size)
{
u64 tlb_size, tlb_start, tlb_end, scope_end;
tlb_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift, vt->wr.level));
tlb_start = addr & ~(tlb_size - 1);
tlb_end = tlb_start + tlb_size - 1;
scope_end = scope_start + scope_size - 1;
return !(tlb_end < scope_start || tlb_start > scope_end);
}
/*
@@ -1007,19 +1062,15 @@ static void kvm_invalidate_vncr_ipa(struct kvm *kvm, u64 start, u64 end)
if (!kvm_has_feat(kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY))
return;
kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) {
u64 ipa_start, ipa_end, ipa_size;
ipa_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift,
vt->wr.level));
ipa_start = vt->wr.pa & ~(ipa_size - 1);
ipa_end = ipa_start + ipa_size;
if (ipa_end <= start || ipa_start >= end)
continue;
invalidate_vncr(vt);
}
/*
* Note that invalidating the VNCR on the back of an MMU notifier
* doesn't require messing with the invalidation counter for a
* parallel walk. The notifier itself will have bumped the counter,
* making sure we rewalk.
*/
kvm_for_each_vncr_tlb(i, vcpu, vt, kvm)
if (vncr_tlb_intersects(vt, vt->wr.pa, start, end - start))
invalidate_vncr(kvm, vt);
}
struct s1e2_tlbi_scope {
@@ -1044,29 +1095,29 @@ static void invalidate_vncr_va(struct kvm *kvm,
lockdep_assert_held_write(&kvm->mmu_lock);
/*
* We might be performing a parallel S1 walk, so bump up the
* invalidation counter even in the absence of an actual VNCR TLB
* invalidation, as this could indicate that the guest has gone
* through a BBM sequence.
*/
kvm->mmu_invalidate_seq++;
smp_wmb();
kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) {
u64 va_start, va_end, va_size;
va_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift,
vt->wr.level));
va_start = vt->gva & ~(va_size - 1);
va_end = va_start + va_size;
switch (scope->type) {
case TLBI_ALL:
break;
case TLBI_VA:
if (va_end <= scope->va ||
va_start >= (scope->va + scope->size))
if (!vncr_tlb_intersects(vt, vt->gva, scope->va, scope->size))
continue;
if (vt->wr.nG && vt->wr.asid != scope->asid)
continue;
break;
case TLBI_VAA:
if (va_end <= scope->va ||
va_start >= (scope->va + scope->size))
if (!vncr_tlb_intersects(vt, vt->gva, scope->va, scope->size))
continue;
break;
@@ -1076,7 +1127,7 @@ static void invalidate_vncr_va(struct kvm *kvm,
break;
}
invalidate_vncr(vt);
invalidate_vncr(kvm, vt);
}
}
@@ -1126,8 +1177,6 @@ static void compute_s1_tlbi_range(struct kvm_vcpu *vcpu, u32 inst, u64 val,
case OP_TLBI_VALE1OSNXS:
scope->type = TLBI_VA;
scope->size = ttl_to_size(FIELD_GET(TLBI_TTL_MASK, val));
if (!scope->size)
scope->size = SZ_1G;
scope->va = tlbi_va_s1_to_va(val) & ~(scope->size - 1);
scope->asid = FIELD_GET(TLBIR_ASID_MASK, val);
break;
@@ -1154,8 +1203,6 @@ static void compute_s1_tlbi_range(struct kvm_vcpu *vcpu, u32 inst, u64 val,
case OP_TLBI_VAALE1OSNXS:
scope->type = TLBI_VAA;
scope->size = ttl_to_size(FIELD_GET(TLBI_TTL_MASK, val));
if (!scope->size)
scope->size = SZ_1G;
scope->va = tlbi_va_s1_to_va(val) & ~(scope->size - 1);
break;
case OP_TLBI_RVAE2:
@@ -1316,13 +1363,20 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)
* intersects with the TLBI request, invalidate it, and unmap the page
* from the fixmap. Because we need to look at all the vcpu-private TLBs,
* this requires some wide-ranging locking to ensure that nothing races
* against it. This may require some refcounting to avoid the search when
* no such TLB is present.
* against it. This requires some refcounting to avoid the search when
* no such TLB is present (see below).
*
* - On MMU notifiers, we must invalidate our TLB in a similar way, but
* looking at the IPA instead. The funny part is that there may not be a
* stage-2 mapping for this page if L1 hasn't accessed it using LD/ST
* instructions.
*
* - vncr_tlb_count tracks the number of valid VNCR TLBs VM-wide. This isn't
* the number of *mapped* L1 VNCR pages, which is likely be a subset (and
* by definition, a TLBI handled from L1 runs with the canonical VNCR
* page, not the L1's). The innermost trap handling code checks this to
* find out whether to return to the guest ASAP (no L1 TLBs) or to visit
* this part of the world for some extra invalidation work.
*/
int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu)
@@ -1377,7 +1431,8 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
*/
scoped_guard(write_lock, &vcpu->kvm->mmu_lock) {
this_cpu_reset_vncr_fixmap(vcpu);
vt->valid = false;
if (vt->valid)
invalidate_vncr(vcpu->kvm, vt);
vt->wi = (struct s1_walk_info) {
.regime = TR_EL20,
@@ -1391,15 +1446,15 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
va = read_vncr_el2(vcpu);
mmu_seq = vcpu->kvm->mmu_invalidate_seq;
smp_rmb();
ret = __kvm_translate_va(vcpu, &vt->wi, &vt->wr, va);
if (ret)
return ret;
write_fault = kvm_is_write_fault(vcpu);
mmu_seq = vcpu->kvm->mmu_invalidate_seq;
smp_rmb();
gfn = vt->wr.pa >> PAGE_SHIFT;
memslot = gfn_to_memslot(vcpu->kvm, gfn);
if (!memslot) {
@@ -1447,7 +1502,7 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem)
vt->hpa = pfn << PAGE_SHIFT;
vt->hpa_writable = writable;
vt->valid = true;
vt->cpu = -1;
atomic_set(&vt->cpu, -1);
kvm_make_request(KVM_REQ_MAP_L1_VNCR_EL2, vcpu);
kvm_release_faultin_page(vcpu->kvm, page, false, vt->wr.pw && vt->hpa_writable);
@@ -1502,7 +1557,20 @@ int kvm_handle_vncr_abort(struct kvm_vcpu *vcpu)
return -EIO;
}
/*
* Speculatively increment the TLB count to make sure concurrent
* TLBIs will take the slow path, and will interact with the retry
* mechanism. Drop it again on error.
*/
atomic_inc(&vcpu->kvm->arch.vncr_tlb_count);
smp_mb__after_atomic();
ret = kvm_translate_vncr(vcpu, &is_gmem);
if (ret) {
smp_mb__before_atomic();
atomic_dec(&vcpu->kvm->arch.vncr_tlb_count);
}
switch (ret) {
case -EAGAIN:
/* Let's try again... */
@@ -1568,14 +1636,16 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu)
if (!vt->valid)
return;
/* We cache the MMU state in the TLB. Check that it matches. */
if (!!(vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_ELx_M) != s1_walk_translated(&vt->wr))
return;
if (read_vncr_el2(vcpu) != vt->gva)
return;
if (vt->wr.nG && get_asid_by_regime(vcpu, TR_EL20) != vt->wr.asid)
return;
vt->cpu = smp_processor_id();
if (vt->hpa_writable && vt->wr.pw && vt->wr.pr)
prot = PAGE_KERNEL;
else if (vt->wr.pr)
@@ -1590,9 +1660,9 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu)
* FIXME: WO doesn't work at all, need POE support in the kernel.
*/
if (pgprot_val(prot) != pgprot_val(PAGE_NONE)) {
__set_fixmap(vncr_fixmap(vt->cpu), vt->hpa, prot);
atomic_set(&vt->cpu, smp_processor_id());
__set_fixmap(vncr_fixmap(atomic_read(&vt->cpu)), vt->hpa, prot);
host_data_set_flag(L1_VNCR_MAPPED);
atomic_inc(&vcpu->kvm->arch.vncr_map_count);
}
}
@@ -1728,7 +1798,7 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val)
* You get EITHER
*
* - FEAT_VHE without FEAT_E2H0
* - FEAT_NV limited to FEAT_NV2
* - FEAT_NV limited to FEAT_NV2(p1)/NV3
* - HCR_EL2.NV1 being RES0
*
* OR
@@ -1740,7 +1810,13 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val)
if (test_bit(KVM_ARM_VCPU_HAS_EL2_E2H0, kvm->arch.vcpu_features)) {
val = 0;
} else {
val = SYS_FIELD_PREP_ENUM(ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY);
val &= ID_AA64MMFR4_EL1_NV_frac;
if (cpus_have_final_cap(ARM64_HAS_NV3))
val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64MMFR4_EL1, NV_frac, NV3);
else if (cpus_have_final_cap(ARM64_HAS_NV2P1))
val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64MMFR4_EL1, NV_frac, NV2P1);
else
val = SYS_FIELD_PREP_ENUM(ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY);
val |= SYS_FIELD_PREP_ENUM(ID_AA64MMFR4_EL1, E2H0, NI_NV1);
}
break;
@@ -1826,6 +1902,10 @@ int kvm_init_nv_sysregs(struct kvm_vcpu *vcpu)
resx = get_reg_fixed_bits(kvm, HCR_EL2);
set_sysreg_masks(kvm, HCR_EL2, resx);
/* NVHCR_EL2 */
resx = get_reg_fixed_bits(kvm, NVHCR_EL2);
set_sysreg_masks(kvm, NVHCR_EL2, resx);
/* HCRX_EL2 */
resx = get_reg_fixed_bits(kvm, HCRX_EL2);
set_sysreg_masks(kvm, HCRX_EL2, resx);
@@ -1906,7 +1986,7 @@ int kvm_init_nv_sysregs(struct kvm_vcpu *vcpu)
/* ICH_HCR_EL2 */
resx.res0 = ICH_HCR_EL2_RES0;
resx.res1 = ICH_HCR_EL2_RES1;
if (!(kvm_vgic_global_state.ich_vtr_el2 & ICH_VTR_EL2_TDS))
if (!(vgic_ich_vtr() & ICH_VTR_EL2_TDS))
resx.res0 |= ICH_HCR_EL2_TDIR;
/* No GICv4 is presented to the guest */
resx.res0 |= ICH_HCR_EL2_DVIM | ICH_HCR_EL2_vSGIEOICount;
arch/arm64/kvm/pkvm.c
+19 -15
View File
@@ -185,7 +185,11 @@ free_pgd:
bool pkvm_hyp_vm_is_created(struct kvm *kvm)
{
return READ_ONCE(kvm->arch.pkvm.is_created);
/*
* Serialised by config_lock/slots_lock, or by VM lifecycle at
* teardown, so a plain read suffices.
*/
return kvm->arch.pkvm.is_created;
}
int pkvm_create_hyp_vm(struct kvm *kvm)
@@ -230,13 +234,6 @@ int pkvm_init_host_vm(struct kvm *kvm, unsigned long type)
int ret;
bool protected = type & KVM_VM_TYPE_ARM_PROTECTED;
if (pkvm_hyp_vm_is_created(kvm))
return -EINVAL;
/* VM is already reserved, no need to proceed. */
if (kvm->arch.pkvm.handle)
return 0;
/* Reserve the VM in hyp and obtain a hyp handle for the VM. */
ret = kvm_call_hyp_nvhe(__pkvm_reserve_vm);
if (ret < 0)
@@ -369,7 +366,7 @@ static int __pkvm_pgtable_stage2_unshare(struct kvm_pgtable *pgt, u64 start, u64
for_each_mapping_in_range_safe(pgt, start, end, mapping) {
ret = kvm_call_hyp_nvhe(__pkvm_host_unshare_guest, handle, mapping->gfn,
mapping->nr_pages);
(u64)mapping->nr_pages);
if (WARN_ON(ret))
return ret;
pkvm_mapping_remove(mapping, &pgt->pkvm_mappings);
@@ -466,13 +463,14 @@ int pkvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size,
size / PAGE_SIZE, prot);
}
if (WARN_ON(ret))
if (ret)
return ret;
swap(mapping, cache->mapping);
mapping->gfn = gfn;
mapping->pfn = pfn;
mapping->nr_pages = size / PAGE_SIZE;
mapping->nc = !!(prot & (KVM_PGTABLE_PROT_DEVICE | KVM_PGTABLE_PROT_NORMAL_NC));
pkvm_mapping_insert(mapping, &pgt->pkvm_mappings);
return ret;
@@ -503,7 +501,7 @@ int pkvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size)
lockdep_assert_held(&kvm->mmu_lock);
for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping) {
ret = kvm_call_hyp_nvhe(__pkvm_host_wrprotect_guest, handle, mapping->gfn,
mapping->nr_pages);
(u64)mapping->nr_pages);
if (WARN_ON(ret))
break;
}
@@ -517,9 +515,15 @@ int pkvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size)
struct pkvm_mapping *mapping;
lockdep_assert_held(&kvm->mmu_lock);
for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping)
__clean_dcache_guest_page(pfn_to_kaddr(mapping->pfn),
PAGE_SIZE * mapping->nr_pages);
if (cpus_have_final_cap(ARM64_HAS_STAGE2_FWB))
return 0;
for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping) {
if (!mapping->nc)
__clean_dcache_guest_page(pfn_to_kaddr(mapping->pfn),
PAGE_SIZE * mapping->nr_pages);
}
return 0;
}
@@ -537,7 +541,7 @@ bool pkvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, u64
lockdep_assert_held(&kvm->mmu_lock);
for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping)
young |= kvm_call_hyp_nvhe(__pkvm_host_test_clear_young_guest, handle, mapping->gfn,
mapping->nr_pages, mkold);
(u64)mapping->nr_pages, mkold);
return young;
}
arch/arm64/kvm/pmu-emul.c
+41 -13
View File
@@ -838,9 +838,9 @@ static u64 __compute_pmceid(struct arm_pmu *pmu, bool pmceid1)
return ((u64)hi[pmceid1] << 32) | lo[pmceid1];
}
static u64 compute_pmceid0(struct arm_pmu *pmu)
static u64 compute_pmceid0(struct kvm_vcpu *vcpu)
{
u64 val = __compute_pmceid(pmu, 0);
u64 val = __compute_pmceid(vcpu->kvm->arch.arm_pmu, 0);
/* always support SW_INCR */
val |= BIT(ARMV8_PMUV3_PERFCTR_SW_INCR);
@@ -849,32 +849,33 @@ static u64 compute_pmceid0(struct arm_pmu *pmu)
return val;
}
static u64 compute_pmceid1(struct arm_pmu *pmu)
static u64 compute_pmceid1(struct kvm_vcpu *vcpu)
{
u64 val = __compute_pmceid(pmu, 1);
u64 val = __compute_pmceid(vcpu->kvm->arch.arm_pmu, 1);
/*
* Don't advertise STALL_SLOT*, as PMMIR_EL0 is handled
* as RAZ
* If KVM_ARM_VCPU_PMU_V3_STRICT is not set, PMMIR_EL1 is
* unconditionally RAZ, so don't advertise STALL_SLOT* events.
*/
val &= ~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT - 32) |
BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND - 32) |
BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND - 32));
if (!kvm_vcpu_has_pmuv3_strict(vcpu))
val &= ~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT - 32) |
BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND - 32) |
BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND - 32));
return val;
}
u64 kvm_pmu_get_pmceid(struct kvm_vcpu *vcpu, bool pmceid1)
{
struct arm_pmu *cpu_pmu = vcpu->kvm->arch.arm_pmu;
unsigned long *bmap = vcpu->kvm->arch.pmu_filter;
u64 val, mask = 0;
int base, i, nr_events;
if (!pmceid1) {
val = compute_pmceid0(cpu_pmu);
val = compute_pmceid0(vcpu);
base = 0;
} else {
val = compute_pmceid1(cpu_pmu);
val = compute_pmceid1(vcpu);
base = 32;
}
@@ -938,6 +939,10 @@ int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu)
static int kvm_arm_pmu_v3_init(struct kvm_vcpu *vcpu)
{
/* Only possible when using KVM_ARM_VCPU_PMU_V3_STRICT */
if (!vcpu->kvm->arch.arm_pmu)
return -ENXIO;
if (irqchip_in_kernel(vcpu->kvm)) {
int ret;
@@ -1008,6 +1013,14 @@ u8 kvm_arm_pmu_get_max_counters(struct kvm *kvm)
{
struct arm_pmu *arm_pmu = kvm->arch.arm_pmu;
/*
* Under KVM_ARM_VCPU_PMU_V3_STRICT no PMU exists until userspace sets
* one, so this can be reached before arm_pmu is set. Report no
* counters in that case.
*/
if (!arm_pmu)
return 0;
/*
* PMUv3 requires that all event counters are capable of counting any
* event, though the same may not be true of non-PMUv3 hardware.
@@ -1049,7 +1062,8 @@ static void kvm_arm_set_pmu(struct kvm *kvm, struct arm_pmu *arm_pmu)
}
/**
* kvm_arm_set_default_pmu - No PMU set, get the default one.
* kvm_arm_set_default_pmu - No PMU set and KVM_ARM_VCPU_PMU_V3_STRICT not
* set, get the default one.
* @kvm: The kvm pointer
*
* The observant among you will notice that the supported_cpus
@@ -1092,6 +1106,17 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
kvm_arm_set_pmu(kvm, arm_pmu);
cpumask_copy(kvm->arch.supported_cpus, &arm_pmu->supported_cpus);
/*
* Since a specific PMU is explicitly selected,
* PMMIR_EL1.SLOTS is deterministic to the guest.
* If KVM_ARM_VCPU_PMU_V3_STRICT is set, snapshot
* the value to allow the guest to read it.
*/
if (kvm_vcpu_has_pmuv3_strict(vcpu))
kvm->arch.pmmir_slots =
FIELD_GET(ARMV8_PMU_SLOTS,
arm_pmu->reg_pmmir);
ret = 0;
break;
}
@@ -1178,6 +1203,9 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
if (kvm_vm_has_ran_once(kvm))
return -EBUSY;
if (!kvm->arch.arm_pmu)
return -ENXIO;
if (!kvm->arch.pmu_filter) {
kvm->arch.pmu_filter = bitmap_alloc(nr_events, GFP_KERNEL_ACCOUNT);
if (!kvm->arch.pmu_filter)
arch/arm64/kvm/psci.c
+1 -29
View File
@@ -21,16 +21,6 @@
* as described in ARM document number ARM DEN 0022A.
*/
#define AFFINITY_MASK(level) ~((0x1UL << ((level) * MPIDR_LEVEL_BITS)) - 1)
static unsigned long psci_affinity_mask(unsigned long affinity_level)
{
if (affinity_level <= 3)
return MPIDR_HWID_BITMASK & AFFINITY_MASK(affinity_level);
return 0;
}
static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)
{
/*
@@ -51,12 +41,6 @@ static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu)
return PSCI_RET_SUCCESS;
}
static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu,
unsigned long affinity)
{
return !(affinity & ~MPIDR_HWID_BITMASK);
}
static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
{
struct vcpu_reset_state *reset_state;
@@ -135,7 +119,7 @@ static unsigned long kvm_psci_vcpu_affinity_info(struct kvm_vcpu *vcpu)
return PSCI_RET_INVALID_PARAMS;
/* Determine target affinity mask */
target_affinity_mask = psci_affinity_mask(lowest_affinity_level);
target_affinity_mask = kvm_psci_affinity_mask(lowest_affinity_level);
if (!target_affinity_mask)
return PSCI_RET_INVALID_PARAMS;
@@ -220,18 +204,6 @@ static void kvm_psci_system_suspend(struct kvm_vcpu *vcpu)
run->exit_reason = KVM_EXIT_SYSTEM_EVENT;
}
static void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu)
{
int i;
/*
* Zero the input registers' upper 32 bits. They will be fully
* zeroed on exit, so we're fine changing them in place.
*/
for (i = 1; i < 4; i++)
vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i)));
}
static unsigned long kvm_psci_check_allowed_function(struct kvm_vcpu *vcpu, u32 fn)
{
/*
arch/arm64/kvm/reset.c
+3 -57
View File
@@ -34,18 +34,6 @@
static u32 __ro_after_init kvm_ipa_limit;
unsigned int __ro_after_init kvm_host_sve_max_vl;
/*
* ARMv8 Reset Values
*/
#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
PSR_F_BIT | PSR_D_BIT)
#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
PSR_F_BIT | PSR_D_BIT)
#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
PSR_AA32_I_BIT | PSR_AA32_F_BIT)
unsigned int __ro_after_init kvm_sve_max_vl;
int __init kvm_arm_init_sve(void)
@@ -191,7 +179,6 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
{
struct vcpu_reset_state reset_state;
bool loaded;
u32 pstate;
spin_lock(&vcpu->arch.mp_state_lock);
reset_state = vcpu->arch.reset_state;
@@ -210,21 +197,8 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
kvm_vcpu_reset_sve(vcpu);
}
if (vcpu_el1_is_32bit(vcpu))
pstate = VCPU_RESET_PSTATE_SVC;
else if (vcpu_has_nv(vcpu))
pstate = VCPU_RESET_PSTATE_EL2;
else
pstate = VCPU_RESET_PSTATE_EL1;
/* Reset core registers */
memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu)));
memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs));
vcpu->arch.ctxt.spsr_abt = 0;
vcpu->arch.ctxt.spsr_und = 0;
vcpu->arch.ctxt.spsr_irq = 0;
vcpu->arch.ctxt.spsr_fiq = 0;
vcpu_gp_regs(vcpu)->pstate = pstate;
kvm_reset_vcpu_core(vcpu);
/* Reset system registers */
kvm_reset_sys_regs(vcpu);
@@ -233,36 +207,8 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
* Additional reset state handling that PSCI may have imposed on us.
* Must be done after all the sys_reg reset.
*/
if (reset_state.reset) {
unsigned long target_pc = reset_state.pc;
/* Gracefully handle Thumb2 entry point */
if (vcpu_mode_is_32bit(vcpu) && (target_pc & 1)) {
target_pc &= ~1UL;
vcpu_set_thumb(vcpu);
}
/* Propagate caller endianness */
if (reset_state.be)
kvm_vcpu_set_be(vcpu);
*vcpu_pc(vcpu) = target_pc;
/*
* We may come from a state where either a PC update was
* pending (SMC call resulting in PC being increpented to
* skip the SMC) or a pending exception. Make sure we get
* rid of all that, as this cannot be valid out of reset.
*
* Note that clearing the exception mask also clears PC
* updates, but that's an implementation detail, and we
* really want to make it explicit.
*/
vcpu_clear_flag(vcpu, PENDING_EXCEPTION);
vcpu_clear_flag(vcpu, EXCEPT_MASK);
vcpu_clear_flag(vcpu, INCREMENT_PC);
vcpu_set_reg(vcpu, 0, reset_state.r0);
}
if (reset_state.reset)
kvm_reset_vcpu_psci(vcpu, &reset_state);
/* Reset timer */
kvm_timer_vcpu_reset(vcpu);
arch/arm64/kvm/sys_regs.c
+129 -23
View File
@@ -183,8 +183,6 @@ static void locate_register(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg,
switch (reg) {
MAPPED_EL2_SYSREG(SCTLR_EL2, SCTLR_EL1,
translate_sctlr_el2_to_sctlr_el1 );
MAPPED_EL2_SYSREG(CPTR_EL2, CPACR_EL1,
translate_cptr_el2_to_cpacr_el1 );
MAPPED_EL2_SYSREG(TTBR0_EL2, TTBR0_EL1,
translate_ttbr0_el2_to_ttbr0_el1 );
MAPPED_EL2_SYSREG(TTBR1_EL2, TTBR1_EL1, NULL );
@@ -210,6 +208,33 @@ static void locate_register(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg,
loc->loc = ((is_hyp_ctxt(vcpu) && vcpu_el2_e2h_is_set(vcpu)) ?
SR_LOC_SPECIAL : SR_LOC_MEMORY);
break;
case CPTR_EL2:
/*
* CPTR_EL2 is just as special, and needs a certain amount
* of handholding. It always lives in memory, due to being
* heavily trapped thanks to CPACR_EL1.TCPAC being RES0.
* FEAT_NV2p1 fixes this.
*/
locate_mapped_el2_register(vcpu, CPTR_EL2, CPACR_EL1,
translate_cptr_el2_to_cpacr_el1,
loc);
if (is_hyp_ctxt(vcpu) && vcpu_el2_e2h_is_set(vcpu))
loc->loc = SR_LOC_SPECIAL;
break;
case NVHCR_EL2:
/*
* Yes, NVHCR_EL2 maps to itself when loaded in nested
* context. If you feel like the architecture is double
* backing on itself upside down, you're not alone.
*/
WARN_ON_ONCE(!kvm_has_nv3(vcpu->kvm));
if (is_hyp_ctxt(vcpu)) {
loc->loc = SR_LOC_MEMORY;
} else {
loc->loc = SR_LOC_LOADED | SR_LOC_MAPPED;
loc->map_reg = NVHCR_EL2;
}
break;
default:
loc->loc = locate_direct_register(vcpu, reg);
}
@@ -249,6 +274,7 @@ static u64 read_sr_from_cpu(enum vcpu_sysreg reg)
case DACR32_EL2: val = read_sysreg_s(SYS_DACR32_EL2); break;
case IFSR32_EL2: val = read_sysreg_s(SYS_IFSR32_EL2); break;
case DBGVCR32_EL2: val = read_sysreg_s(SYS_DBGVCR32_EL2); break;
case NVHCR_EL2: val = read_sysreg_s(SYS_NVHCR_EL2); break;
default: WARN_ON_ONCE(1);
}
@@ -287,6 +313,7 @@ static void write_sr_to_cpu(enum vcpu_sysreg reg, u64 val)
case DACR32_EL2: write_sysreg_s(val, SYS_DACR32_EL2); break;
case IFSR32_EL2: write_sysreg_s(val, SYS_IFSR32_EL2); break;
case DBGVCR32_EL2: write_sysreg_s(val, SYS_DBGVCR32_EL2); break;
case NVHCR_EL2: write_sysreg_s(val, SYS_NVHCR_EL2); break;
default: WARN_ON_ONCE(1);
}
}
@@ -311,9 +338,16 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg)
switch (reg) {
case CNTHCTL_EL2:
val = read_sysreg_el1(SYS_CNTKCTL);
val &= CNTKCTL_VALID_BITS;
val |= __vcpu_sys_reg(vcpu, reg) & ~CNTKCTL_VALID_BITS;
if (!cpus_have_final_cap(ARM64_HAS_NV2P1)) {
val &= CNTKCTL_VALID_BITS;
val |= __vcpu_sys_reg(vcpu, reg) & ~CNTKCTL_VALID_BITS;
}
return val;
case CPTR_EL2:
if (cpus_have_final_cap(ARM64_HAS_NV2P1))
return read_sysreg_el1(SYS_CPACR);
else
return __vcpu_sys_reg(vcpu, reg);
default:
WARN_ON_ONCE(1);
}
@@ -359,6 +393,9 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg)
*/
write_sysreg_el1(val, SYS_CNTKCTL);
break;
case CPTR_EL2:
write_sysreg_el1(val, SYS_CPACR);
break;
default:
WARN_ON_ONCE(1);
}
@@ -976,21 +1013,9 @@ static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
{
u64 mpidr;
u64 mpidr = kvm_calculate_mpidr(vcpu);
/*
* Map the vcpu_id into the first three affinity level fields of
* the MPIDR. We limit the number of VCPUs in level 0 due to a
* limitation to 16 CPUs in that level in the ICC_SGIxR registers
* of the GICv3 to be able to address each CPU directly when
* sending IPIs.
*/
mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
mpidr |= (1ULL << 31);
vcpu_write_sys_reg(vcpu, mpidr, MPIDR_EL1);
return mpidr;
}
@@ -1367,6 +1392,64 @@ static bool access_pminten(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
return true;
}
static bool access_pmmir(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
const struct sys_reg_desc *r)
{
if (p->is_write)
return write_to_read_only(vcpu, p, r);
/*
* If KVM_ARM_VCPU_PMU_V3_STRICT is set and PMU was explicitly
* selected, the underlying hardware SLOTS value was read into this
* field. Otherwise, it stays 0. All other PMMIR_EL1 fields are RAZ.
*/
p->regval = FIELD_PREP(ARMV8_PMU_SLOTS, vcpu->kvm->arch.pmmir_slots);
return true;
}
static int get_pmmir(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
u64 *val)
{
*val = FIELD_PREP(ARMV8_PMU_SLOTS, vcpu->kvm->arch.pmmir_slots);
return 0;
}
static int set_pmmir(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
u64 val)
{
struct kvm *kvm = vcpu->kvm;
u8 slots = FIELD_GET(ARMV8_PMU_SLOTS, val);
/*
* Only the SLOTS field is exposed (get_pmmir returns just that field),
* so reject a write that sets any other bit rather than silently
* masking it.
*/
if (val & ~(u64)ARMV8_PMU_SLOTS)
return -EINVAL;
guard(mutex)(&kvm->arch.config_lock);
/*
* Once the VM has started PMMIR_EL1 is immutable. Reject any write
* that does not match the current value.
*/
if (kvm_vm_has_ran_once(kvm))
return slots == kvm->arch.pmmir_slots ? 0 : -EBUSY;
/*
* Only SLOTS = 0 is honored for backwards compatibility with the
* old RAZ behavior. Reject any non-zero write that does not match
* the current value.
*/
if (!slots)
kvm->arch.pmmir_slots = 0;
else if (slots != kvm->arch.pmmir_slots)
return -EINVAL;
return 0;
}
static bool access_pmovs(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
const struct sys_reg_desc *r)
{
@@ -1444,6 +1527,7 @@ static int set_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
*/
if (!kvm_vm_has_ran_once(kvm) &&
!vcpu_has_nv(vcpu) &&
!kvm_vcpu_has_pmuv3_strict(vcpu) &&
new_n <= kvm_arm_pmu_get_max_counters(kvm))
kvm->arch.nr_pmu_counters = new_n;
@@ -2840,6 +2924,16 @@ static unsigned int vncr_el2_visibility(const struct kvm_vcpu *vcpu,
return REG_HIDDEN;
}
static unsigned int nvhcr_el2_visibility(const struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
{
if (el2_visibility(vcpu, rd) == 0 &&
kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV3))
return 0;
return REG_HIDDEN;
}
static unsigned int sctlr2_visibility(const struct kvm_vcpu *vcpu,
const struct sys_reg_desc *rd)
{
@@ -3448,7 +3542,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
{ PMU_SYS_REG(PMINTENCLR_EL1),
.access = access_pminten, .reg = PMINTENSET_EL1,
.get_user = get_pmreg, .set_user = set_pmreg },
{ SYS_DESC(SYS_PMMIR_EL1), trap_raz_wi },
{ PMU_SYS_REG(PMMIR_EL1), .access = access_pmmir, .reset = NULL,
.get_user = get_pmmir, .set_user = set_pmmir },
{ SYS_DESC(SYS_MAIR_EL1), access_vm_reg, reset_unknown, MAIR_EL1 },
{ SYS_DESC(SYS_PIRE0_EL1), NULL, reset_unknown, PIRE0_EL1,
@@ -3753,6 +3848,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
sve_el2_visibility),
EL2_REG_VNCR(HCRX_EL2, reset_val, 0),
EL2_REG_FILTERED(NVHCR_EL2, undef_access, reset_val, 0,
nvhcr_el2_visibility),
EL2_REG(TTBR0_EL2, access_rw, reset_val, 0),
EL2_REG(TTBR1_EL2, access_rw, reset_val, 0),
@@ -4057,6 +4154,7 @@ static bool handle_ripas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2);
u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2);
u64 base, range;
int pa_bits;
if (!kvm_supported_tlbi_ipas2_op(vcpu, sys_encoding))
return undef_access(vcpu, p, r);
@@ -4068,6 +4166,16 @@ static bool handle_ripas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
*/
base = decode_range_tlbi(p->regval, &range, NULL);
/*
* Ignore TLBIs that start out of PA_bits range, and cap the
* invalidation to the [base:bit(PA_bits)] interval.
*/
pa_bits = kvm_get_pa_bits(vcpu->kvm);
if (fls64(base) > pa_bits)
return true;
range = min(range, BIT_ULL(pa_bits) - base);
kvm_s2_mmu_iterate_by_vmid(vcpu->kvm, get_vmid(vttbr),
&(union tlbi_info) {
.range = {
@@ -4593,7 +4701,7 @@ static const struct sys_reg_desc cp15_regs[] = {
{ CP15_PMU_SYS_REG(HI, 0, 9, 14, 4), .access = access_pmceid },
{ CP15_PMU_SYS_REG(HI, 0, 9, 14, 5), .access = access_pmceid },
/* PMMIR */
{ CP15_PMU_SYS_REG(DIRECT, 0, 9, 14, 6), .access = trap_raz_wi },
{ CP15_PMU_SYS_REG(DIRECT, 0, 9, 14, 6), .access = access_pmmir },
/* PRRR/MAIR0 */
{ AA32(LO), Op1( 0), CRn(10), CRm( 2), Op2( 0), access_vm_reg, NULL, MAIR_EL1 },
@@ -4861,10 +4969,8 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu,
* Make a 64-bit value out of Rt and Rt2. As we use the same trap
* backends between AArch32 and AArch64, we get away with it.
*/
if (params.is_write) {
params.regval = vcpu_get_reg(vcpu, Rt) & 0xffffffff;
params.regval |= vcpu_get_reg(vcpu, Rt2) << 32;
}
params.regval = vcpu_get_reg(vcpu, Rt) & 0xffffffff;
params.regval |= vcpu_get_reg(vcpu, Rt2) << 32;
/*
* If the table contains a handler, handle the
arch/arm64/kvm/sys_regs.h
+19
View File
@@ -222,6 +222,25 @@ find_reg(const struct sys_reg_params *params, const struct sys_reg_desc table[],
return __inline_bsearch((void *)pval, table, num, sizeof(table[0]), match_sys_reg);
}
static inline u64 kvm_calculate_mpidr(const struct kvm_vcpu *vcpu)
{
u64 mpidr;
/*
* Map the vcpu_id into the first three affinity level fields of
* the MPIDR. We limit the number of VCPUs in level 0 due to a
* limitation to 16 CPUs in that level in the ICC_SGIxR registers
* of the GICv3 to be able to address each CPU directly when
* sending IPIs.
*/
mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
mpidr |= (1ULL << 31);
return mpidr;
}
const struct sys_reg_desc *get_reg_by_id(u64 id,
const struct sys_reg_desc table[],
unsigned int num);
arch/arm64/kvm/vgic-sys-reg-v3.c
+4 -4
View File
@@ -35,12 +35,12 @@ static int set_gic_ctlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
vgic_v3_cpu->num_id_bits = host_id_bits;
host_seis = FIELD_GET(ICH_VTR_EL2_SEIS, kvm_vgic_global_state.ich_vtr_el2);
host_seis = FIELD_GET(ICH_VTR_EL2_SEIS, vgic_ich_vtr());
seis = FIELD_GET(ICC_CTLR_EL1_SEIS_MASK, val);
if (host_seis != seis)
return -EINVAL;
host_a3v = FIELD_GET(ICH_VTR_EL2_A3V, kvm_vgic_global_state.ich_vtr_el2);
host_a3v = FIELD_GET(ICH_VTR_EL2_A3V, vgic_ich_vtr());
a3v = FIELD_GET(ICC_CTLR_EL1_A3V_MASK, val);
if (host_a3v != a3v)
return -EINVAL;
@@ -69,9 +69,9 @@ static int get_gic_ctlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
val |= FIELD_PREP(ICC_CTLR_EL1_ID_BITS_MASK, vgic_v3_cpu->num_id_bits);
val |= FIELD_PREP(ICC_CTLR_EL1_SEIS_MASK,
FIELD_GET(ICH_VTR_EL2_SEIS,
kvm_vgic_global_state.ich_vtr_el2));
vgic_ich_vtr()));
val |= FIELD_PREP(ICC_CTLR_EL1_A3V_MASK,
FIELD_GET(ICH_VTR_EL2_A3V, kvm_vgic_global_state.ich_vtr_el2));
FIELD_GET(ICH_VTR_EL2_A3V, vgic_ich_vtr()));
/*
* The VMCR.CTLR value is in ICC_CTLR_EL1 layout.
* Extract it directly using ICC_CTLR_EL1 reg definitions.
arch/arm64/kvm/vgic/vgic-init.c
+7 -1
View File
@@ -176,6 +176,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
}
kvm->arch.vgic.vgic_model = 0;
kvm->arch.vgic.in_kernel = false;
goto out_unlock;
}
@@ -210,6 +211,9 @@ static int kvm_vgic_dist_init(struct kvm *kvm, unsigned int nr_spis)
struct kvm_vcpu *vcpu0 = kvm_get_vcpu(kvm, 0);
int i;
if (dist->spis)
return 0;
dist->active_spis = (atomic_t)ATOMIC_INIT(0);
dist->spis = kzalloc_objs(struct vgic_irq, nr_spis, GFP_KERNEL_ACCOUNT);
if (!dist->spis)
@@ -787,7 +791,8 @@ int kvm_vgic_hyp_init(void)
if (has_mask && !gic_kvm_info->maint_irq) {
kvm_err("No vgic maintenance irq\n");
return -ENXIO;
ret = -ENXIO;
goto out_free;
}
/*
@@ -820,6 +825,7 @@ int kvm_vgic_hyp_init(void)
kvm_vgic_global_state.maint_irq = gic_kvm_info->maint_irq;
out_free:
kfree(gic_kvm_info);
gic_kvm_info = NULL;
arch/arm64/kvm/vgic/vgic-its.c
+19 -7
View File
@@ -2035,15 +2035,16 @@ static u32 compute_next_devid_offset(struct list_head *h,
static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
{
struct its_ite *next;
u32 next_offset;
struct its_ite *next = ite;
if (list_is_last(&ite->ite_list, h))
return 0;
next = list_next_entry(ite, ite_list);
next_offset = next->event_id - ite->event_id;
/* Point at the next ITE that vgic_its_save_ite() stores as valid. */
list_for_each_entry_continue(next, h, ite_list) {
if (next->collection)
return min_t(u32, next->event_id - ite->event_id,
VITS_ITE_MAX_EVENTID_OFFSET);
}
return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
return 0;
}
/**
@@ -2119,6 +2120,14 @@ static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
u32 next_offset;
u64 val;
/*
* MAPC with V=0 keeps the ITEs mapped but drops their collection,
* and with it the ICID. Save a zeroed entry, which the restore path
* reads back as invalid.
*/
if (!ite->collection)
return vgic_its_write_entry_lock(its, gpa, 0ULL, ite);
next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
((u64)ite->irq->intid << KVM_ITS_ITE_PINTID_SHIFT) |
@@ -2532,6 +2541,9 @@ static int vgic_its_save_collection_table(struct vgic_its *its)
max_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
list_for_each_entry(collection, &its->collection_list, coll_list) {
if (!vgic_its_check_id(its, baser, collection->collection_id, NULL))
return -EINVAL;
ret = vgic_its_save_cte(its, collection, gpa);
if (ret)
return ret;
arch/arm64/kvm/vgic/vgic-v2.c
+2 -1
View File
@@ -170,8 +170,9 @@ void vgic_v2_deactivate(struct kvm_vcpu *vcpu, u32 val)
/* Make sure we're in the same context as LR handling */
local_irq_save(flags);
/* Guest-supplied INTID: out of range yields no irq, so ignore it */
irq = vgic_get_vcpu_irq(vcpu, val);
if (WARN_ON_ONCE(!irq))
if (!irq)
goto out;
/* See the corresponding v3 code for the rationale */
arch/arm64/kvm/vgic/vgic-v3-nested.c
+1 -1
View File
@@ -152,7 +152,7 @@ static void vgic_compute_mi_state(struct kvm_vcpu *vcpu, struct mi_state *mi_sta
eisr |= BIT(i);
if (!(lr & ICH_LR_STATE))
elrsr |= BIT(i);
pend |= (lr & ICH_LR_PENDING_BIT);
pend |= (lr & ICH_LR_STATE) == ICH_LR_PENDING_BIT;
}
mi_state->eisr = eisr;
arch/arm64/kvm/vgic/vgic-v3.c
+71 -34
View File
@@ -496,9 +496,9 @@ void vgic_v3_reset(struct kvm_vcpu *vcpu)
}
vcpu->arch.vgic_cpu.num_id_bits = FIELD_GET(ICH_VTR_EL2_IDbits,
kvm_vgic_global_state.ich_vtr_el2);
vgic_ich_vtr());
vcpu->arch.vgic_cpu.num_pri_bits = FIELD_GET(ICH_VTR_EL2_PRIbits,
kvm_vgic_global_state.ich_vtr_el2) + 1;
vgic_ich_vtr()) + 1;
}
void vcpu_set_ich_hcr(struct kvm_vcpu *vcpu)
@@ -617,9 +617,13 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
bool is_pending;
bool stored;
irq = vgic_get_irq(kvm, index);
if (!irq)
continue;
vcpu = irq->target_vcpu;
if (!vcpu)
continue;
goto put_irq;
pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
@@ -630,7 +634,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
if (ptr != last_ptr) {
ret = kvm_read_guest_lock(kvm, ptr, &val, 1);
if (ret)
goto out;
goto put_irq;
last_ptr = ptr;
}
@@ -642,7 +646,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
vgic_v4_get_vlpi_state(irq, &is_pending);
if (stored == is_pending)
continue;
goto put_irq;
if (is_pending)
val |= 1 << bit_nr;
@@ -650,6 +654,8 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
val &= ~(1 << bit_nr);
ret = vgic_write_guest_lock(kvm, ptr, &val, 1);
put_irq:
vgic_put_irq(kvm, irq);
if (ret)
goto out;
}
@@ -815,27 +821,9 @@ static int __init early_gicv4_enable(char *buf)
}
early_param("kvm-arm.vgic_v4_enable", early_gicv4_enable);
static const struct midr_range broken_seis[] = {
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM),
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM),
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM_PRO),
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM_PRO),
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM_MAX),
MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM_MAX),
MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD),
MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE),
MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD_PRO),
MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE_PRO),
MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD_MAX),
MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE_MAX),
{},
};
static bool vgic_v3_broken_seis(void)
static __always_inline bool vgic_v3_broken_seis(void)
{
return (is_kernel_in_hyp_mode() &&
is_midr_in_range_list(broken_seis) &&
(read_sysreg_s(SYS_ICH_VTR_EL2) & ICH_VTR_EL2_SEIS));
return cpus_have_cap(ARM64_WORKAROUND_GICv3_BROKEN_SEIS);
}
void noinstr kvm_compute_ich_hcr_trap_bits(struct alt_instr *alt,
@@ -882,6 +870,61 @@ void noinstr kvm_compute_ich_hcr_trap_bits(struct alt_instr *alt,
*updptr = cpu_to_le32(insn);
}
void noinstr kvm_patch_ich_vtr_el2(struct alt_instr *alt,
__le32 *origptr, __le32 *updptr,
int nr_inst)
{
struct arm_smccc_res res = {};
u32 insn, oinsn, rd, vtr;
/* No KVM? Nothing to do */
if (!is_hyp_mode_available())
return;
/* No v3, compat, nor the fruity erzatz of a GIC? Bugger off */
if (!cpus_have_cap(ARM64_HAS_GICV5_LEGACY) &&
!cpus_have_cap(ARM64_HAS_GICV3_CPUIF) &&
!vgic_v3_broken_seis())
return;
/*
* At the point where this is called, we are guaranteed that if
* we're running at EL1, then the EL2 stubs are still in place.
*/
if (is_kernel_in_hyp_mode())
res.a1 = read_sysreg_s(SYS_ICH_VTR_EL2);
else
arm_smccc_1_1_hvc(HVC_GET_ICH_VTR_EL2, &res);
if (res.a0 == HVC_STUB_ERR)
return;
vtr = res.a1;
if (vgic_v3_broken_seis())
vtr &= ~ICH_VTR_EL2_SEIS;
/* Compute target register */
oinsn = le32_to_cpu(*origptr);
rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn);
/* movz rd, #(vtr & 0xffff) */
insn = aarch64_insn_gen_movewide(rd,
(u16)vtr,
0,
AARCH64_INSN_VARIANT_64BIT,
AARCH64_INSN_MOVEWIDE_ZERO);
*updptr++ = cpu_to_le32(insn);
/* movk rd, #((vtr >> 16) & 0xffff), lsl #16 */
insn = aarch64_insn_gen_movewide(rd,
(u16)(vtr >> 16),
16,
AARCH64_INSN_VARIANT_64BIT,
AARCH64_INSN_MOVEWIDE_KEEP);
*updptr++ = cpu_to_le32(insn);
}
void vgic_v3_enable_cpuif_traps(void)
{
u64 traps = vgic_ich_hcr_trap_bits();
@@ -905,12 +948,12 @@ void vgic_v3_enable_cpuif_traps(void)
*/
int vgic_v3_probe(const struct gic_kvm_info *info)
{
u64 ich_vtr_el2 = kvm_call_hyp_ret(__vgic_v3_get_gic_config);
u64 ich_vtr_el2;
bool has_v2;
int ret;
has_v2 = ich_vtr_el2 >> 63;
ich_vtr_el2 = (u32)ich_vtr_el2;
has_v2 = kvm_call_hyp_ret(__vgic_v3_get_gic_config);
ich_vtr_el2 = vgic_ich_vtr();
/*
* The ListRegs field is 5 bits, but there is an architectural
@@ -918,7 +961,6 @@ int vgic_v3_probe(const struct gic_kvm_info *info)
*/
kvm_vgic_global_state.nr_lr = (ich_vtr_el2 & 0xf) + 1;
kvm_vgic_global_state.can_emulate_gicv2 = false;
kvm_vgic_global_state.ich_vtr_el2 = ich_vtr_el2;
/* GICv4 support? */
if (info->has_v4) {
@@ -965,11 +1007,6 @@ int vgic_v3_probe(const struct gic_kvm_info *info)
if (has_v2)
static_branch_enable(&vgic_v3_has_v2_compat);
if (vgic_v3_broken_seis()) {
kvm_info("GICv3 with broken locally generated SEI\n");
kvm_vgic_global_state.ich_vtr_el2 &= ~ICH_VTR_EL2_SEIS;
}
vgic_v3_enable_cpuif_traps();
kvm_vgic_global_state.vctrl_base = NULL;
arch/arm64/kvm/vgic/vgic-v5.c
+1 -4
View File
@@ -40,7 +40,6 @@ static void vgic_v5_get_implemented_ppis(void)
int vgic_v5_probe(const struct gic_kvm_info *info)
{
bool v5_registered = false;
u64 ich_vtr_el2;
int ret;
kvm_vgic_global_state.type = VGIC_V5;
@@ -83,14 +82,12 @@ skip_v5:
}
kvm_vgic_global_state.has_gcie_v3_compat = true;
ich_vtr_el2 = kvm_call_hyp_ret(__vgic_v3_get_gic_config);
kvm_vgic_global_state.ich_vtr_el2 = (u32)ich_vtr_el2;
/*
* The ListRegs field is 5 bits, but there is an architectural
* maximum of 16 list registers. Just ignore bit 4...
*/
kvm_vgic_global_state.nr_lr = (ich_vtr_el2 & 0xf) + 1;
kvm_vgic_global_state.nr_lr = (vgic_ich_vtr() & 0xf) + 1;
ret = kvm_register_vgic_device(KVM_DEV_TYPE_ARM_VGIC_V3);
if (ret) {
arch/arm64/kvm/vgic/vgic.c
+5 -2
View File
@@ -93,8 +93,9 @@ struct vgic_irq *vgic_get_irq(struct kvm *kvm, u32 intid)
/* SPIs */
if (intid >= VGIC_NR_PRIVATE_IRQS &&
intid < (kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS)) {
intid = array_index_nospec(intid, kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS);
return &kvm->arch.vgic.spis[intid - VGIC_NR_PRIVATE_IRQS];
intid -= VGIC_NR_PRIVATE_IRQS;
intid = array_index_nospec(intid, kvm->arch.vgic.nr_spis);
return &kvm->arch.vgic.spis[intid];
}
/* LPIs */
@@ -117,6 +118,8 @@ struct vgic_irq *vgic_get_vcpu_irq(struct kvm_vcpu *vcpu, u32 intid)
switch (type) {
case KVM_DEV_TYPE_ARM_VGIC_V5:
intid = vgic_v5_get_hwirq_id(intid);
if (intid >= VGIC_V5_NR_PRIVATE_IRQS)
return NULL;
intid = array_index_nospec(intid, VGIC_V5_NR_PRIVATE_IRQS);
break;
default:
arch/arm64/kvm/vgic/vgic.h
+18 -1
View File
@@ -71,11 +71,28 @@
ICH_VTR_EL2_IDbits)
#define KVM_ICH_VTR_EL2_RES1 ICH_VTR_EL2_nV4
void kvm_patch_ich_vtr_el2(struct alt_instr *alt,
__le32 *origptr, __le32 *updptr, int nr_inst);
static inline u64 vgic_ich_vtr(void)
{
u64 vtr;
/* All non-RES0 bits are in the bottom 32bits */
asm volatile(ALTERNATIVE_CB("movz %0, #0\n"
"movk %0, #0, lsl #16\n",
ARM64_ALWAYS_SYSTEM,
kvm_patch_ich_vtr_el2)
: "=r" (vtr));
return vtr;
}
static inline u64 kvm_get_guest_vtr_el2(void)
{
u64 vtr;
vtr = kvm_vgic_global_state.ich_vtr_el2;
vtr = vgic_ich_vtr();
vtr &= ~KVM_ICH_VTR_EL2_RES0;
vtr |= KVM_ICH_VTR_EL2_RES1;
arch/arm64/tools/cpucaps
+3
View File
@@ -52,6 +52,8 @@ HAS_LS64_V
HAS_LSUI
HAS_MOPS
HAS_NESTED_VIRT
HAS_NV2P1
HAS_NV3
HAS_PAN
HAS_PMUV3
HAS_S1PIE
@@ -121,6 +123,7 @@ WORKAROUND_CAVIUM_TX2_219_TVM
WORKAROUND_CLEAN_CACHE
WORKAROUND_DEVICE_LOAD_ACQUIRE
WORKAROUND_DISABLE_CNP
WORKAROUND_GICv3_BROKEN_SEIS
WORKAROUND_NVIDIA_OLYMPUS_1027
WORKAROUND_PMUV3_IMPDEF_TRAPS
WORKAROUND_QCOM_FALKOR_E1003
arch/arm64/tools/gen-sysreg.awk
+4 -4
View File
@@ -228,7 +228,7 @@ $1 == "EndSysreg" && block_current() == "Sysreg" {
}
# Currently this is effectivey a comment, in future we may want to emit
# defines for the fields.
# defines for the fields. We do emit RESx and UNKN values in any case.
($1 == "Fields" || $1 == "Mapping") && block_current() == "Sysreg" {
expect_fields(2)
@@ -239,9 +239,9 @@ $1 == "EndSysreg" && block_current() == "Sysreg" {
print ""
next_bit = -1
res0 = null
res1 = null
unkn = null
res0 = $2 "_RES0"
res1 = $2 "_RES1"
unkn = $2 "_UNKN"
next
}
arch/arm64/tools/sysreg
+38 -4
View File
@@ -2387,17 +2387,40 @@ EndEnum
EndSysreg
Sysreg ID_AA64MMFR4_EL1 3 0 0 7 4
Res0 63:48
UnsignedEnum 63:60 MTEFGT
0b0000 NI
0b0001 IMP
EndEnum
UnsignedEnum 59:56 SCRX
0b0000 NI
0b0001 IMP
EndEnum
UnsignedEnum 55:52 TEV
0b0000 NI
0b0001 IMP
EndEnum
UnsignedEnum 51:48 TPS
0b0000 VAL_0000
0b0001 VAL_0001
0b0010 VAL_0010
EndEnum
UnsignedEnum 47:44 SRMASK
0b0000 NI
0b0001 IMP
0b0010 SRMASK2
EndEnum
UnsignedEnum 43:40 TLBID
0b0000 NI
0b0001 IMP
EndEnum
Res0 43:40
UnsignedEnum 39:36 E3DSE
0b0000 NI
0b0001 IMP
EndEnum
Res0 35:32
UnsignedEnum 35:32 EAESR
0b0000 NI
0b0001 IMP
EndEnum
UnsignedEnum 31:28 RMEGDI
0b0000 NI
0b0001 IMP
@@ -2411,6 +2434,7 @@ UnsignedEnum 23:20 NV_frac
0b0000 NV_NV2
0b0001 NV2_ONLY
0b0010 NV2P1
0b0011 NV3
EndEnum
UnsignedEnum 19:16 FGWTE3
0b0000 NI
@@ -4243,6 +4267,9 @@ Field 1 E2TRE
Field 0 E0HTRE
EndSysreg
Sysreg NVHCR_EL2 3 4 1 5 0
Mapping HCR_EL2
EndSysreg
Sysreg HDFGRTR2_EL2 3 4 3 1 0
Res0 63:25
@@ -4522,7 +4549,14 @@ Fields ZCR_ELx
EndSysreg
Sysreg HCRX_EL2 3 4 1 2 2
Res0 63:25
Res0 63:35
Field 34 NVnTTLBOS
Field 33 NVnTTLBIS
Field 32 NVnTTLB
Res0 31:28
Field 27 NVTGE
Field 26 SRMASKEn
Res0 25
Field 24 PACMEn
Field 23 EnFPM
Field 22 GCSEn
arch/loongarch/kvm/exit.c
+9 -10
View File
@@ -252,7 +252,7 @@ int kvm_complete_iocsr_read(struct kvm_vcpu *vcpu, struct kvm_run *run)
*gpr = *(s64 *)run->iocsr_io.data;
break;
default:
kvm_err("Bad IOCSR length: %d, addr is 0x%lx\n",
kvm_pr_unimpl("Bad IOCSR length: %d, addr is 0x%lx\n",
run->iocsr_io.len, vcpu->arch.badv);
er = EMULATE_FAIL;
break;
@@ -326,8 +326,8 @@ static int kvm_trap_handle_gspr(struct kvm_vcpu *vcpu)
/* Rollback PC only if emulation was unsuccessful */
if (er == EMULATE_FAIL) {
kvm_err("[%#lx]%s: unsupported gspr instruction 0x%08x\n",
curr_pc, __func__, inst.word);
kvm_pr_unimpl("[%#lx]%s: unsupported gspr instruction 0x%08x\n",
curr_pc, __func__, inst.word);
kvm_arch_vcpu_dump_regs(vcpu);
vcpu->arch.pc = curr_pc;
@@ -481,7 +481,6 @@ int kvm_emu_mmio_read(struct kvm_vcpu *vcpu, larch_inst inst)
srcu_read_unlock(&vcpu->kvm->srcu, idx);
if (!ret) {
kvm_complete_mmio_read(vcpu, run);
update_pc(&vcpu->arch);
vcpu->mmio_needed = 0;
return EMULATE_DONE;
}
@@ -491,7 +490,7 @@ int kvm_emu_mmio_read(struct kvm_vcpu *vcpu, larch_inst inst)
return EMULATE_DO_MMIO;
}
kvm_err("Read not supported Inst=0x%08x @%lx BadVaddr:%#lx\n",
kvm_pr_unimpl("Read not supported Inst=0x%08x @%lx BadVaddr:%#lx\n",
inst.word, vcpu->arch.pc, vcpu->arch.badv);
kvm_arch_vcpu_dump_regs(vcpu);
vcpu->mmio_needed = 0;
@@ -529,7 +528,7 @@ int kvm_complete_mmio_read(struct kvm_vcpu *vcpu, struct kvm_run *run)
*gpr = *(s64 *)run->mmio.data;
break;
default:
kvm_err("Bad MMIO length: %d, addr is 0x%lx\n",
kvm_pr_unimpl("Bad MMIO length: %d, addr is 0x%lx\n",
run->mmio.len, vcpu->arch.badv);
er = EMULATE_FAIL;
break;
@@ -656,7 +655,7 @@ int kvm_emu_mmio_write(struct kvm_vcpu *vcpu, larch_inst inst)
}
vcpu->arch.pc = curr_pc;
kvm_err("Write not supported Inst=0x%08x @%lx BadVaddr:%#lx\n",
kvm_pr_unimpl("Write not supported Inst=0x%08x @%lx BadVaddr:%#lx\n",
inst.word, vcpu->arch.pc, vcpu->arch.badv);
kvm_arch_vcpu_dump_regs(vcpu);
/* Rollback PC if emulation was unsuccessful */
@@ -748,8 +747,8 @@ static int kvm_handle_fpu_disabled(struct kvm_vcpu *vcpu, int ecode)
* treated as a reserved instruction!
* If FPU already in use, we shouldn't get this at all.
*/
if (WARN_ON(vcpu->arch.aux_inuse & KVM_LARCH_FPU)) {
kvm_err("%s internal error\n", __func__);
if (vcpu->arch.aux_inuse & KVM_LARCH_FPU) {
kvm_pr_unimpl("%s internal error\n", __func__);
run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
return RESUME_HOST;
}
@@ -943,7 +942,7 @@ static int kvm_fault_ni(struct kvm_vcpu *vcpu, int ecode)
/* Fetch the instruction */
inst = vcpu->arch.badi;
badv = vcpu->arch.badv;
kvm_err("ECode: %d PC=%#lx Inst=0x%08x BadVaddr=%#lx ESTAT=%#lx\n",
kvm_pr_unimpl("ECode: %d PC=%#lx Inst=0x%08x BadVaddr=%#lx ESTAT=%#lx\n",
ecode, vcpu->arch.pc, inst, badv, read_gcsr_estat());
kvm_arch_vcpu_dump_regs(vcpu);
kvm_queue_exception(vcpu, EXCCODE_INE, 0);
arch/loongarch/kvm/intc/dmsintc.c
+9 -9
View File
@@ -19,8 +19,7 @@ void dmsintc_inject_irq(struct kvm_vcpu *vcpu)
for (i = 0; i < 4; i++) {
old = atomic64_read(&(ds->vector_map[i]));
if (old)
vector[i] = atomic64_xchg(&(ds->vector_map[i]), 0);
vector[i] = old ? atomic64_xchg(&(ds->vector_map[i]), 0) : 0;
}
if (vector[0]) {
@@ -47,7 +46,6 @@ void dmsintc_inject_irq(struct kvm_vcpu *vcpu)
int dmsintc_deliver_msi_to_vcpu(struct kvm *kvm,
struct kvm_vcpu *vcpu, u32 vector, int level)
{
struct kvm_interrupt vcpu_irq;
struct dmsintc_state *ds = &vcpu->arch.dmsintc_state;
if (!level)
@@ -57,9 +55,11 @@ int dmsintc_deliver_msi_to_vcpu(struct kvm *kvm,
if (!ds)
return -ENODEV;
vcpu_irq.irq = INT_AVEC;
if (!kvm_guest_has_msgint(&vcpu->arch))
return -EINVAL;
set_bit(vector, (unsigned long *)&ds->vector_map);
kvm_vcpu_ioctl_interrupt(vcpu, &vcpu_irq);
kvm_queue_irq(vcpu, INT_AVEC);
kvm_vcpu_kick(vcpu);
return 0;
@@ -114,7 +114,7 @@ static int kvm_dmsintc_ctrl_access(struct kvm_device *dev,
}
break;
default:
kvm_err("%s: unknown dmsintc register, addr = %d\n", __func__, addr);
kvm_pr_unimpl("%s: unknown dmsintc register, addr = %d\n", __func__, addr);
return -ENXIO;
}
@@ -128,7 +128,7 @@ static int kvm_dmsintc_set_attr(struct kvm_device *dev,
case KVM_DEV_LOONGARCH_DMSINTC_GRP_CTRL:
return kvm_dmsintc_ctrl_access(dev, attr, true);
default:
kvm_err("%s: unknown group (%d)\n", __func__, attr->group);
kvm_pr_unimpl("%s: unknown group (%d)\n", __func__, attr->group);
return -EINVAL;
}
}
@@ -139,13 +139,13 @@ static int kvm_dmsintc_create(struct kvm_device *dev, u32 type)
struct loongarch_dmsintc *s;
if (!dev) {
kvm_err("%s: kvm_device ptr is invalid!\n", __func__);
kvm_pr_unimpl("%s: kvm_device ptr is invalid!\n", __func__);
return -EINVAL;
}
kvm = dev->kvm;
if (kvm->arch.dmsintc) {
kvm_err("%s: LoongArch DMSINTC has already been created!\n", __func__);
kvm_pr_unimpl("%s: LoongArch DMSINTC has already been created!\n", __func__);
return -EINVAL;
}
arch/loongarch/kvm/intc/eiointc.c
+18 -14
View File
@@ -17,7 +17,8 @@ static void eiointc_set_sw_coreisr(struct loongarch_eiointc *s)
if (!(s->status & BIT(EIOINTC_ENABLE_INT_ENCODE))) {
ipnum = count_trailing_zeros(ipnum);
ipnum = ipnum < 4 ? ipnum : 0;
}
} else
ipnum = ipnum < LOONGSON_IP_NUM ? ipnum : 0;
cpuid = ((u8 *)s->coremap)[irq];
vcpu = kvm_get_vcpu_by_cpuid(s->kvm, cpuid);
@@ -36,18 +37,18 @@ static void eiointc_update_irq(struct loongarch_eiointc *s, int irq, int level)
{
int ipnum, cpu, found;
struct kvm_vcpu *vcpu;
struct kvm_interrupt vcpu_irq;
ipnum = (s->ipmap >> (irq / 32 * 8)) & 0xff;
if (!(s->status & BIT(EIOINTC_ENABLE_INT_ENCODE))) {
ipnum = count_trailing_zeros(ipnum);
ipnum = ipnum < 4 ? ipnum : 0;
}
} else
ipnum = ipnum < LOONGSON_IP_NUM ? ipnum : 0;
cpu = s->sw_coremap[irq];
vcpu = kvm_get_vcpu_by_id(s->kvm, cpu);
if (unlikely(vcpu == NULL)) {
kvm_err("%s: invalid target cpu: %d\n", __func__, cpu);
kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu);
return;
}
@@ -67,8 +68,11 @@ static void eiointc_update_irq(struct loongarch_eiointc *s, int irq, int level)
if (found < EIOINTC_IRQS)
return; /* other irq is handling, needn't update parent irq */
vcpu_irq.irq = level ? (INT_HWI0 + ipnum) : -(INT_HWI0 + ipnum);
kvm_vcpu_ioctl_interrupt(vcpu, &vcpu_irq);
if (level)
kvm_queue_irq(vcpu, INT_HWI0 + ipnum);
else
kvm_dequeue_irq(vcpu, INT_HWI0 + ipnum);
kvm_vcpu_kick(vcpu);
}
static inline void eiointc_update_sw_coremap(struct loongarch_eiointc *s,
@@ -165,12 +169,12 @@ static int kvm_eiointc_read(struct kvm_vcpu *vcpu,
struct loongarch_eiointc *eiointc = vcpu->kvm->arch.eiointc;
if (!eiointc) {
kvm_err("%s: eiointc irqchip not valid!\n", __func__);
kvm_pr_unimpl("%s: eiointc irqchip not valid!\n", __func__);
return 0;
}
if (addr & (len - 1)) {
kvm_err("%s: eiointc not aligned addr %llx len %d\n", __func__, addr, len);
kvm_pr_unimpl("%s: eiointc not aligned addr %llx len %d\n", __func__, addr, len);
return 0;
}
@@ -297,12 +301,12 @@ static int kvm_eiointc_write(struct kvm_vcpu *vcpu,
struct loongarch_eiointc *eiointc = vcpu->kvm->arch.eiointc;
if (!eiointc) {
kvm_err("%s: eiointc irqchip not valid!\n", __func__);
kvm_pr_unimpl("%s: eiointc irqchip not valid!\n", __func__);
return 0;
}
if (addr & (len - 1)) {
kvm_err("%s: eiointc not aligned addr %llx len %d\n", __func__, addr, len);
kvm_pr_unimpl("%s: eiointc not aligned addr %llx len %d\n", __func__, addr, len);
return 0;
}
@@ -345,7 +349,7 @@ static int kvm_eiointc_virt_read(struct kvm_vcpu *vcpu,
struct loongarch_eiointc *eiointc = vcpu->kvm->arch.eiointc;
if (!eiointc) {
kvm_err("%s: eiointc irqchip not valid!\n", __func__);
kvm_pr_unimpl("%s: eiointc irqchip not valid!\n", __func__);
return 0;
}
@@ -375,7 +379,7 @@ static int kvm_eiointc_virt_write(struct kvm_vcpu *vcpu,
struct loongarch_eiointc *eiointc = vcpu->kvm->arch.eiointc;
if (!eiointc) {
kvm_err("%s: eiointc irqchip not valid!\n", __func__);
kvm_pr_unimpl("%s: eiointc irqchip not valid!\n", __func__);
return 0;
}
@@ -502,7 +506,7 @@ static int kvm_eiointc_regs_access(struct kvm_device *dev,
p = (void *)s->coremap + offset * 4;
break;
default:
kvm_err("%s: unknown eiointc register, addr = %d\n", __func__, addr);
kvm_pr_unimpl("%s: unknown eiointc register, addr = %d\n", __func__, addr);
return -EINVAL;
}
@@ -546,7 +550,7 @@ static int kvm_eiointc_sw_status_access(struct kvm_device *dev,
p = &s->status;
break;
default:
kvm_err("%s: unknown eiointc register, addr = %d\n", __func__, addr);
kvm_pr_unimpl("%s: unknown eiointc register, addr = %d\n", __func__, addr);
return -EINVAL;
}
spin_lock_irqsave(&s->lock, flags);
arch/loongarch/kvm/intc/ipi.c
+37 -40
View File
@@ -10,16 +10,15 @@
static void ipi_set(struct kvm_vcpu *vcpu, uint32_t data)
{
uint32_t status;
struct kvm_interrupt irq;
spin_lock(&vcpu->arch.ipi_state.lock);
status = vcpu->arch.ipi_state.status;
vcpu->arch.ipi_state.status |= data;
spin_unlock(&vcpu->arch.ipi_state.lock);
if ((status == 0) && data) {
irq.irq = LARCH_INT_IPI;
kvm_vcpu_ioctl_interrupt(vcpu, &irq);
kvm_queue_irq(vcpu, LARCH_INT_IPI);
kvm_vcpu_kick(vcpu);
}
spin_unlock(&vcpu->arch.ipi_state.lock);
}
static void ipi_send(struct kvm *kvm, uint64_t data)
@@ -30,7 +29,7 @@ static void ipi_send(struct kvm *kvm, uint64_t data)
cpu = ((data & 0xffffffff) >> 16) & 0x3ff;
vcpu = kvm_get_vcpu_by_cpuid(kvm, cpu);
if (unlikely(vcpu == NULL)) {
kvm_err("%s: invalid target cpu: %d\n", __func__, cpu);
kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu);
return;
}
@@ -40,16 +39,15 @@ static void ipi_send(struct kvm *kvm, uint64_t data)
static void ipi_clear(struct kvm_vcpu *vcpu, uint64_t data)
{
uint32_t status;
struct kvm_interrupt irq;
spin_lock(&vcpu->arch.ipi_state.lock);
vcpu->arch.ipi_state.status &= ~data;
status = vcpu->arch.ipi_state.status;
spin_unlock(&vcpu->arch.ipi_state.lock);
if (status == 0) {
irq.irq = -LARCH_INT_IPI;
kvm_vcpu_ioctl_interrupt(vcpu, &irq);
kvm_dequeue_irq(vcpu, LARCH_INT_IPI);
kvm_vcpu_kick(vcpu);
}
spin_unlock(&vcpu->arch.ipi_state.lock);
}
static uint64_t read_mailbox(struct kvm_vcpu *vcpu, int offset, int len)
@@ -67,11 +65,8 @@ static uint64_t read_mailbox(struct kvm_vcpu *vcpu, int offset, int len)
return data & 0xffff;
case 4:
return data & 0xffffffff;
case 8:
return data;
default:
kvm_err("%s: unknown data len: %d\n", __func__, len);
return 0;
return data;
}
}
@@ -92,11 +87,9 @@ static void write_mailbox(struct kvm_vcpu *vcpu, int offset, uint64_t data, int
case 4:
*(unsigned int *)pbuf = (unsigned int)data;
break;
case 8:
default:
*(unsigned long *)pbuf = (unsigned long)data;
break;
default:
kvm_err("%s: unknown data len: %d\n", __func__, len);
}
spin_unlock(&vcpu->arch.ipi_state.lock);
}
@@ -110,7 +103,7 @@ static int mail_send(struct kvm *kvm, uint64_t data)
cpu = ((data & 0xffffffff) >> 16) & 0x3ff;
vcpu = kvm_get_vcpu_by_cpuid(kvm, cpu);
if (unlikely(vcpu == NULL)) {
kvm_err("%s: invalid target cpu: %d\n", __func__, cpu);
kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu);
return 0;
}
mailbox = ((data & 0xffffffff) >> 2) & 0x7;
@@ -144,7 +137,7 @@ static int send_ipi_data(struct kvm_vcpu *vcpu, gpa_t addr, uint64_t data)
ret = kvm_io_bus_read(vcpu, KVM_IOCSR_BUS, addr, 4, &val);
srcu_read_unlock(&vcpu->kvm->srcu, idx);
if (unlikely(ret)) {
kvm_err("%s: : read data from addr %llx failed\n", __func__, addr);
kvm_pr_unimpl("%s: : read data from addr %llx failed\n", __func__, addr);
return 0;
}
/* Construct the mask by scanning the bit 27-30 */
@@ -160,7 +153,7 @@ static int send_ipi_data(struct kvm_vcpu *vcpu, gpa_t addr, uint64_t data)
ret = kvm_io_bus_write(vcpu, KVM_IOCSR_BUS, addr, 4, &val);
srcu_read_unlock(&vcpu->kvm->srcu, idx);
if (unlikely(ret))
kvm_err("%s: : write data to addr %llx failed\n", __func__, addr);
kvm_pr_unimpl("%s: : write data to addr %llx failed\n", __func__, addr);
return 0;
}
@@ -173,7 +166,7 @@ static int any_send(struct kvm *kvm, uint64_t data)
cpu = ((data & 0xffffffff) >> 16) & 0x3ff;
vcpu = kvm_get_vcpu_by_cpuid(kvm, cpu);
if (unlikely(vcpu == NULL)) {
kvm_err("%s: invalid target cpu: %d\n", __func__, cpu);
kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu);
return 0;
}
offset = data & 0xffff;
@@ -186,8 +179,13 @@ static int loongarch_ipi_readl(struct kvm_vcpu *vcpu, gpa_t addr, int len, void
uint32_t offset;
uint64_t res = 0;
offset = (uint32_t)(addr & 0x1ff);
WARN_ON_ONCE(offset & (len - 1));
if (addr & (len - 1)) {
*(uint64_t *)val = res;
kvm_pr_unimpl("%s: ipi not aligned addr %llx len %d\n", __func__, addr, len);
return 0;
}
offset = addr - IOCSR_IPI_BASE;
switch (offset) {
case IOCSR_IPI_STATUS:
@@ -204,15 +202,10 @@ static int loongarch_ipi_readl(struct kvm_vcpu *vcpu, gpa_t addr, int len, void
case IOCSR_IPI_CLEAR:
break;
case IOCSR_IPI_BUF_20 ... IOCSR_IPI_BUF_38 + 7:
if (offset + len > IOCSR_IPI_BUF_38 + 8) {
kvm_err("%s: invalid offset or len: offset = %d, len = %d\n",
__func__, offset, len);
break;
}
res = read_mailbox(vcpu, offset, len);
break;
default:
kvm_err("%s: unknown addr: %llx\n", __func__, addr);
kvm_pr_unimpl("%s: unknown addr: %llx\n", __func__, addr);
break;
}
*(uint64_t *)val = res;
@@ -227,8 +220,12 @@ static int loongarch_ipi_writel(struct kvm_vcpu *vcpu, gpa_t addr, int len, cons
data = *(uint64_t *)val;
offset = (uint32_t)(addr & 0x1ff);
WARN_ON_ONCE(offset & (len - 1));
if (addr & (len - 1)) {
kvm_pr_unimpl("%s: ipi not aligned addr %llx len %d\n", __func__, addr, len);
return 0;
}
offset = addr - IOCSR_IPI_BASE;
switch (offset) {
case IOCSR_IPI_STATUS:
@@ -247,8 +244,8 @@ static int loongarch_ipi_writel(struct kvm_vcpu *vcpu, gpa_t addr, int len, cons
break;
case IOCSR_IPI_BUF_20 ... IOCSR_IPI_BUF_38 + 7:
if (offset + len > IOCSR_IPI_BUF_38 + 8) {
kvm_err("%s: invalid offset or len: offset = %d, len = %d\n",
__func__, offset, len);
kvm_pr_unimpl("%s: invalid offset or len: offset = %d, len = %d\n",
__func__, offset, len);
break;
}
write_mailbox(vcpu, offset, data, len);
@@ -263,7 +260,7 @@ static int loongarch_ipi_writel(struct kvm_vcpu *vcpu, gpa_t addr, int len, cons
any_send(vcpu->kvm, data);
break;
default:
kvm_err("%s: unknown addr: %llx\n", __func__, addr);
kvm_pr_unimpl("%s: unknown addr: %llx\n", __func__, addr);
break;
}
@@ -306,7 +303,7 @@ static int kvm_ipi_regs_access(struct kvm_device *dev,
vcpu = kvm_get_vcpu_by_id(dev->kvm, cpu);
if (unlikely(vcpu == NULL)) {
kvm_err("%s: invalid target cpu: %d\n", __func__, cpu);
kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu);
return -EINVAL;
}
@@ -340,7 +337,7 @@ static int kvm_ipi_regs_access(struct kvm_device *dev,
len = 8;
break;
default:
kvm_err("%s: unknown ipi register, addr = %d\n", __func__, addr);
kvm_pr_unimpl("%s: unknown ipi register, addr = %d\n", __func__, addr);
return -EINVAL;
}
@@ -374,7 +371,7 @@ static int kvm_ipi_get_attr(struct kvm_device *dev,
case KVM_DEV_LOONGARCH_IPI_GRP_REGS:
return kvm_ipi_regs_access(dev, attr, false);
default:
kvm_err("%s: unknown group (%d)\n", __func__, attr->group);
kvm_pr_unimpl("%s: unknown group (%d)\n", __func__, attr->group);
return -EINVAL;
}
}
@@ -386,7 +383,7 @@ static int kvm_ipi_set_attr(struct kvm_device *dev,
case KVM_DEV_LOONGARCH_IPI_GRP_REGS:
return kvm_ipi_regs_access(dev, attr, true);
default:
kvm_err("%s: unknown group (%d)\n", __func__, attr->group);
kvm_pr_unimpl("%s: unknown group (%d)\n", __func__, attr->group);
return -EINVAL;
}
}
@@ -399,13 +396,13 @@ static int kvm_ipi_create(struct kvm_device *dev, u32 type)
struct loongarch_ipi *s;
if (!dev) {
kvm_err("%s: kvm_device ptr is invalid!\n", __func__);
kvm_pr_unimpl("%s: kvm_device ptr is invalid!\n", __func__);
return -EINVAL;
}
kvm = dev->kvm;
if (kvm->arch.ipi) {
kvm_err("%s: LoongArch IPI has already been created!\n", __func__);
kvm_pr_unimpl("%s: LoongArch IPI has already been created!\n", __func__);
return -EINVAL;
}
@@ -425,7 +422,7 @@ static int kvm_ipi_create(struct kvm_device *dev, u32 type)
ret = kvm_io_bus_register_dev(kvm, KVM_IOCSR_BUS, IOCSR_IPI_BASE, IOCSR_IPI_SIZE, device);
mutex_unlock(&kvm->slots_lock);
if (ret < 0) {
kvm_err("%s: Initialize IOCSR dev failed, ret = %d\n", __func__, ret);
kvm_pr_unimpl("%s: Initialize IOCSR dev failed, ret = %d\n", __func__, ret);
goto err;
}
arch/loongarch/kvm/intc/pch_pic.c
+6 -6
View File
@@ -151,12 +151,12 @@ static int kvm_pch_pic_read(struct kvm_vcpu *vcpu,
struct loongarch_pch_pic *s = vcpu->kvm->arch.pch_pic;
if (!s) {
kvm_err("%s: pch pic irqchip not valid!\n", __func__);
kvm_pr_unimpl("%s: pch pic irqchip not valid!\n", __func__);
return ret;
}
if (addr & (len - 1)) {
kvm_err("%s: pch pic not aligned addr %llx len %d\n", __func__, addr, len);
kvm_pr_unimpl("%s: pch pic not aligned addr %llx len %d\n", __func__, addr, len);
return ret;
}
@@ -250,12 +250,12 @@ static int kvm_pch_pic_write(struct kvm_vcpu *vcpu,
struct loongarch_pch_pic *s = vcpu->kvm->arch.pch_pic;
if (!s) {
kvm_err("%s: pch pic irqchip not valid!\n", __func__);
kvm_pr_unimpl("%s: pch pic irqchip not valid!\n", __func__);
return ret;
}
if (addr & (len - 1)) {
kvm_err("%s: pch pic not aligned addr %llx len %d\n", __func__, addr, len);
kvm_pr_unimpl("%s: pch pic not aligned addr %llx len %d\n", __func__, addr, len);
return ret;
}
@@ -390,13 +390,13 @@ static int kvm_pch_pic_set_attr(struct kvm_device *dev,
return -EFAULT;
if (!dev->kvm->arch.pch_pic) {
kvm_err("%s: please create pch_pic irqchip first!\n", __func__);
kvm_pr_unimpl("%s: pch_pic irqchip not created\n", __func__);
return -ENODEV;
}
return kvm_pch_pic_init(dev, addr);
default:
kvm_err("%s: unknown group (%d) attr (%lld)\n", __func__, attr->group,
kvm_pr_unimpl("%s: unknown group (%d) attr (%lld)\n", __func__, attr->group,
attr->attr);
return -EINVAL;
}
arch/loongarch/kvm/timer.c
+14
View File
@@ -119,10 +119,24 @@ void kvm_restore_timer(struct kvm_vcpu *vcpu)
delta = 0;
now = ktime_get();
expire = vcpu->arch.expire;
if (!expire) {
/*
* vcpu->arch.expire is host-internal and is not migrated,
* so it is 0 after migration. Reload the remaining countdown
* from the migrated TVAL. This covers both one-shot and
* periodic timers.
*/
if (ticks < cfg)
delta = tick_to_ns(vcpu, ticks);
expire = ktime_add_ns(now, delta);
}
if (ktime_before(now, expire))
delta = ktime_to_tick(vcpu, ktime_sub(expire, now));
else if (cfg & CSR_TCFG_PERIOD) {
period = cfg & CSR_TCFG_VAL;
if (!period)
period = 1;
delta = ktime_to_tick(vcpu, ktime_sub(now, expire));
delta = period - (delta % period);
arch/loongarch/kvm/vcpu.c
+5 -2
View File
@@ -300,10 +300,10 @@ static int kvm_pre_enter_guest(struct kvm_vcpu *vcpu)
*/
local_irq_disable();
kvm_deliver_exception(vcpu);
kvm_check_vpid(vcpu);
/* Make sure the vcpu mode has been written */
smp_store_mb(vcpu->mode, IN_GUEST_MODE);
kvm_deliver_intr(vcpu);
kvm_check_vpid(vcpu);
/*
* Called after function kvm_check_vpid()
@@ -1465,6 +1465,9 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq)
if (vector >= EXCCODE_INT_NUM)
return -EINVAL;
if (kvm_arch_irqchip_in_kernel(vcpu->kvm))
return -EINVAL;
if (!kvm_guest_has_msgint(&vcpu->arch) && (vector == INT_AVEC))
return -EINVAL;
@@ -1473,7 +1476,7 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq)
else if (intr < 0)
kvm_dequeue_irq(vcpu, -intr);
else {
kvm_err("%s: invalid interrupt ioctl %d\n", __func__, irq->irq);
kvm_pr_unimpl("%s: invalid interrupt ioctl %d\n", __func__, irq->irq);
return -EINVAL;
}
arch/loongarch/kvm/vm.c
+4
View File
@@ -122,6 +122,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_IOEVENTFD:
case KVM_CAP_MP_STATE:
case KVM_CAP_SET_GUEST_DEBUG:
case KVM_CAP_VCPU_ATTRIBUTES:
r = 1;
break;
case KVM_CAP_NR_VCPUS:
@@ -136,6 +137,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_NR_MEMSLOTS:
r = KVM_USER_MEM_SLOTS;
break;
case KVM_CAP_STEAL_TIME:
r = kvm_pvtime_supported();
break;
default:
r = 0;
break;
arch/powerpc/kvm/book3s_pr.c
+4 -7
View File
@@ -639,27 +639,24 @@ static void kvmppc_set_pvr_pr(struct kvm_vcpu *vcpu, u32 pvr)
*/
static void kvmppc_patch_dcbz(struct kvm_vcpu *vcpu, struct kvmppc_pte *pte)
{
struct kvm_host_map map;
u64 hpage_offset;
u32 *page;
int i, r;
int i;
r = kvm_vcpu_map(vcpu, pte->raddr >> PAGE_SHIFT, &map);
if (r)
CLASS(kvm_vcpu_map_local, m)(vcpu, pte->raddr >> PAGE_SHIFT);
if (m.ret)
return;
hpage_offset = pte->raddr & ~PAGE_MASK;
hpage_offset &= ~0xFFFULL;
hpage_offset /= 4;
page = map.hva;
page = m.map.hva;
/* patch dcbz into reserved instruction, so we trap */
for (i=hpage_offset; i < hpage_offset + (HW_PAGE_SIZE / 4); i++)
if ((be32_to_cpu(page[i]) & 0xff0007ff) == INS_DCBZ)
page[i] &= cpu_to_be32(0xfffffff7);
kvm_vcpu_unmap(vcpu, &map);
}
static bool kvmppc_visible_gpa(struct kvm_vcpu *vcpu, gpa_t gpa)
arch/riscv/include/asm/csr.h
+1
View File
@@ -121,6 +121,7 @@
#define EXC_INST_PAGE_FAULT 12
#define EXC_LOAD_PAGE_FAULT 13
#define EXC_STORE_PAGE_FAULT 15
#define EXC_SOFTWARE_CHECK 18
#define EXC_INST_GUEST_PAGE_FAULT 20
#define EXC_LOAD_GUEST_PAGE_FAULT 21
#define EXC_VIRTUAL_INST_FAULT 22
arch/riscv/include/asm/kvm_aia.h
+3
View File
@@ -165,6 +165,9 @@ int kvm_riscv_aia_alloc_hgei(int cpu, struct kvm_vcpu *owner,
void __iomem **hgei_va, phys_addr_t *hgei_pa);
void kvm_riscv_aia_free_hgei(int cpu, int hgei);
void kvm_riscv_aia_pm_exit(void);
void kvm_riscv_aia_pm_enter(void);
void kvm_riscv_aia_enable(void);
void kvm_riscv_aia_disable(void);
int kvm_riscv_aia_init(void);
arch/riscv/include/asm/kvm_gstage.h
+6 -3
View File
@@ -64,9 +64,9 @@ int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage,
bool page_rdonly, bool page_exec,
struct kvm_gstage_mapping *out_map);
int kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage,
struct kvm_mmu_memory_cache *pcache,
gpa_t addr, u32 target_level, bool flush);
bool kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage,
struct kvm_mmu_memory_cache *pcache,
gpa_t addr, u32 target_level, bool flush);
enum kvm_riscv_gstage_op {
GSTAGE_OP_NOP = 0, /* Nothing */
@@ -82,6 +82,9 @@ bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage,
bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end);
bool kvm_riscv_gstage_wp_pt_masked(struct kvm_gstage *gstage, gfn_t base_gfn,
unsigned long mask);
void kvm_riscv_gstage_mode_detect(void);
static inline unsigned long kvm_riscv_gstage_mode(unsigned long pgd_levels)
arch/riscv/include/asm/kvm_host.h
+10
View File
@@ -86,6 +86,7 @@ struct kvm_arch {
pgd_t *pgd;
phys_addr_t pgd_phys;
unsigned long pgd_levels;
struct kvm_mmu_memory_cache pgd_split_page_cache;
/* Guest Timer */
struct kvm_guest_timer timer;
@@ -163,6 +164,10 @@ struct kvm_vcpu_smstateen_csr {
unsigned long sstateen0;
};
struct kvm_vcpu_zicfiss_csr {
unsigned long ssp;
};
struct kvm_vcpu_reset_state {
spinlock_t lock;
unsigned long pc;
@@ -203,6 +208,9 @@ struct kvm_vcpu_arch {
/* CPU Smstateen CSR context of Guest VCPU */
struct kvm_vcpu_smstateen_csr smstateen_csr;
/* CPU Zicfiss CSR context of Guest VCPU */
struct kvm_vcpu_zicfiss_csr zicfiss_csr;
/* CPU reset state of Guest VCPU */
struct kvm_vcpu_reset_state reset_state;
@@ -282,6 +290,8 @@ static inline bool kvm_arch_pmi_in_guest(struct kvm_vcpu *vcpu)
static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
void kvm_riscv_clear_former_vcpu(void);
int kvm_riscv_setup_default_irq_routing(struct kvm *kvm, u32 lines);
void __kvm_riscv_unpriv_trap(void);
arch/riscv/include/asm/kvm_vcpu_insn.h
+2 -2
View File
@@ -38,10 +38,10 @@ int kvm_riscv_vcpu_virtual_insn(struct kvm_vcpu *vcpu, struct kvm_run *run,
struct kvm_cpu_trap *trap);
int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run,
unsigned long fault_addr,
gpa_t fault_addr,
unsigned long htinst);
int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run,
unsigned long fault_addr,
gpa_t fault_addr,
unsigned long htinst);
int kvm_riscv_vcpu_mmio_return(struct kvm_vcpu *vcpu, struct kvm_run *run);
arch/riscv/include/asm/kvm_vcpu_sbi.h
+4
View File
@@ -60,6 +60,9 @@ struct kvm_vcpu_sbi_extension {
void (*reset)(struct kvm_vcpu *vcpu);
/* Allow the extension to correct its parameters before the first run */
void (*validate)(struct kvm_vcpu *vcpu);
unsigned long state_reg_subtype;
unsigned long (*get_state_reg_count)(struct kvm_vcpu *vcpu);
int (*get_state_reg_id)(struct kvm_vcpu *vcpu, int index, u64 *reg_id);
@@ -93,6 +96,7 @@ int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run);
void kvm_riscv_vcpu_sbi_init(struct kvm_vcpu *vcpu);
void kvm_riscv_vcpu_sbi_deinit(struct kvm_vcpu *vcpu);
void kvm_riscv_vcpu_sbi_reset(struct kvm_vcpu *vcpu);
void kvm_riscv_vcpu_sbi_validate(struct kvm_vcpu *vcpu);
#ifdef CONFIG_RISCV_SBI_V01
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01;
arch/riscv/include/asm/kvm_vcpu_vector.h
+24
View File
@@ -35,6 +35,22 @@ void kvm_riscv_vcpu_host_vector_save(struct kvm_cpu_context *cntx);
void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx);
int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu);
void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu);
void kvm_riscv_register_vctx_callback(void (*func)(void));
void kvm_riscv_unregister_vctx_callback(void);
void kvm_riscv_vcpu_flush_vector(void);
static inline void kvm_riscv_v_init(void)
{
if (has_vector())
kvm_riscv_register_vctx_callback(&kvm_riscv_vcpu_flush_vector);
}
static inline void kvm_riscv_v_exit(void)
{
if (has_vector())
kvm_riscv_unregister_vctx_callback();
}
#else
struct kvm_cpu_context;
@@ -69,6 +85,14 @@ static inline int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu)
static inline void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu)
{
}
static inline void kvm_riscv_v_init(void)
{
}
static inline void kvm_riscv_v_exit(void)
{
}
#endif
int kvm_riscv_vcpu_get_reg_vector(struct kvm_vcpu *vcpu,
arch/riscv/include/asm/processor.h
+8
View File
@@ -67,6 +67,12 @@ struct pt_regs;
* - bit 0: indicates whether the in-kernel Vector context is active. The
* activation of this state disables the preemption. On a non-RT kernel, it
* also disable bh.
* - bit 1: tells kvm that the vcpu process has guest context saved in vcpu's
* context memory and need to be restore upon returing back to the guest.
* - bit 2: represents that the vector context has now loaded and belongs to
* the guest kernel. Any non-scheduler context saving routing needs to save
* the register file to vcpu's context memory. The bit is set upon returing
* back to the guest and cleared after loading the host's vector context.
* - bits 8: is used for tracking preemptible kernel-mode Vector, when
* RISCV_ISA_V_PREEMPTIVE is enabled. Calling kernel_vector_begin() does not
* disable the preemption if the thread's kernel_vstate.datap is allocated.
@@ -97,6 +103,8 @@ struct pt_regs;
#define RISCV_V_CTX_UNIT_DEPTH 0x00010000
#define RISCV_KERNEL_MODE_V 0x00000001
#define RISCV_V_VCPU_NEED_RESTORE 0x00000002
#define RISCV_V_VCPU_CTX 0x00000004
#define RISCV_PREEMPT_V 0x00000100
#define RISCV_PREEMPT_V_DIRTY 0x80000000
#define RISCV_PREEMPT_V_NEED_RESTORE 0x40000000
arch/riscv/include/asm/simd.h
+3 -13
View File
@@ -36,20 +36,10 @@ static __must_check inline bool may_use_simd(void)
/*
* Nesting is achieved in preempt_v by spreading the control for
* preemptible and non-preemptible kernel-mode Vector into two fields.
* Always try to match with preempt_v if kernel V-context exists. Then,
* fallback to check non preempt_v if nesting happens, or if the config
* is not set.
* Only non-preempt_v can nest on top of preempt_v, if non-preempt_v is
* unavailable, then preempt_v is not allowed.
*/
if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && current->thread.kernel_vstate.datap) {
if (!riscv_preempt_v_started(current))
return true;
}
/*
* Non-preemptible kernel-mode Vector temporarily disables bh. So we
* must not return true on irq_disabled(). Otherwise we would fail the
* lockdep check calling local_bh_enable()
*/
return !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V);
return !(riscv_v_flags() & RISCV_KERNEL_MODE_V);
}
#else /* ! CONFIG_RISCV_ISA_V */
arch/riscv/include/asm/vector.h
+5
View File
@@ -58,6 +58,11 @@ static inline u32 riscv_v_flags(void)
return READ_ONCE(current->thread.riscv_v_flags);
}
static inline void riscv_v_flags_set(u32 flags)
{
WRITE_ONCE(current->thread.riscv_v_flags, flags);
}
static __always_inline bool has_vector(void)
{
return riscv_has_extension_unlikely(RISCV_ISA_EXT_ZVE32X);
arch/riscv/include/uapi/asm/kvm.h
+13
View File
@@ -102,6 +102,11 @@ struct kvm_riscv_smstateen_csr {
unsigned long sstateen0;
};
/* Zicfiss CSR for KVM_GET_ONE_REG and KVM_SET_ONE_REG */
struct kvm_riscv_zicfiss_csr {
unsigned long ssp;
};
/* TIMER registers for KVM_GET_ONE_REG and KVM_SET_ONE_REG */
struct kvm_riscv_timer {
__u64 frequency;
@@ -199,6 +204,8 @@ enum KVM_RISCV_ISA_EXT_ID {
KVM_RISCV_ISA_EXT_ZCLSD,
KVM_RISCV_ISA_EXT_ZILSD,
KVM_RISCV_ISA_EXT_ZALASR,
KVM_RISCV_ISA_EXT_ZICFILP,
KVM_RISCV_ISA_EXT_ZICFISS,
KVM_RISCV_ISA_EXT_MAX,
};
@@ -240,6 +247,9 @@ struct kvm_riscv_sbi_fwft_feature {
struct kvm_riscv_sbi_fwft {
struct kvm_riscv_sbi_fwft_feature misaligned_deleg;
struct kvm_riscv_sbi_fwft_feature pointer_masking;
struct kvm_riscv_sbi_fwft_feature pte_ad_hw_updating;
struct kvm_riscv_sbi_fwft_feature landing_pad;
struct kvm_riscv_sbi_fwft_feature shadow_stack;
};
/* If you need to interpret the index values, here is the key: */
@@ -263,12 +273,15 @@ struct kvm_riscv_sbi_fwft {
#define KVM_REG_RISCV_CSR_GENERAL (0x0 << KVM_REG_RISCV_SUBTYPE_SHIFT)
#define KVM_REG_RISCV_CSR_AIA (0x1 << KVM_REG_RISCV_SUBTYPE_SHIFT)
#define KVM_REG_RISCV_CSR_SMSTATEEN (0x2 << KVM_REG_RISCV_SUBTYPE_SHIFT)
#define KVM_REG_RISCV_CSR_ZICFISS (0x3 << KVM_REG_RISCV_SUBTYPE_SHIFT)
#define KVM_REG_RISCV_CSR_REG(name) \
(offsetof(struct kvm_riscv_csr, name) / sizeof(unsigned long))
#define KVM_REG_RISCV_CSR_AIA_REG(name) \
(offsetof(struct kvm_riscv_aia_csr, name) / sizeof(unsigned long))
#define KVM_REG_RISCV_CSR_SMSTATEEN_REG(name) \
(offsetof(struct kvm_riscv_smstateen_csr, name) / sizeof(unsigned long))
#define KVM_REG_RISCV_CSR_ZICFISS_REG(name) \
(offsetof(struct kvm_riscv_zicfiss_csr, name) / sizeof(unsigned long))
/* Timer registers are mapped as type 4 */
#define KVM_REG_RISCV_TIMER (0x04 << KVM_REG_RISCV_TYPE_SHIFT)
arch/riscv/kernel/kernel_mode_vector.c
+63 -25
View File
@@ -10,18 +10,34 @@
#include <linux/percpu.h>
#include <linux/preempt.h>
#include <linux/types.h>
#include <linux/kvm_types.h>
#include <asm/vector.h>
#include <asm/kvm_vcpu_vector.h>
#include <asm/switch_to.h>
#include <asm/simd.h>
#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
#include <asm/asm-prototypes.h>
#endif
static inline void riscv_v_flags_set(u32 flags)
static void (* __rcu kvm_flush_vector_ctx_callback)(void);
void kvm_riscv_register_vctx_callback(void (*func)(void))
{
WRITE_ONCE(current->thread.riscv_v_flags, flags);
if (WARN_ON_ONCE(rcu_access_pointer(kvm_flush_vector_ctx_callback)))
return;
rcu_assign_pointer(kvm_flush_vector_ctx_callback, func);
}
EXPORT_SYMBOL_GPL(kvm_riscv_register_vctx_callback);
void kvm_riscv_unregister_vctx_callback(void)
{
rcu_assign_pointer(kvm_flush_vector_ctx_callback, NULL);
synchronize_rcu();
}
EXPORT_SYMBOL_GPL(kvm_riscv_unregister_vctx_callback);
static inline void riscv_v_start(u32 flags)
{
@@ -55,13 +71,16 @@ void get_cpu_vector_context(void)
* disable softirqs so it is impossible for softirqs to nest
* get_cpu_vector_context() when kernel is actively using Vector.
*/
if (!IS_ENABLED(CONFIG_PREEMPT_RT))
local_bh_disable();
else
if (!IS_ENABLED(CONFIG_PREEMPT_RT)) {
if (!irqs_disabled())
local_bh_disable();
} else {
preempt_disable();
}
riscv_v_start(RISCV_KERNEL_MODE_V);
}
EXPORT_SYMBOL_FOR_KVM(get_cpu_vector_context);
/*
* Release the CPU vector context.
@@ -74,10 +93,29 @@ void put_cpu_vector_context(void)
{
riscv_v_stop(RISCV_KERNEL_MODE_V);
if (!IS_ENABLED(CONFIG_PREEMPT_RT))
local_bh_enable();
else
if (!IS_ENABLED(CONFIG_PREEMPT_RT)) {
if (!irqs_disabled())
local_bh_enable();
} else {
preempt_enable();
}
}
EXPORT_SYMBOL_FOR_KVM(put_cpu_vector_context);
static void __riscv_flush_vector_context(void)
{
void (*vcpu_flush_v_callback)(void);
if (riscv_v_flags() & RISCV_V_VCPU_CTX) {
rcu_read_lock();
vcpu_flush_v_callback = rcu_dereference(kvm_flush_vector_ctx_callback);
vcpu_flush_v_callback();
rcu_read_unlock();
return;
}
riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
riscv_v_vstate_set_restore(current, task_pt_regs(current));
}
#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
@@ -121,9 +159,9 @@ static int riscv_v_stop_kernel_context(void)
return 0;
}
static int riscv_v_start_kernel_context(bool *is_nested)
static int riscv_v_start_kernel_context(void)
{
struct __riscv_v_ext_state *kvstate, *uvstate;
struct __riscv_v_ext_state *kvstate;
kvstate = &current->thread.kernel_vstate;
if (!kvstate->datap)
@@ -131,7 +169,6 @@ static int riscv_v_start_kernel_context(bool *is_nested)
if (riscv_preempt_v_started(current)) {
WARN_ON(riscv_v_ctx_get_depth() == 0);
*is_nested = true;
get_cpu_vector_context();
if (riscv_preempt_v_dirty(current)) {
__riscv_v_vstate_save(kvstate, kvstate->datap);
@@ -142,12 +179,18 @@ static int riscv_v_start_kernel_context(bool *is_nested)
}
/* Transfer the ownership of V from user to kernel, then save */
riscv_v_start(RISCV_PREEMPT_V | RISCV_PREEMPT_V_DIRTY);
if (__riscv_v_vstate_check(task_pt_regs(current)->status, DIRTY)) {
uvstate = &current->thread.vstate;
__riscv_v_vstate_save(uvstate, uvstate->datap);
}
riscv_preempt_v_clear_dirty(current);
get_cpu_vector_context();
__riscv_flush_vector_context();
put_cpu_vector_context();
/*
* A voluntary context switch caused by put_cpu_vector_context() can
* raise the NEED_RESTORE flag if preempt_v starts too early due to a
* failed risv_v_is_on() check.
*
* This causes the next context_nesting_end pollute the v-reg from
* the stale context memory in kernel-mode vector.
*/
riscv_v_start(RISCV_PREEMPT_V);
return 0;
}
@@ -187,7 +230,7 @@ asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs)
}
}
#else
#define riscv_v_start_kernel_context(nested) (-ENOENT)
#define riscv_v_start_kernel_context() (-ENOENT)
#define riscv_v_stop_kernel_context() (-ENOENT)
#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */
@@ -206,21 +249,16 @@ asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs)
*/
void kernel_vector_begin(void)
{
bool nested = false;
if (WARN_ON(!(has_vector() || has_xtheadvector())))
return;
BUG_ON(!may_use_simd());
if (riscv_v_start_kernel_context(&nested)) {
if (riscv_v_start_kernel_context()) {
get_cpu_vector_context();
riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
__riscv_flush_vector_context();
}
if (!nested)
riscv_v_vstate_set_restore(current, task_pt_regs(current));
riscv_v_enable();
}
EXPORT_SYMBOL_GPL(kernel_vector_begin);
arch/riscv/kvm/aia.c
+42
View File
@@ -25,6 +25,7 @@ struct aia_hgei_control {
unsigned long free_bitmap;
struct kvm_vcpu *owners[BITS_PER_LONG];
unsigned int nr_hgei;
unsigned long saved_hgeie;
};
static DEFINE_PER_CPU(struct aia_hgei_control, aia_hgei);
static int hgei_parent_irq;
@@ -553,6 +554,47 @@ static void aia_hgei_exit(void)
free_percpu_irq(hgei_parent_irq, &aia_hgei);
}
void kvm_riscv_aia_pm_exit(void)
{
struct aia_hgei_control *hgctrl;
if (!kvm_riscv_aia_available())
return;
hgctrl = this_cpu_ptr(&aia_hgei);
csr_write(CSR_HGEIE, hgctrl->saved_hgeie);
csr_write(CSR_HVICTL, aia_hvictl_value(false));
csr_write(CSR_HVIPRIO1, 0x0);
csr_write(CSR_HVIPRIO2, 0x0);
#ifdef CONFIG_32BIT
csr_write(CSR_HVIPH, 0x0);
csr_write(CSR_HIDELEGH, 0x0);
csr_write(CSR_HVIPRIO1H, 0x0);
csr_write(CSR_HVIPRIO2H, 0x0);
#endif
csr_set(CSR_HIE, BIT(IRQ_S_GEXT));
/* Enable IRQ filtering for overflow interrupt only if sscofpmf is present */
if (__riscv_isa_extension_available(NULL, RISCV_ISA_EXT_SSCOFPMF))
csr_set(CSR_HVIEN, BIT(IRQ_PMU_OVF));
}
void kvm_riscv_aia_pm_enter(void)
{
struct aia_hgei_control *hgctrl;
if (!kvm_riscv_aia_available())
return;
if (__riscv_isa_extension_available(NULL, RISCV_ISA_EXT_SSCOFPMF))
csr_clear(CSR_HVIEN, BIT(IRQ_PMU_OVF));
csr_write(CSR_HVICTL, aia_hvictl_value(false));
hgctrl = this_cpu_ptr(&aia_hgei);
hgctrl->saved_hgeie = csr_read(CSR_HGEIE);
}
void kvm_riscv_aia_enable(void)
{
const struct imsic_global_config *gc;
arch/riscv/kvm/aia_aplic.c
+2 -1
View File
@@ -581,7 +581,8 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm)
return 0;
/* Allocate APLIC global state */
aplic = kzalloc_flex(*aplic, irqs, kvm->arch.aia.nr_sources + 1);
aplic = kzalloc_flex(*aplic, irqs, kvm->arch.aia.nr_sources + 1,
GFP_KERNEL_ACCOUNT);
if (!aplic)
return -ENOMEM;
kvm->arch.aia.aplic_state = aplic;
arch/riscv/kvm/aia_device.c
+6
View File
@@ -248,6 +248,12 @@ static int aia_init(struct kvm *kvm)
if (aia->nr_sources && aia->aplic_addr == KVM_RISCV_AIA_UNDEF_ADDR)
return -EINVAL;
/* Group index bits must not overlap guest and HART index bits. */
if (aia->nr_group_bits &&
aia->nr_group_shift < (IMSIC_MMIO_PAGE_SHIFT +
aia->nr_guest_bits + aia->nr_hart_bits))
return -EINVAL;
/* Initialize APLIC */
ret = kvm_riscv_aia_aplic_init(kvm);
if (ret)
arch/riscv/kvm/aia_imsic.c
+7 -11
View File
@@ -220,15 +220,11 @@ static unsigned long imsic_mrif_atomic_rmw(struct imsic_mrif *mrif,
{
unsigned long old_val = 0, tmp = 0;
__asm__ __volatile__ (
"0: lr.w.aq %1, %0\n"
" and %2, %1, %3\n"
" or %2, %2, %4\n"
" sc.w.rl %2, %2, %0\n"
" bnez %2, 0b"
: "+A" (*ptr), "+r" (old_val), "+r" (tmp)
: "r" (~wr_mask), "r" (new_val & wr_mask)
: "memory");
new_val &= wr_mask;
old_val = READ_ONCE(*ptr);
do {
tmp = (old_val & ~wr_mask) | new_val;
} while (!try_cmpxchg(ptr, &old_val, tmp));
return old_val;
}
@@ -1108,7 +1104,7 @@ int kvm_riscv_vcpu_aia_imsic_init(struct kvm_vcpu *vcpu)
return -EINVAL;
/* Allocate IMSIC context */
imsic = kzalloc_obj(*imsic);
imsic = kzalloc_obj(*imsic, GFP_KERNEL_ACCOUNT);
if (!imsic)
return -ENOMEM;
vcpu->arch.aia_context.imsic_state = imsic;
@@ -1121,7 +1117,7 @@ int kvm_riscv_vcpu_aia_imsic_init(struct kvm_vcpu *vcpu)
imsic->vsfile_hgei = imsic->vsfile_cpu = -1;
/* Setup IMSIC SW-file */
swfile_page = alloc_pages(GFP_KERNEL | __GFP_ZERO,
swfile_page = alloc_pages(GFP_KERNEL_ACCOUNT | __GFP_ZERO,
get_order(sizeof(*imsic->swfile)));
if (!swfile_page) {
ret = -ENOMEM;
arch/riscv/kvm/gstage.c
+75 -7
View File
@@ -307,19 +307,20 @@ static inline unsigned long make_child_pte(unsigned long huge_pte, int index,
return child_pte;
}
int kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage,
struct kvm_mmu_memory_cache *pcache,
gpa_t addr, u32 target_level, bool flush)
bool kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage,
struct kvm_mmu_memory_cache *pcache,
gpa_t addr, u32 target_level, bool flush)
{
u32 current_level = gstage->pgd_levels - 1;
pte_t *next_ptep = (pte_t *)gstage->pgd;
unsigned long huge_pte, child_pte;
unsigned long child_page_size;
bool need_flush = false;
pte_t *ptep;
int i, ret;
if (!pcache)
return -ENOMEM;
return false;
while(current_level > target_level) {
ptep = (pte_t *)&next_ptep[gstage_pte_index(gstage, addr, current_level)];
@@ -337,27 +338,35 @@ int kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage,
ret = gstage_level_to_page_size(gstage, current_level - 1, &child_page_size);
if (ret)
return ret;
return need_flush;
next_ptep = kvm_mmu_memory_cache_alloc(pcache);
if (!next_ptep)
return -ENOMEM;
return need_flush;
for (i = 0; i < PTRS_PER_PTE; i++) {
child_pte = make_child_pte(huge_pte, i, child_page_size);
set_pte((pte_t *)&next_ptep[i], __pte(child_pte));
}
/*
* Ensure the writes to the child PTEs are visible before
* linking the new page table to the parent PTE.
*/
smp_wmb();
set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
__pgprot(_PAGE_TABLE)));
if (flush)
gstage_tlb_flush(gstage, current_level, addr);
else
need_flush = true;
current_level--;
}
return 0;
return need_flush;
}
bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
@@ -476,6 +485,65 @@ bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end
return flush;
}
static inline void clear_huge_mask(unsigned long *mask, unsigned long page_size,
gfn_t base_gfn, gpa_t addr)
{
unsigned long start_index = 0;
unsigned long end_index = BITS_PER_LONG - 1;
unsigned long end_gfn = base_gfn + end_index;
unsigned long aligned_start_gfn = addr >> PAGE_SHIFT;
unsigned long aligned_end_gfn = aligned_start_gfn + (page_size >> PAGE_SHIFT) - 1;
unsigned int nbits = 0;
if (aligned_start_gfn > base_gfn)
start_index = aligned_start_gfn - base_gfn;
if (aligned_end_gfn < end_gfn)
end_index = aligned_end_gfn - base_gfn;
nbits = end_index - start_index + 1;
bitmap_clear(mask, start_index, nbits);
}
bool kvm_riscv_gstage_wp_pt_masked(struct kvm_gstage *gstage, gfn_t base_gfn,
unsigned long mask)
{
unsigned long page_size;
bool flush = false;
bool found_leaf;
u32 ptep_level;
pte_t *ptep;
gpa_t addr = 0;
int ret;
while (mask) {
addr = (base_gfn + __ffs(mask)) << PAGE_SHIFT;
found_leaf = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_level);
ret = gstage_level_to_page_size(gstage, ptep_level, &page_size);
if (ret)
break;
if (found_leaf) {
if (ptep_level) {
addr = ALIGN_DOWN(addr, page_size);
clear_huge_mask(&mask, page_size, base_gfn, addr);
}
flush |= kvm_riscv_gstage_op_pte(gstage, addr, ptep,
ptep_level, GSTAGE_OP_WP);
if (ptep_level)
continue;
}
/* clear the first set bit*/
mask &= mask - 1;
}
return flush;
}
void __init kvm_riscv_gstage_mode_detect(void)
{
#ifdef CONFIG_64BIT
arch/riscv/kvm/isa.c
+2
View File
@@ -65,6 +65,8 @@ static const unsigned long kvm_isa_ext_arr[] = {
KVM_ISA_EXT_ARR(ZICBOP),
KVM_ISA_EXT_ARR(ZICBOZ),
KVM_ISA_EXT_ARR(ZICCRSE),
KVM_ISA_EXT_ARR(ZICFILP),
KVM_ISA_EXT_ARR(ZICFISS),
KVM_ISA_EXT_ARR(ZICNTR),
KVM_ISA_EXT_ARR(ZICOND),
KVM_ISA_EXT_ARR(ZICSR),
arch/riscv/kvm/main.c
+95 -19
View File
@@ -10,10 +10,14 @@
#include <linux/err.h>
#include <linux/module.h>
#include <linux/kvm_host.h>
#include <linux/cpu_pm.h>
#include <asm/cpufeature.h>
#include <asm/kvm_mmu.h>
#include <asm/kvm_nacl.h>
#include <asm/sbi.h>
#include <asm/kvm_vcpu_vector.h>
static DEFINE_PER_CPU(bool, kvm_riscv_virtualization_enabled);
DEFINE_STATIC_KEY_FALSE(kvm_riscv_vsstage_tlb_no_gpa);
@@ -33,14 +37,9 @@ long kvm_arch_dev_ioctl(struct file *filp,
return -EINVAL;
}
int kvm_arch_enable_virtualization_cpu(void)
/* Initialize hypervisor CSRs - called during CPU online and non-retention idle resume */
static void kvm_riscv_csr_init(void)
{
int rc;
rc = kvm_riscv_nacl_enable();
if (rc)
return rc;
csr_write(CSR_HEDELEG, 0);
csr_write(CSR_HIDELEG, 0);
@@ -48,16 +47,11 @@ int kvm_arch_enable_virtualization_cpu(void)
csr_write(CSR_HCOUNTEREN, 0x02);
csr_write(CSR_HVIP, 0);
kvm_riscv_aia_enable();
return 0;
}
void kvm_arch_disable_virtualization_cpu(void)
/* Clear hypervisor CSRs - called during CPU offline and non-retention idle entry */
static void kvm_riscv_csr_cleanup(void)
{
kvm_riscv_aia_disable();
/*
* After clearing the hideleg CSR, the host kernel will receive
* spurious interrupts if hvip CSR has pending interrupts and the
@@ -69,13 +63,75 @@ void kvm_arch_disable_virtualization_cpu(void)
csr_write(CSR_HEDELEG, 0);
csr_write(CSR_HIDELEG, 0);
kvm_riscv_nacl_disable();
kvm_riscv_clear_former_vcpu();
}
int kvm_arch_enable_virtualization_cpu(void)
{
int rc;
rc = kvm_riscv_nacl_enable();
if (rc)
return rc;
kvm_riscv_csr_init();
kvm_riscv_aia_enable();
__this_cpu_write(kvm_riscv_virtualization_enabled, true);
return 0;
}
void kvm_arch_disable_virtualization_cpu(void)
{
kvm_riscv_aia_disable();
kvm_riscv_csr_cleanup();
kvm_riscv_nacl_disable();
__this_cpu_write(kvm_riscv_virtualization_enabled, false);
}
static int kvm_riscv_cpu_pm_notifier(struct notifier_block *self, unsigned long cmd, void *v)
{
switch (cmd) {
case CPU_PM_EXIT:
case CPU_PM_ENTER_FAILED:
/*
* Only restore hypervisor state if KVM virtualization is
* enabled on this CPU. This prevents unintentional re-enabling
* of virtualization after it has been explicitly disabled.
*/
if (__this_cpu_read(kvm_riscv_virtualization_enabled)) {
kvm_riscv_csr_init();
kvm_riscv_aia_pm_exit();
}
return NOTIFY_OK;
case CPU_PM_ENTER:
/*
* Only save and clear hypervisor state if KVM virtualization
* is enabled on this CPU.
*/
if (__this_cpu_read(kvm_riscv_virtualization_enabled)) {
kvm_riscv_aia_pm_enter();
kvm_riscv_csr_cleanup();
}
return NOTIFY_OK;
default:
break;
}
return NOTIFY_DONE;
}
static struct notifier_block kvm_riscv_cpu_pm_nb = {
.notifier_call = kvm_riscv_cpu_pm_notifier,
};
static void kvm_riscv_teardown(void)
{
kvm_riscv_aia_exit();
kvm_riscv_nacl_exit();
kvm_riscv_v_exit();
kvm_unregister_perf_callbacks();
}
@@ -170,19 +226,35 @@ static int __init riscv_kvm_init(void)
kvm_riscv_setup_vendor_features();
kvm_riscv_v_init();
kvm_register_perf_callbacks();
rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
if (rc) {
kvm_riscv_teardown();
return rc;
/* Register CPU PM notifier for CPU idle non-retention states */
if (IS_ENABLED(CONFIG_CPU_PM)) {
rc = cpu_pm_register_notifier(&kvm_riscv_cpu_pm_nb);
if (rc) {
kvm_err("Failed to register CPU PM notifier: %d\n", rc);
goto err_teardown;
}
}
rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE);
if (rc)
goto err_unregister_cpu_pm;
if (kvm_riscv_aia_available())
kvm_info("AIA available with %d guest external interrupts\n",
atomic_read(&kvm_riscv_aia_nr_hgei));
return 0;
err_unregister_cpu_pm:
if (IS_ENABLED(CONFIG_CPU_PM))
cpu_pm_unregister_notifier(&kvm_riscv_cpu_pm_nb);
err_teardown:
kvm_riscv_teardown();
return rc;
}
module_init(riscv_kvm_init);
@@ -190,6 +262,10 @@ static void __exit riscv_kvm_exit(void)
{
kvm_exit();
/* Unregister CPU PM notifier */
if (IS_ENABLED(CONFIG_CPU_PM))
cpu_pm_unregister_notifier(&kvm_riscv_cpu_pm_nb);
kvm_riscv_teardown();
}
module_exit(riscv_kvm_exit);
arch/riscv/kvm/mmu.c
+99 -9
View File
@@ -16,6 +16,9 @@
#include <asm/kvm_mmu.h>
#include <asm/kvm_nacl.h>
static bool __read_mostly eager_page_split = true;
module_param(eager_page_split, bool, 0644);
static void mmu_wp_memory_region(struct kvm *kvm, int slot)
{
struct kvm_memslots *slots = kvm_memslots(kvm);
@@ -98,6 +101,62 @@ void kvm_riscv_mmu_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long size)
size >> PAGE_SHIFT);
}
static bool need_topup_split_caches_or_resched(struct kvm *kvm, int count)
{
struct kvm_mmu_memory_cache *cache;
if (need_resched() || rwlock_needbreak(&kvm->mmu_lock))
return true;
cache = &kvm->arch.pgd_split_page_cache;
return kvm_mmu_memory_cache_nr_free_objects(cache) < count;
}
static bool mmu_split_huge_pages(struct kvm_gstage *gstage,
phys_addr_t start, phys_addr_t end)
{
struct kvm *kvm = gstage->kvm;
struct kvm_mmu_memory_cache *pcache = &kvm->arch.pgd_split_page_cache;
phys_addr_t addr = ALIGN_DOWN(start, PMD_SIZE);
phys_addr_t last_flush_gfn = addr >> PAGE_SHIFT;
int count = gstage->pgd_levels;
bool flush = false;
int ret;
lockdep_assert_held_write(&kvm->mmu_lock);
while (addr < end) {
if (need_topup_split_caches_or_resched(kvm, count)) {
if (flush) {
kvm_flush_remote_tlbs_range(kvm, last_flush_gfn,
(addr >> PAGE_SHIFT) - last_flush_gfn);
last_flush_gfn = addr >> PAGE_SHIFT;
flush = false;
}
write_unlock(&kvm->mmu_lock);
cond_resched();
ret = kvm_mmu_topup_memory_cache(pcache, count);
if (ret) {
kvm_err("Failed to toup split page cache\n");
write_lock(&kvm->mmu_lock);
return flush;
}
write_lock(&kvm->mmu_lock);
}
if (!kvm->arch.pgd)
return flush;
flush |= kvm_riscv_gstage_split_huge(gstage, pcache, addr, 0, false);
addr += PMD_SIZE;
}
return flush;
}
void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
struct kvm_memory_slot *slot,
gfn_t gfn_offset,
@@ -107,14 +166,20 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT;
phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT;
struct kvm_gstage gstage;
bool flush;
kvm_riscv_gstage_init(&gstage, kvm);
flush = kvm_riscv_gstage_wp_range(&gstage, start, end);
if (flush)
kvm_flush_remote_tlbs_range(kvm, start >> PAGE_SHIFT,
(end - start) >> PAGE_SHIFT);
kvm_riscv_gstage_wp_pt_masked(&gstage, base_gfn, mask);
if (kvm_dirty_log_manual_protect_and_init_set(kvm)) {
if (READ_ONCE(eager_page_split))
mmu_split_huge_pages(&gstage, start, end);
}
/*
* Remote TLB flush is not needed here since callers of
* kvm_arch_mmu_enable_log_dirty_pt_masked() already do it.
*/
}
void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot)
@@ -152,6 +217,25 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
size >> PAGE_SHIFT);
}
static void mmu_split_memory_region(struct kvm *kvm, int slot)
{
struct kvm_memslots *slots = kvm_memslots(kvm);
struct kvm_memory_slot *memslot = id_to_memslot(slots, slot);
phys_addr_t start = memslot->base_gfn << PAGE_SHIFT;
phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
struct kvm_gstage gstage;
bool flush;
kvm_riscv_gstage_init(&gstage, kvm);
write_lock(&kvm->mmu_lock);
flush = mmu_split_huge_pages(&gstage, start, end);
write_unlock(&kvm->mmu_lock);
if (flush)
kvm_flush_remote_tlbs_memslot(kvm, memslot);
}
void kvm_arch_commit_memory_region(struct kvm *kvm,
struct kvm_memory_slot *old,
const struct kvm_memory_slot *new,
@@ -165,6 +249,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
if (kvm_dirty_log_manual_protect_and_init_set(kvm))
return;
mmu_wp_memory_region(kvm, new->id);
if (READ_ONCE(eager_page_split))
mmu_split_memory_region(kvm, new->id);
}
}
@@ -185,7 +272,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
* Prevent userspace from creating a memory region outside of the GPA
* space addressable by the KVM guest GPA space.
*/
if ((new->base_gfn + new->npages) >=
if ((new->base_gfn + new->npages) >
kvm_riscv_gstage_gpa_size(kvm->arch.pgd_levels) >> PAGE_SHIFT)
return -EFAULT;
@@ -541,7 +628,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
kvm_pfn_t hfn;
bool is_hugetlb;
bool writable;
short vma_pageshift;
unsigned int vma_pageshift;
gfn_t gfn = gpa >> PAGE_SHIFT;
struct vm_area_struct *vma;
struct kvm *kvm = vcpu->kvm;
@@ -670,13 +757,14 @@ int kvm_riscv_mmu_alloc_pgd(struct kvm *kvm)
return -EINVAL;
}
pgd_page = alloc_pages(GFP_KERNEL | __GFP_ZERO,
get_order(kvm_riscv_gstage_pgd_size));
pgd_page = alloc_pages(GFP_KERNEL_ACCOUNT | __GFP_ZERO,
get_order(kvm_riscv_gstage_pgd_size));
if (!pgd_page)
return -ENOMEM;
kvm->arch.pgd = page_to_virt(pgd_page);
kvm->arch.pgd_phys = page_to_phys(pgd_page);
kvm->arch.pgd_levels = kvm_riscv_gstage_max_pgd_levels;
kvm->arch.pgd_split_page_cache.gfp_zero = __GFP_ZERO;
return 0;
}
@@ -704,6 +792,8 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm)
if (pgd)
free_pages((unsigned long)pgd, get_order(kvm_riscv_gstage_pgd_size));
kvm_mmu_free_memory_cache(&kvm->arch.pgd_split_page_cache);
}
void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu)
arch/riscv/kvm/nacl.c
+1 -1
View File
@@ -33,7 +33,7 @@ again:
}
if (ent < 0) {
if (try_count) {
if (try_count--) {
nacl_sync_hfence(-1UL);
goto again;
} else {
arch/riscv/kvm/tlb.c
+94 -23
View File
@@ -8,6 +8,7 @@
#include <linux/errno.h>
#include <linux/err.h>
#include <linux/module.h>
#include <linux/overflow.h>
#include <linux/smp.h>
#include <linux/kvm_host.h>
#include <asm/cacheflush.h>
@@ -24,7 +25,12 @@ void kvm_riscv_local_hfence_gvma_vmid_gpa(unsigned long vmid,
gpa_t gpa, gpa_t gpsz,
unsigned long order)
{
gpa_t pos;
gpa_t end, pos, step = BIT(order);
if (check_add_overflow(gpa, gpsz, &end)) {
kvm_riscv_local_hfence_gvma_vmid_all(vmid);
return;
}
if (PTRS_PER_PTE < (gpsz >> order)) {
kvm_riscv_local_hfence_gvma_vmid_all(vmid);
@@ -33,14 +39,20 @@ void kvm_riscv_local_hfence_gvma_vmid_gpa(unsigned long vmid,
if (has_svinval()) {
asm volatile (SFENCE_W_INVAL() ::: "memory");
for (pos = gpa; pos < (gpa + gpsz); pos += BIT(order))
for (pos = gpa; pos < end; pos += step) {
asm volatile (HINVAL_GVMA(%0, %1)
: : "r" (pos >> 2), "r" (vmid) : "memory");
if (end - pos <= step)
break;
}
asm volatile (SFENCE_INVAL_IR() ::: "memory");
} else {
for (pos = gpa; pos < (gpa + gpsz); pos += BIT(order))
for (pos = gpa; pos < end; pos += step) {
asm volatile (HFENCE_GVMA(%0, %1)
: : "r" (pos >> 2), "r" (vmid) : "memory");
if (end - pos <= step)
break;
}
}
}
@@ -52,7 +64,12 @@ void kvm_riscv_local_hfence_gvma_vmid_all(unsigned long vmid)
void kvm_riscv_local_hfence_gvma_gpa(gpa_t gpa, gpa_t gpsz,
unsigned long order)
{
gpa_t pos;
gpa_t end, pos, step = BIT(order);
if (check_add_overflow(gpa, gpsz, &end)) {
kvm_riscv_local_hfence_gvma_all();
return;
}
if (PTRS_PER_PTE < (gpsz >> order)) {
kvm_riscv_local_hfence_gvma_all();
@@ -61,14 +78,20 @@ void kvm_riscv_local_hfence_gvma_gpa(gpa_t gpa, gpa_t gpsz,
if (has_svinval()) {
asm volatile (SFENCE_W_INVAL() ::: "memory");
for (pos = gpa; pos < (gpa + gpsz); pos += BIT(order))
for (pos = gpa; pos < end; pos += step) {
asm volatile(HINVAL_GVMA(%0, zero)
: : "r" (pos >> 2) : "memory");
if (end - pos <= step)
break;
}
asm volatile (SFENCE_INVAL_IR() ::: "memory");
} else {
for (pos = gpa; pos < (gpa + gpsz); pos += BIT(order))
for (pos = gpa; pos < end; pos += step) {
asm volatile(HFENCE_GVMA(%0, zero)
: : "r" (pos >> 2) : "memory");
if (end - pos <= step)
break;
}
}
}
@@ -83,7 +106,13 @@ void kvm_riscv_local_hfence_vvma_asid_gva(unsigned long vmid,
unsigned long gvsz,
unsigned long order)
{
unsigned long pos, hgatp;
unsigned long end, pos, step = BIT(order);
unsigned long hgatp;
if (check_add_overflow(gva, gvsz, &end)) {
kvm_riscv_local_hfence_vvma_asid_all(vmid, asid);
return;
}
if (PTRS_PER_PTE < (gvsz >> order)) {
kvm_riscv_local_hfence_vvma_asid_all(vmid, asid);
@@ -94,14 +123,20 @@ void kvm_riscv_local_hfence_vvma_asid_gva(unsigned long vmid,
if (has_svinval()) {
asm volatile (SFENCE_W_INVAL() ::: "memory");
for (pos = gva; pos < (gva + gvsz); pos += BIT(order))
for (pos = gva; pos < end; pos += step) {
asm volatile(HINVAL_VVMA(%0, %1)
: : "r" (pos), "r" (asid) : "memory");
if (end - pos <= step)
break;
}
asm volatile (SFENCE_INVAL_IR() ::: "memory");
} else {
for (pos = gva; pos < (gva + gvsz); pos += BIT(order))
for (pos = gva; pos < end; pos += step) {
asm volatile(HFENCE_VVMA(%0, %1)
: : "r" (pos), "r" (asid) : "memory");
if (end - pos <= step)
break;
}
}
csr_write(CSR_HGATP, hgatp);
@@ -123,7 +158,13 @@ void kvm_riscv_local_hfence_vvma_gva(unsigned long vmid,
unsigned long gva, unsigned long gvsz,
unsigned long order)
{
unsigned long pos, hgatp;
unsigned long end, pos, step = BIT(order);
unsigned long hgatp;
if (check_add_overflow(gva, gvsz, &end)) {
kvm_riscv_local_hfence_vvma_all(vmid);
return;
}
if (PTRS_PER_PTE < (gvsz >> order)) {
kvm_riscv_local_hfence_vvma_all(vmid);
@@ -134,14 +175,20 @@ void kvm_riscv_local_hfence_vvma_gva(unsigned long vmid,
if (has_svinval()) {
asm volatile (SFENCE_W_INVAL() ::: "memory");
for (pos = gva; pos < (gva + gvsz); pos += BIT(order))
for (pos = gva; pos < end; pos += step) {
asm volatile(HINVAL_VVMA(%0, zero)
: : "r" (pos) : "memory");
if (end - pos <= step)
break;
}
asm volatile (SFENCE_INVAL_IR() ::: "memory");
} else {
for (pos = gva; pos < (gva + gvsz); pos += BIT(order))
for (pos = gva; pos < end; pos += step) {
asm volatile(HFENCE_VVMA(%0, zero)
: : "r" (pos) : "memory");
if (end - pos <= step)
break;
}
}
csr_write(CSR_HGATP, hgatp);
@@ -325,14 +372,11 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu)
}
}
static void make_xfence_request(struct kvm *kvm,
unsigned long hbase, unsigned long hmask,
unsigned int req, unsigned int fallback_req,
const struct kvm_riscv_hfence *data)
static void make_xfence_request_nodata(struct kvm *kvm, unsigned long hbase,
unsigned long hmask, unsigned int req)
{
unsigned long i;
struct kvm_vcpu *vcpu;
unsigned int actual_req = req;
DECLARE_BITMAP(vcpu_mask, KVM_MAX_VCPUS);
bitmap_zero(vcpu_mask, KVM_MAX_VCPUS);
@@ -346,9 +390,34 @@ static void make_xfence_request(struct kvm *kvm,
}
bitmap_set(vcpu_mask, i, 1);
}
if (!data || !data->type)
continue;
kvm_make_vcpus_request_mask(kvm, req, vcpu_mask);
}
static void make_xfence_request(struct kvm *kvm,
unsigned long hbase, unsigned long hmask,
unsigned int req, unsigned int fallback_req,
const struct kvm_riscv_hfence *data)
{
unsigned long i;
struct kvm_vcpu *vcpu;
DECLARE_BITMAP(req_vcpu_mask, KVM_MAX_VCPUS);
DECLARE_BITMAP(fallback_req_vcpu_mask, KVM_MAX_VCPUS);
if (!data || !data->type)
return;
bitmap_zero(req_vcpu_mask, KVM_MAX_VCPUS);
bitmap_zero(fallback_req_vcpu_mask, KVM_MAX_VCPUS);
kvm_for_each_vcpu(i, vcpu, kvm) {
if (hbase != -1UL) {
if (vcpu->vcpu_id < hbase ||
vcpu->vcpu_id >= hbase + BITS_PER_LONG)
continue;
if (!(hmask & (1UL << (vcpu->vcpu_id - hbase))))
continue;
}
/*
* Enqueue hfence data to VCPU hfence queue. If we don't
@@ -356,17 +425,19 @@ static void make_xfence_request(struct kvm *kvm,
* a more conservative hfence request.
*/
if (!vcpu_hfence_enqueue(vcpu, data))
actual_req = fallback_req;
bitmap_set(fallback_req_vcpu_mask, i, 1);
else
bitmap_set(req_vcpu_mask, i, 1);
}
kvm_make_vcpus_request_mask(kvm, actual_req, vcpu_mask);
kvm_make_vcpus_request_mask(kvm, req, req_vcpu_mask);
kvm_make_vcpus_request_mask(kvm, fallback_req, fallback_req_vcpu_mask);
}
void kvm_riscv_fence_i(struct kvm *kvm,
unsigned long hbase, unsigned long hmask)
{
make_xfence_request(kvm, hbase, hmask, KVM_REQ_FENCE_I,
KVM_REQ_FENCE_I, NULL);
make_xfence_request_nodata(kvm, hbase, hmask, KVM_REQ_FENCE_I);
}
void kvm_riscv_hfence_gvma_vmid_gpa(struct kvm *kvm,
arch/riscv/kvm/trace.h
+79 -2
View File
@@ -25,7 +25,7 @@ TRACE_EVENT(kvm_entry,
__entry->pc = vcpu->arch.guest_context.sepc;
),
TP_printk("PC: 0x016%lx", __entry->pc)
TP_printk("PC: 0x%016lx", __entry->pc)
);
TRACE_EVENT(kvm_exit,
@@ -56,7 +56,84 @@ TRACE_EVENT(kvm_exit,
__entry->htinst)
);
#endif /* _TRACE_RSICV_KVM_H */
TRACE_EVENT(kvm_mmio_emulate,
TP_PROTO(unsigned long vcpu_id, unsigned long sepc, unsigned long insn,
unsigned long fault_addr, bool write, int len),
TP_ARGS(vcpu_id, sepc, insn, fault_addr, write, len),
TP_STRUCT__entry(
__field(unsigned long, vcpu_id)
__field(unsigned long, sepc)
__field(unsigned long, insn)
__field(unsigned long, fault_addr)
__field(bool, write)
__field(int, len)
),
TP_fast_assign(
__entry->vcpu_id = vcpu_id;
__entry->sepc = sepc;
__entry->insn = insn;
__entry->fault_addr = fault_addr;
__entry->write = write;
__entry->len = len;
),
TP_printk("VCPU: %lu, %s MMIO at 0x%lx, len %d, insn 0x%lx, sepc 0x%lx",
__entry->vcpu_id, __entry->write ? "Store" : "Load",
__entry->fault_addr, __entry->len, __entry->insn,
__entry->sepc)
);
TRACE_EVENT(kvm_vcpu_exit,
TP_PROTO(unsigned long vcpu_id, unsigned long sepc, unsigned long scause,
unsigned long stval, unsigned long htval, unsigned long htinst),
TP_ARGS(vcpu_id, sepc, scause, stval, htval, htinst),
TP_STRUCT__entry(
__field(unsigned long, vcpu_id)
__field(unsigned long, sepc)
__field(unsigned long, scause)
__field(unsigned long, stval)
__field(unsigned long, htval)
__field(unsigned long, htinst)
),
TP_fast_assign(
__entry->vcpu_id = vcpu_id;
__entry->sepc = sepc;
__entry->scause = scause;
__entry->stval = stval;
__entry->htval = htval;
__entry->htinst = htinst;
),
TP_printk("VCPU: %lu, SEPC: 0x%lx, SCAUSE: 0x%lx, STVAL: 0x%lx, HTVAL: 0x%lx, HTINST: 0x%lx",
__entry->vcpu_id, __entry->sepc, __entry->scause,
__entry->stval, __entry->htval, __entry->htinst)
);
TRACE_EVENT(kvm_vcpu_irq,
TP_PROTO(unsigned long vcpu_id, unsigned int irq, int level),
TP_ARGS(vcpu_id, irq, level),
TP_STRUCT__entry(
__field(unsigned long, vcpu_id)
__field(unsigned int, irq)
__field(int, level)
),
TP_fast_assign(
__entry->vcpu_id = vcpu_id;
__entry->irq = irq;
__entry->level = level;
),
TP_printk("VCPU: %lu, IRQ: %u, level: %d",
__entry->vcpu_id, __entry->irq, __entry->level)
);
#endif /* _TRACE_KVM_H */
#undef TRACE_INCLUDE_PATH
#define TRACE_INCLUDE_PATH .
arch/riscv/kvm/vcpu.c
+34
View File
@@ -26,6 +26,17 @@
static DEFINE_PER_CPU(struct kvm_vcpu *, kvm_former_vcpu);
void kvm_riscv_clear_former_vcpu(void)
{
/*
* Clear the per-CPU former VCPU pointer because hypervisor CSR state
* will be lost. This ensures that the next VCPU entry will properly
* restore all CSRs instead of incorrectly skipping CSR restoration
* via the fast-path optimization.
*/
__this_cpu_write(kvm_former_vcpu, NULL);
}
const struct kvm_stats_desc kvm_vcpu_stats_desc[] = {
KVM_GENERIC_VCPU_STATS(),
STATS_DESC_COUNTER(VCPU, ecall_exit_stat),
@@ -63,6 +74,7 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu,
memset(cntx, 0, sizeof(*cntx));
memset(csr, 0, sizeof(*csr));
memset(&vcpu->arch.smstateen_csr, 0, sizeof(vcpu->arch.smstateen_csr));
memset(&vcpu->arch.zicfiss_csr, 0, sizeof(vcpu->arch.zicfiss_csr));
/* Restore datap as it's not a part of the guest context. */
cntx->vector.datap = vector_datap;
@@ -439,6 +451,8 @@ int kvm_riscv_vcpu_set_interrupt(struct kvm_vcpu *vcpu, unsigned int irq)
__set_bit(irq, vcpu->arch.irqs_pending_mask);
raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
trace_kvm_vcpu_irq(vcpu->vcpu_id, irq, 1);
kvm_vcpu_kick(vcpu);
return 0;
@@ -465,6 +479,8 @@ int kvm_riscv_vcpu_unset_interrupt(struct kvm_vcpu *vcpu, unsigned int irq)
__set_bit(irq, vcpu->arch.irqs_pending_mask);
raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags);
trace_kvm_vcpu_irq(vcpu->vcpu_id, irq, 0);
return 0;
}
@@ -631,9 +647,11 @@ csr_restore_done:
kvm_riscv_vcpu_host_fp_save(&vcpu->arch.host_context);
kvm_riscv_vcpu_guest_fp_restore(&vcpu->arch.guest_context,
vcpu->arch.isa);
get_cpu_vector_context();
kvm_riscv_vcpu_host_vector_save(&vcpu->arch.host_context);
kvm_riscv_vcpu_guest_vector_restore(&vcpu->arch.guest_context,
vcpu->arch.isa);
put_cpu_vector_context();
kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu);
@@ -654,9 +672,11 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
kvm_riscv_vcpu_host_fp_restore(&vcpu->arch.host_context);
kvm_riscv_vcpu_timer_save(vcpu);
get_cpu_vector_context();
kvm_riscv_vcpu_guest_vector_save(&vcpu->arch.guest_context,
vcpu->arch.isa);
kvm_riscv_vcpu_host_vector_restore(&vcpu->arch.host_context);
put_cpu_vector_context();
if (kvm_riscv_nacl_available()) {
nsh = nacl_shmem();
@@ -748,6 +768,7 @@ static void kvm_riscv_update_hvip(struct kvm_vcpu *vcpu)
static __always_inline void kvm_riscv_vcpu_swap_in_guest_state(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_zicfiss_csr *zicficsr = &vcpu->arch.zicfiss_csr;
struct kvm_vcpu_smstateen_csr *smcsr = &vcpu->arch.smstateen_csr;
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
@@ -755,10 +776,13 @@ static __always_inline void kvm_riscv_vcpu_swap_in_guest_state(struct kvm_vcpu *
vcpu->arch.host_senvcfg = csr_swap(CSR_SENVCFG, csr->senvcfg);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN))
vcpu->arch.host_sstateen0 = csr_swap(CSR_SSTATEEN0, smcsr->sstateen0);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICFISS))
csr_write(CSR_SSP, zicficsr->ssp);
}
static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_zicfiss_csr *zicficsr = &vcpu->arch.zicfiss_csr;
struct kvm_vcpu_smstateen_csr *smcsr = &vcpu->arch.smstateen_csr;
struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr;
@@ -766,6 +790,8 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v
csr->senvcfg = csr_swap(CSR_SENVCFG, vcpu->arch.host_senvcfg);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN))
smcsr->sstateen0 = csr_swap(CSR_SSTATEEN0, vcpu->arch.host_sstateen0);
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICFISS))
zicficsr->ssp = csr_swap(CSR_SSP, 0);
}
/*
@@ -793,6 +819,14 @@ static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu,
kvm_riscv_vcpu_swap_in_guest_state(vcpu);
guest_state_enter_irqoff();
/* sstatus.VS != SR_VS_OFF is guaranteed when NEED_RESTORE is set */
if (current->thread.riscv_v_flags & RISCV_V_VCPU_NEED_RESTORE) {
current->thread.riscv_v_flags &= ~RISCV_V_VCPU_NEED_RESTORE;
current->thread.riscv_v_flags |= RISCV_V_VCPU_CTX;
__kvm_riscv_vector_restore(gcntx);
gcntx->sstatus = (gcntx->sstatus & ~SR_VS) | SR_VS_CLEAN;
}
if (kvm_riscv_nacl_sync_sret_available()) {
nsh = nacl_shmem();
arch/riscv/kvm/vcpu_config.c
+2
View File
@@ -69,6 +69,8 @@ void kvm_riscv_vcpu_config_ran_once(struct kvm_vcpu *vcpu)
if (vcpu->guest_debug)
cfg->hedeleg &= ~BIT(EXC_BREAKPOINT);
kvm_riscv_vcpu_sbi_validate(vcpu);
}
void kvm_riscv_vcpu_config_load(struct kvm_vcpu *vcpu)
arch/riscv/kvm/vcpu_exit.c
+18 -2
View File
@@ -11,18 +11,20 @@
#include <asm/insn-def.h>
#include <asm/kvm_mmu.h>
#include <asm/kvm_nacl.h>
#include "trace.h"
static int gstage_page_fault(struct kvm_vcpu *vcpu, struct kvm_run *run,
struct kvm_cpu_trap *trap)
{
struct kvm_gstage_mapping host_map;
struct kvm_memory_slot *memslot;
unsigned long hva, fault_addr;
unsigned long hva;
gpa_t fault_addr;
bool writable;
gfn_t gfn;
int ret;
fault_addr = (trap->htval << 2) | (trap->stval & 0x3);
fault_addr = ((gpa_t)trap->htval << 2) | (trap->stval & 0x3);
gfn = fault_addr >> PAGE_SHIFT;
memslot = gfn_to_memslot(vcpu->kvm, gfn);
hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable);
@@ -173,6 +175,13 @@ void kvm_riscv_vcpu_trap_redirect(struct kvm_vcpu *vcpu,
/* Clear Guest SSTATUS.SIE bit */
vsstatus &= ~SR_SIE;
/* Change Guest SSTATUS.SPELP bit */
if (vcpu->arch.cfg.henvcfg & ENVCFG_LPE) {
vsstatus &= ~SR_SPELP;
vsstatus |= vcpu->arch.guest_context.sstatus & SR_SPELP;
vcpu->arch.guest_context.sstatus &= ~SR_SPELP;
}
/* Update Guest SSTATUS */
ncsr_write(CSR_VSSTATUS, vsstatus);
@@ -212,6 +221,9 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
if (trap->scause & CAUSE_IRQ_FLAG)
return 1;
trace_kvm_vcpu_exit(vcpu->vcpu_id, trap->sepc, trap->scause,
trap->stval, trap->htval, trap->htinst);
/* Handle guest traps */
ret = -EFAULT;
run->exit_reason = KVM_EXIT_UNKNOWN;
@@ -262,6 +274,10 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run,
run->exit_reason = KVM_EXIT_DEBUG;
ret = 0;
break;
case EXC_SOFTWARE_CHECK:
if (vcpu->arch.cfg.henvcfg & (ENVCFG_LPE | ENVCFG_SSE))
ret = vcpu_redirect(vcpu, trap);
break;
default:
break;
}
arch/riscv/kvm/vcpu_insn.c
+13 -4
View File
@@ -9,6 +9,7 @@
#include <asm/cpufeature.h>
#include <asm/insn.h>
#include "trace.h"
struct insn_func {
unsigned long mask;
@@ -371,11 +372,11 @@ int kvm_riscv_vcpu_virtual_insn(struct kvm_vcpu *vcpu, struct kvm_run *run,
* Returns < 0 to report failure and exit run-loop
*/
int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run,
unsigned long fault_addr,
gpa_t fault_addr,
unsigned long htinst)
{
u8 data_buf[8];
unsigned long insn;
unsigned long insn, raw_insn;
int shift = 0, len = 0, insn_len = 0;
struct kvm_cpu_trap utrap = { 0 };
struct kvm_cpu_context *ct = &vcpu->arch.guest_context;
@@ -405,6 +406,7 @@ int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run,
}
insn_len = INSN_LEN(insn);
}
raw_insn = insn;
/* Decode length of MMIO and shift */
if ((insn & INSN_MASK_LW) == INSN_MATCH_LW) {
@@ -453,6 +455,9 @@ int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run,
if (fault_addr & (len - 1))
return -EIO;
trace_kvm_mmio_emulate(vcpu->vcpu_id, ct->sepc, raw_insn, fault_addr,
false, len);
/* Save instruction decode info */
vcpu->arch.mmio_decode.insn = insn;
vcpu->arch.mmio_decode.insn_len = insn_len;
@@ -494,7 +499,7 @@ int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run,
* Returns < 0 to report failure and exit run-loop
*/
int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run,
unsigned long fault_addr,
gpa_t fault_addr,
unsigned long htinst)
{
u8 data8;
@@ -502,7 +507,7 @@ int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run,
u32 data32;
u64 data64;
ulong data;
unsigned long insn;
unsigned long insn, raw_insn;
int len = 0, insn_len = 0;
struct kvm_cpu_trap utrap = { 0 };
struct kvm_cpu_context *ct = &vcpu->arch.guest_context;
@@ -532,6 +537,7 @@ int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run,
}
insn_len = INSN_LEN(insn);
}
raw_insn = insn;
data = GET_RS2(insn, &vcpu->arch.guest_context);
data8 = data16 = data32 = data64 = data;
@@ -570,6 +576,9 @@ int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run,
if (fault_addr & (len - 1))
return -EIO;
trace_kvm_mmio_emulate(vcpu->vcpu_id, ct->sepc, raw_insn, fault_addr,
true, len);
/* Save instruction decode info */
vcpu->arch.mmio_decode.insn = insn;
vcpu->arch.mmio_decode.insn_len = insn_len;
arch/riscv/kvm/vcpu_onereg.c
+66 -2
View File
@@ -347,6 +347,44 @@ static int kvm_riscv_vcpu_smstateen_get_csr(struct kvm_vcpu *vcpu,
return 0;
}
static inline int kvm_riscv_vcpu_zicfiss_set_csr(struct kvm_vcpu *vcpu,
unsigned long reg_num,
unsigned long reg_val)
{
struct kvm_vcpu_zicfiss_csr *csr = &vcpu->arch.zicfiss_csr;
unsigned long regs_max = sizeof(struct kvm_vcpu_zicfiss_csr) /
sizeof(unsigned long);
if (!riscv_isa_extension_available(vcpu->arch.isa, ZICFISS))
return -ENOENT;
if (reg_num >= regs_max)
return -ENOENT;
reg_num = array_index_nospec(reg_num, regs_max);
((unsigned long *)csr)[reg_num] = reg_val;
return 0;
}
static int kvm_riscv_vcpu_zicfiss_get_csr(struct kvm_vcpu *vcpu,
unsigned long reg_num,
unsigned long *out_val)
{
struct kvm_vcpu_zicfiss_csr *csr = &vcpu->arch.zicfiss_csr;
unsigned long regs_max = sizeof(struct kvm_vcpu_zicfiss_csr) /
sizeof(unsigned long);
if (!riscv_isa_extension_available(vcpu->arch.isa, ZICFISS))
return -ENOENT;
if (reg_num >= regs_max)
return -ENOENT;
reg_num = array_index_nospec(reg_num, regs_max);
*out_val = ((unsigned long *)csr)[reg_num];
return 0;
}
static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu,
const struct kvm_one_reg *reg)
{
@@ -373,6 +411,9 @@ static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu,
case KVM_REG_RISCV_CSR_SMSTATEEN:
rc = kvm_riscv_vcpu_smstateen_get_csr(vcpu, reg_num, &reg_val);
break;
case KVM_REG_RISCV_CSR_ZICFISS:
rc = kvm_riscv_vcpu_zicfiss_get_csr(vcpu, reg_num, &reg_val);
break;
default:
rc = -ENOENT;
break;
@@ -415,6 +456,9 @@ static int kvm_riscv_vcpu_set_reg_csr(struct kvm_vcpu *vcpu,
case KVM_REG_RISCV_CSR_SMSTATEEN:
rc = kvm_riscv_vcpu_smstateen_set_csr(vcpu, reg_num, reg_val);
break;
case KVM_REG_RISCV_CSR_ZICFISS:
rc = kvm_riscv_vcpu_zicfiss_set_csr(vcpu, reg_num, reg_val);
break;
default:
rc = -ENOENT;
break;
@@ -658,6 +702,8 @@ static inline unsigned long num_csr_regs(const struct kvm_vcpu *vcpu)
n += sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long);
if (riscv_isa_extension_available(vcpu->arch.isa, SMSTATEEN))
n += sizeof(struct kvm_riscv_smstateen_csr) / sizeof(unsigned long);
if (riscv_isa_extension_available(vcpu->arch.isa, ZICFISS))
n += sizeof(struct kvm_riscv_zicfiss_csr) / sizeof(unsigned long);
return n;
}
@@ -666,7 +712,7 @@ static int copy_csr_reg_indices(const struct kvm_vcpu *vcpu,
u64 __user *uindices)
{
int n1 = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long);
int n2 = 0, n3 = 0;
int n2 = 0, n3 = 0, n4 = 0;
/* copy general csr regs */
for (int i = 0; i < n1; i++) {
@@ -718,7 +764,25 @@ static int copy_csr_reg_indices(const struct kvm_vcpu *vcpu,
}
}
return n1 + n2 + n3;
/* copy Zicfiss csr regs */
if (riscv_isa_extension_available(vcpu->arch.isa, ZICFISS)) {
n4 = sizeof(struct kvm_riscv_zicfiss_csr) / sizeof(unsigned long);
for (int i = 0; i < n4; i++) {
u64 size = IS_ENABLED(CONFIG_32BIT) ?
KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64;
u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_CSR |
KVM_REG_RISCV_CSR_ZICFISS | i;
if (uindices) {
if (put_user(reg, uindices))
return -EFAULT;
uindices++;
}
}
}
return n1 + n2 + n3 + n4;
}
static inline unsigned long num_timer_regs(void)
arch/riscv/kvm/vcpu_pmu.c
+10 -6
View File
@@ -12,7 +12,9 @@
#include <linux/err.h>
#include <linux/kvm_host.h>
#include <linux/nospec.h>
#include <linux/overflow.h>
#include <linux/perf/riscv_pmu.h>
#include <linux/slab.h>
#include <asm/csr.h>
#include <asm/kvm_isa.h>
#include <asm/kvm_vcpu_sbi.h>
@@ -452,7 +454,7 @@ int kvm_riscv_vcpu_pmu_snapshot_set_shmem(struct kvm_vcpu *vcpu, unsigned long s
}
}
kvpmu->sdata = kzalloc(snapshot_area_size, GFP_ATOMIC);
kvpmu->sdata = kzalloc(snapshot_area_size, GFP_ATOMIC | __GFP_ACCOUNT);
if (!kvpmu->sdata) {
sbiret = SBI_ERR_FAILURE;
goto out;
@@ -479,13 +481,14 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low
unsigned long flags, struct kvm_vcpu_sbi_return *retdata)
{
struct riscv_pmu_event_info *einfo = NULL;
int shmem_size = num_events * sizeof(*einfo);
size_t shmem_size;
gpa_t shmem;
u32 eidx, etype;
u64 econfig;
int ret;
if (flags != 0 || (saddr_low & (SZ_16 - 1) || num_events == 0)) {
if (flags != 0 || (saddr_low & (SZ_16 - 1)) || num_events == 0 ||
check_mul_overflow(num_events, sizeof(*einfo), &shmem_size)) {
ret = SBI_ERR_INVALID_PARAM;
goto out;
}
@@ -500,7 +503,8 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low
}
}
einfo = kzalloc(shmem_size, GFP_KERNEL);
einfo = kvcalloc(num_events, sizeof(*einfo),
GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
if (!einfo) {
ret = SBI_ERR_FAILURE;
goto out;
@@ -512,7 +516,7 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low
goto free_mem;
}
for (int i = 0; i < num_events; i++) {
for (unsigned long i = 0; i < num_events; i++) {
eidx = einfo[i].event_idx;
etype = kvm_pmu_get_perf_event_type(eidx);
econfig = kvm_pmu_get_perf_event_config(eidx, einfo[i].event_data);
@@ -525,7 +529,7 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low
ret = SBI_ERR_INVALID_ADDRESS;
free_mem:
kfree(einfo);
kvfree(einfo);
out:
retdata->err_val = ret;
arch/riscv/kvm/vcpu_sbi.c
+30
View File
@@ -221,6 +221,7 @@ static int riscv_vcpu_set_sbi_ext_single(struct kvm_vcpu *vcpu,
{
struct kvm_vcpu_sbi_context *scontext = &vcpu->arch.sbi_context;
const struct kvm_riscv_sbi_extension_entry *sext;
const struct kvm_vcpu_sbi_extension *ext;
if (reg_val != 1 && reg_val != 0)
return -EINVAL;
@@ -229,6 +230,12 @@ static int riscv_vcpu_set_sbi_ext_single(struct kvm_vcpu *vcpu,
if (!sext || scontext->ext_status[sext->ext_idx] == KVM_RISCV_SBI_EXT_STATUS_UNAVAILABLE)
return -ENOENT;
ext = sext->ext_ptr;
if (!reg_val && scontext->ext_status[sext->ext_idx] == KVM_RISCV_SBI_EXT_STATUS_ENABLED &&
ext->reset)
ext->reset(vcpu);
scontext->ext_status[sext->ext_idx] = (reg_val) ?
KVM_RISCV_SBI_EXT_STATUS_ENABLED :
KVM_RISCV_SBI_EXT_STATUS_DISABLED;
@@ -723,3 +730,26 @@ void kvm_riscv_vcpu_sbi_reset(struct kvm_vcpu *vcpu)
ext->reset(vcpu);
}
}
void kvm_riscv_vcpu_sbi_validate(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_sbi_context *scontext = &vcpu->arch.sbi_context;
const struct kvm_riscv_sbi_extension_entry *entry;
const struct kvm_vcpu_sbi_extension *ext;
int idx, i;
for (i = 0; i < ARRAY_SIZE(sbi_ext); i++) {
entry = &sbi_ext[i];
ext = entry->ext_ptr;
idx = entry->ext_idx;
if (idx < 0 || idx >= ARRAY_SIZE(scontext->ext_status))
continue;
if (scontext->ext_status[idx] != KVM_RISCV_SBI_EXT_STATUS_ENABLED ||
!ext->validate)
continue;
ext->validate(vcpu);
}
}
arch/riscv/kvm/vcpu_sbi_fwft.c
+206 -1
View File
@@ -94,6 +94,44 @@ static bool kvm_fwft_is_defined_feature(enum sbi_fwft_feature_t feature)
return false;
}
static void kvm_sbi_fwft_envcfg_flag_reset(struct kvm_vcpu *vcpu, u64 flag)
{
vcpu->arch.cfg.henvcfg &= ~flag;
}
static long kvm_sbi_fwft_envcfg_flag_set(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access,
unsigned long value, u64 flag)
{
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
if (value == 0)
cfg->henvcfg &= ~flag;
else if (value == 1)
cfg->henvcfg |= flag;
else
return SBI_ERR_INVALID_PARAM;
if (!one_reg_access) {
csr_write(CSR_HENVCFG, vcpu->arch.cfg.henvcfg);
if (IS_ENABLED(CONFIG_32BIT))
csr_write(CSR_HENVCFGH, vcpu->arch.cfg.henvcfg >> 32);
}
return SBI_SUCCESS;
}
static long kvm_sbi_fwft_envcfg_flag_get(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access,
unsigned long *value, u64 flag)
{
*value = (vcpu->arch.cfg.henvcfg & flag) == flag;
return SBI_SUCCESS;
}
static bool kvm_sbi_fwft_misaligned_delegation_supported(struct kvm_vcpu *vcpu)
{
return misaligned_traps_can_delegate();
@@ -137,6 +175,119 @@ static long kvm_sbi_fwft_get_misaligned_delegation(struct kvm_vcpu *vcpu,
return SBI_SUCCESS;
}
static long kvm_sbi_fwft_set_cfi(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access, unsigned long value,
u64 flag)
{
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
if (value == 0)
cfg->henvcfg &= ~flag;
else if (value == 1)
cfg->henvcfg |= flag;
else
return SBI_ERR_INVALID_PARAM;
if (cfg->henvcfg & (ENVCFG_LPE | ENVCFG_SSE))
cfg->hedeleg |= BIT(EXC_SOFTWARE_CHECK);
else
cfg->hedeleg &= ~BIT(EXC_SOFTWARE_CHECK);
if (!one_reg_access) {
csr_write(CSR_HEDELEG, cfg->hedeleg);
/*
* Both Bit LPE and SSE are in the lower part, so it is safe
* to only write the henvcfg
*/
csr_write(CSR_HENVCFG, vcpu->arch.cfg.henvcfg);
}
return SBI_SUCCESS;
}
static bool kvm_sbi_fwft_landing_pad_supported(struct kvm_vcpu *vcpu)
{
return riscv_isa_extension_available(vcpu->arch.isa, ZICFILP);
}
static void kvm_sbi_fwft_reset_landing_pad(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
kvm_sbi_fwft_envcfg_flag_reset(vcpu, ENVCFG_LPE);
if ((cfg->henvcfg & (ENVCFG_LPE | ENVCFG_SSE)) == 0)
cfg->hedeleg &= ~BIT(EXC_SOFTWARE_CHECK);
}
static long kvm_sbi_fwft_set_landing_pad(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access, unsigned long value)
{
return kvm_sbi_fwft_set_cfi(vcpu, conf, one_reg_access, value, ENVCFG_LPE);
}
static long kvm_sbi_fwft_get_landing_pad(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access, unsigned long *value)
{
return kvm_sbi_fwft_envcfg_flag_get(vcpu, conf, one_reg_access, value, ENVCFG_LPE);
}
static bool kvm_sbi_fwft_shadow_stack_supported(struct kvm_vcpu *vcpu)
{
return riscv_isa_extension_available(vcpu->arch.isa, ZICFISS);
}
static void kvm_sbi_fwft_reset_shadow_stack(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu_config *cfg = &vcpu->arch.cfg;
kvm_sbi_fwft_envcfg_flag_reset(vcpu, ENVCFG_SSE);
if ((cfg->henvcfg & (ENVCFG_LPE | ENVCFG_SSE)) == 0)
cfg->hedeleg &= ~BIT(EXC_SOFTWARE_CHECK);
}
static long kvm_sbi_fwft_set_shadow_stack(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access, unsigned long value)
{
return kvm_sbi_fwft_set_cfi(vcpu, conf, one_reg_access, value, ENVCFG_SSE);
}
static long kvm_sbi_fwft_get_shadow_stack(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access, unsigned long *value)
{
return kvm_sbi_fwft_envcfg_flag_get(vcpu, conf, one_reg_access, value, ENVCFG_SSE);
}
static bool kvm_sbi_fwft_pte_ad_hw_updating_supported(struct kvm_vcpu *vcpu)
{
return riscv_isa_extension_available(vcpu->arch.isa, SVADU) &&
riscv_isa_extension_available(vcpu->arch.isa, SVADE);
}
static void kvm_sbi_fwft_reset_pte_ad_hw_updating(struct kvm_vcpu *vcpu)
{
if (kvm_sbi_fwft_pte_ad_hw_updating_supported(vcpu))
kvm_sbi_fwft_envcfg_flag_reset(vcpu, ENVCFG_ADUE);
}
static long kvm_sbi_fwft_set_pte_ad_hw_updating(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access, unsigned long value)
{
return kvm_sbi_fwft_envcfg_flag_set(vcpu, conf, one_reg_access, value, ENVCFG_ADUE);
}
static long kvm_sbi_fwft_get_pte_ad_hw_updating(struct kvm_vcpu *vcpu,
struct kvm_sbi_fwft_config *conf,
bool one_reg_access, unsigned long *value)
{
return kvm_sbi_fwft_envcfg_flag_get(vcpu, conf, one_reg_access, value, ENVCFG_ADUE);
}
#ifndef CONFIG_32BIT
static bool try_to_set_pmm(unsigned long value)
@@ -246,6 +397,33 @@ static const struct kvm_sbi_fwft_feature features[] = {
.set = kvm_sbi_fwft_set_misaligned_delegation,
.get = kvm_sbi_fwft_get_misaligned_delegation,
},
{
.id = SBI_FWFT_LANDING_PAD,
.first_reg_num = offsetof(struct kvm_riscv_sbi_fwft, landing_pad.enable) /
sizeof(unsigned long),
.supported = kvm_sbi_fwft_landing_pad_supported,
.reset = kvm_sbi_fwft_reset_landing_pad,
.set = kvm_sbi_fwft_set_landing_pad,
.get = kvm_sbi_fwft_get_landing_pad,
},
{
.id = SBI_FWFT_SHADOW_STACK,
.first_reg_num = offsetof(struct kvm_riscv_sbi_fwft, shadow_stack.enable) /
sizeof(unsigned long),
.supported = kvm_sbi_fwft_shadow_stack_supported,
.reset = kvm_sbi_fwft_reset_shadow_stack,
.set = kvm_sbi_fwft_set_shadow_stack,
.get = kvm_sbi_fwft_get_shadow_stack,
},
{
.id = SBI_FWFT_PTE_AD_HW_UPDATING,
.first_reg_num = offsetof(struct kvm_riscv_sbi_fwft, pte_ad_hw_updating.enable) /
sizeof(unsigned long),
.supported = kvm_sbi_fwft_pte_ad_hw_updating_supported,
.reset = kvm_sbi_fwft_reset_pte_ad_hw_updating,
.set = kvm_sbi_fwft_set_pte_ad_hw_updating,
.get = kvm_sbi_fwft_get_pte_ad_hw_updating,
},
#ifndef CONFIG_32BIT
{
.id = SBI_FWFT_POINTER_MASKING_PMLEN,
@@ -379,7 +557,7 @@ static int kvm_sbi_ext_fwft_init(struct kvm_vcpu *vcpu)
int i;
fwft->configs = kzalloc_objs(struct kvm_sbi_fwft_config,
ARRAY_SIZE(features));
ARRAY_SIZE(features), GFP_KERNEL_ACCOUNT);
if (!fwft->configs)
return -ENOMEM;
@@ -420,6 +598,32 @@ static void kvm_sbi_ext_fwft_reset(struct kvm_vcpu *vcpu)
if (feature->reset)
feature->reset(vcpu);
}
vcpu->arch.csr_dirty = true;
}
static void kvm_sbi_ext_fwft_validate(struct kvm_vcpu *vcpu)
{
struct kvm_sbi_fwft *fwft = vcpu_to_fwft(vcpu);
const struct kvm_sbi_fwft_feature *feature;
struct kvm_sbi_fwft_config *conf;
int i;
for (i = 0; i < ARRAY_SIZE(features); i++) {
feature = &features[i];
conf = &fwft->configs[i];
if (!conf->supported)
continue;
if (!feature->supported || feature->supported(vcpu))
continue;
conf->enabled = false;
conf->flags = 0;
if (feature->reset)
feature->reset(vcpu);
}
}
static unsigned long kvm_sbi_ext_fwft_get_reg_count(struct kvm_vcpu *vcpu)
@@ -574,6 +778,7 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_fwft = {
.init = kvm_sbi_ext_fwft_init,
.deinit = kvm_sbi_ext_fwft_deinit,
.reset = kvm_sbi_ext_fwft_reset,
.validate = kvm_sbi_ext_fwft_validate,
.state_reg_subtype = KVM_REG_RISCV_SBI_FWFT,
.get_state_reg_count = kvm_sbi_ext_fwft_get_reg_count,
.get_state_reg_id = kvm_sbi_ext_fwft_get_reg_id,
arch/riscv/kvm/vcpu_sbi_v01.c
+30 -22
View File
@@ -16,8 +16,9 @@
static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
struct kvm_vcpu_sbi_return *retdata)
{
ulong hmask;
int i, ret = 0;
ulong hbase = 0, hmask;
unsigned long i;
int ret = 0;
u64 next_cycle;
struct kvm_vcpu *rvcpu;
struct kvm *kvm = vcpu->kvm;
@@ -46,20 +47,25 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
ret = kvm_riscv_vcpu_unset_interrupt(vcpu, IRQ_VS_SOFT);
break;
case SBI_EXT_0_1_SEND_IPI:
if (cp->a0)
if (cp->a0) {
hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0, utrap);
else
hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1;
if (utrap->scause)
break;
for_each_set_bit(i, &hmask, BITS_PER_LONG) {
rvcpu = kvm_get_vcpu_by_id(vcpu->kvm, i);
if (!rvcpu)
continue;
ret = kvm_riscv_vcpu_set_interrupt(rvcpu, IRQ_VS_SOFT);
if (ret < 0)
if (utrap->scause)
break;
for_each_set_bit(i, &hmask, BITS_PER_LONG) {
rvcpu = kvm_get_vcpu_by_id(vcpu->kvm, i);
if (!rvcpu)
continue;
ret = kvm_riscv_vcpu_set_interrupt(rvcpu, IRQ_VS_SOFT);
if (ret < 0)
break;
}
} else {
kvm_for_each_vcpu(i, rvcpu, kvm) {
ret = kvm_riscv_vcpu_set_interrupt(rvcpu, IRQ_VS_SOFT);
if (ret < 0)
break;
}
}
break;
case SBI_EXT_0_1_SHUTDOWN:
@@ -70,29 +76,31 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run,
case SBI_EXT_0_1_REMOTE_FENCE_I:
case SBI_EXT_0_1_REMOTE_SFENCE_VMA:
case SBI_EXT_0_1_REMOTE_SFENCE_VMA_ASID:
if (cp->a0)
if (cp->a0) {
hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0, utrap);
else
hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1;
} else {
hbase = -1UL;
hmask = 0;
}
if (utrap->scause)
break;
if (cp->a7 == SBI_EXT_0_1_REMOTE_FENCE_I)
kvm_riscv_fence_i(vcpu->kvm, 0, hmask);
kvm_riscv_fence_i(vcpu->kvm, hbase, hmask);
else if (cp->a7 == SBI_EXT_0_1_REMOTE_SFENCE_VMA) {
vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
if (cp->a1 == 0 && cp->a2 == 0)
kvm_riscv_hfence_vvma_all(vcpu->kvm, 0, hmask, vmid);
kvm_riscv_hfence_vvma_all(vcpu->kvm, hbase, hmask, vmid);
else
kvm_riscv_hfence_vvma_gva(vcpu->kvm, 0, hmask, cp->a1,
kvm_riscv_hfence_vvma_gva(vcpu->kvm, hbase, hmask, cp->a1,
cp->a2, PAGE_SHIFT, vmid);
} else {
vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid);
if (cp->a1 == 0 && cp->a2 == 0)
kvm_riscv_hfence_vvma_asid_all(vcpu->kvm, 0, hmask,
kvm_riscv_hfence_vvma_asid_all(vcpu->kvm, hbase, hmask,
cp->a3, vmid);
else
kvm_riscv_hfence_vvma_asid_gva(vcpu->kvm, 0, hmask,
kvm_riscv_hfence_vvma_asid_gva(vcpu->kvm, hbase, hmask,
cp->a1, cp->a2, PAGE_SHIFT,
cp->a3, vmid);
}

Some files were not shown because too many files have changed in this diff Show More