Messages in this thread | | | From | Oded Gabbay <> | Subject | [PATCH 0/8] Fixing DMA mask issues in habanalabs driver | Date | Tue, 11 Jun 2019 08:50:37 +0300 |
| |
This patch-set changes the way the Goya internal CPU access memory on the Host machine. This is needed to prevent the non-standard way the driver used the PCI DMA set mask kernel API so far.
The DMA set mask should be called only once at the start of the driver. This is because changing the DMA mask to a new value after allocations were made using a previous mask value, might cause the previous allocations to become unaccessible (usually if there is IOMMU present).
The driver did that because of a limitation in Goya's internal CPU. The limitation was that the internal CPU can only access 40-bit addresses, while the entire ASIC can access 50-bit addresses. Therefore, the driver set the DMA mask to 39-bits, allocated memory for the internal CPU on the host and then changed the DMA mask to 48-bits.
This patch-set eliminates the double DMA set by using Goya's MMU to overcome the limitation. The driver now sets the DMA mask only once to 48-bits and allocates a single DMA region of 2MB for the internal CPU. It then maps that region in Goya's MMU to a device virtual address under 40-bits.
In addition, this patch-set enables the use of 64-bit mask on POWER9 systems. POWER9 DMA mask can be set ONLY to 32-bit or 64-bit. To use 64-bit, the device must set bit 59 to 1 in all its outbound transactions. This is achieved by setting a special configuration in Goya's PCIe controller. The configuration must be done only in POWER9 machines, as it will make the device non-functional on other architectures (e.g. x86-64, ARM).
Thanks, Oded
Oded Gabbay (8): habanalabs: initialize device CPU queues after MMU init habanalabs: de-couple MMU and VM module initialization habanalabs: initialize MMU context for driver habanalabs: add MMU mappings for Goya CPU habanalabs: set Goya CPU to use ASIC MMU habanalabs: remove DMA mask hack for Goya habanalabs: add WARN in case of bad MMU mapping habanalabs: enable 64-bit DMA mask in POWER9
drivers/misc/habanalabs/asid.c | 2 +- drivers/misc/habanalabs/context.c | 7 + drivers/misc/habanalabs/debugfs.c | 7 +- drivers/misc/habanalabs/device.c | 45 +++-- drivers/misc/habanalabs/goya/goya.c | 234 +++++++++++++++++------ drivers/misc/habanalabs/goya/goyaP.h | 12 +- drivers/misc/habanalabs/habanalabs.h | 9 +- drivers/misc/habanalabs/habanalabs_drv.c | 7 + drivers/misc/habanalabs/memory.c | 13 +- drivers/misc/habanalabs/mmu.c | 20 +- drivers/misc/habanalabs/pci.c | 7 +- 11 files changed, 259 insertions(+), 104 deletions(-)
-- 2.17.1
| |