lkml.org 
[lkml]   [2020]   [Jun]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    Patch in this message
    /
    From
    Subject[PATCH 5.7 210/477] habanalabs: increase timeout during reset
    Date
    From: Oded Gabbay <oded.gabbay@gmail.com>

    [ Upstream commit 7a65ee046b2238e053f6ebb610e1a082cfc49490 ]

    When doing training, the DL framework (e.g. tensorflow) performs hundreds
    of thousands of memory allocations and mappings. In case the driver needs
    to perform hard-reset during training, the driver kills the application and
    unmaps all those memory allocations. Unfortunately, because of that large
    amount of mappings, the driver isn't able to do that in the current timeout
    (5 seconds). Therefore, increase the timeout significantly to 30 seconds
    to avoid situation where the driver resets the device with active mappings,
    which sometime can cause a kernel bug.

    BTW, it doesn't mean we will spend all the 30 seconds because the reset
    thread checks every one second if the unmap operation is done.

    Reviewed-by: Omer Shpigelman <oshpigelman@habana.ai>
    Signed-off-by: Oded Gabbay <oded.gabbay@gmail.com>
    Signed-off-by: Sasha Levin <sashal@kernel.org>
    ---
    drivers/misc/habanalabs/habanalabs.h | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)

    diff --git a/drivers/misc/habanalabs/habanalabs.h b/drivers/misc/habanalabs/habanalabs.h
    index 31ebcf9458fe1..a6dd8e6ca594c 100644
    --- a/drivers/misc/habanalabs/habanalabs.h
    +++ b/drivers/misc/habanalabs/habanalabs.h
    @@ -23,7 +23,7 @@

    #define HL_MMAP_CB_MASK (0x8000000000000000ull >> PAGE_SHIFT)

    -#define HL_PENDING_RESET_PER_SEC 5
    +#define HL_PENDING_RESET_PER_SEC 30

    #define HL_DEVICE_TIMEOUT_USEC 1000000 /* 1 s */

    --
    2.25.1


    \
     
     \ /
      Last update: 2020-06-23 23:38    [W:2.568 / U:0.236 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site