This message generated a parse failure. Raw output follows here. Please use 'back' to navigate. From devnull@lkml.org Wed May 29 02:42:12 2024 >From spaans Fri Jul 24 01:15:38 2015 Received: from pop.ziggo.nl [212.54.42.19] by squeeze.vs19.net with POP3 (fetchmail-6.3.21) for (single-drop); Fri, 24 Jul 2015 01:15:38 +0200 (CEST) Received: from md1.tb.mail.iss.local ([212.54.42.141]) by mc5.tb.mail.iss.local (Dovecot) with LMTP id Qc3kMZN0sVW/FgAAh0Kp1A for ; Fri, 24 Jul 2015 01:12:23 +0200 Received: from mx13.tb.mail.iss.as9143.net ([212.54.42.141]) by md1.tb.mail.iss.local (Dovecot) with LMTP id anu2HHTxLFUEQAAAP8oO3g ; Fri, 24 Jul 2015 01:08:25 +0200 Received: from vger.kernel.org ([209.132.180.67]) by mx13.tb.mail.iss.as9143.net with esmtp (Exim 4.82) (envelope-from ) id 1ZIPf5-0007CQ-GR for bmw12-lkml@ziggo.nl; Fri, 24 Jul 2015 01:12:23 +0200 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754207AbbGWXMM (ORCPT ); Thu, 23 Jul 2015 19:12:12 -0400 Received: from rrw.me.uk ([93.93.129.10]:38204 "EHLO rrw.me.uk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753642AbbGWXMK (ORCPT ); Thu, 23 Jul 2015 19:12:10 -0400 X-Greylist: delayed 1553 seconds by postgrey-1.27 at vger.kernel.org; Thu, 23 Jul 2015 19:12:10 EDT Received: from 201.81.2.81.in-addr.arpa ([81.2.81.201] helo=[172.19.92.20]) by rrw.me.uk with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1ZIPFn-0008Be-2a; Thu, 23 Jul 2015 23:46:15 +0100 Message-Id: <55B16EA9.3020609@kynesim.co.uk> Date: Thu, 23 Jul 2015 23:46:01 +0100 From: Richard Watts User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 Mime-Version: 1.0 To: Greg KH Cc: linux-serial@vger.kernel.org, linux-usb@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 1/1] Avoid usb reset crashes by making tty_io cdevs truly dynamic References: <555B518D.2010102@kynesim.co.uk> <20150723220854.GA24492@kroah.com> In-Reply-To: <20150723220854.GA24492@kroah.com> Content-Type: multipart/mixed; boundary="------------060900040608040805050000" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-Id: X-Mailing-List: linux-kernel@vger.kernel.org X-Ziggo-spamsetting: Instelling=Gemiddeld Scorelimiet=14 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.1 cv=HL4+bdZv c=1 sm=0 tr=0 a=UK1r566ZdBxH71SXbqIOeA==:17 a=k3qmffNXAAAA:8 a=VwQbUJbxAAAA:8 a=cIF5Tx0qAAAA:8 a=H5aKjGBIAAAA:8 a=hTJJpgObMeYA:10 a=zOBTXjUuO1YA:10 a=r77TgQKjGQsHNAKrUKIA:9 a=gZbpxnkM3yUA:10 a=v_q-BkzkgbWEs_G8kYkA:9 a=pIL X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No This is a multi-part message in MIME format. --------------060900040608040805050000 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Hi, Sure - sorry, my description was a little .. basic. So, I have a client who was having problems with machines hanging in the field. Very rare, associated with a h/w change that introduced more cores. Kernel dumps implied that the timer list was getting corrupted. This configuration of machine is an SBC on a board which communicates with the SBC (partly) via a USB CDC device, which pops up as /dev/ttyACM0. So one of the things we turned on was CONFIG_DEBUG_KOBJECT_RELEASE. One of the side-effects of this is to delay kobject destruction. When we did that, we could reproduce the crash by performing a USB reset on the CDC device - and logs suggest that this was happening in the field too. When the USB reset happens, we get a bunch of complaints from the kernel. Some of these are to do with races on the kobjects associated with the sysfs entries for the ttyACM0 device. They turn out not to be fatal, and have their own patch series ('Attempt to cope with device changes and delayed kobject deallocation' on linux-kernel). The fatal one turns out to be an execution path that goes like this: 1 USB device declares itself to be CDC 2 tty driver fires up and allocates a cdev for the relevant tty. 3 driver->cdevs[0].kobj gets initialised as part of the cdev_alloc() 4 USB reset happens, queueing driver->cdevs[0].kobj for release. 5 The tty driver calls cdev_init(&driver->cdevs[0]), which reinitialises driver->cdevs[0].kobj with a refcount of 1. 6 tty driver starts using that new cdev, queueing an operation on it. This causes a timer entry to be added including the kobj. 7 At this point, the release we scheduled in (4) happens and the members of kobj are deallocated. 8 Someone allocates the newly released memory for one of the members of cdriver->cdevs[0].kobj somewhere else and overwrites it. 9 The timer goes off. 10 Boom My patch (ham-fistedly) fixes this by ensuring that because we never reuse the cdev pointer, we are never fooled into reinitialising a kobject queued for deletion. I'm not all that familiar with how the locking should go here, and there is a definite argument that under non CONFIG_DEBUG_KOBJECT_RELEASE conditions, the kobject_release() would have happened by 5, and therefore this situation should never exist "for real". .. but (a) that makes it rather hard to test kernels with CONFIG_DEBUG_KOBJECT_RELEASE, and (b) my customer's crashes have (allegedly) now gone away even without CONFIG_DEBUG_KOBJECT_RELEASE set. Does that help at all? I've attached my 0/1, just in case that got lost somewhere. Richard. --------------060900040608040805050000 Content-Type: message/rfc822; name="Attached Message" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="Attached Message" Message-ID: <555B5100.8010708@kynesim.co.uk> Date: Tue, 19 May 2015 16:04:32 +0100 From: Richard Watts User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: linux-kernel@vger.kernel.org, gregkh@linuxfoundation.org Subject: [PATCH 0/1] Avoid usb reset crashes by making tty_io cdevs truly dynamic Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Sometimes, usb buses on which CDC ACM devices sit encounter a usb reset. When this happens, particularly when CONFIG_DEBUG_KOBJECT_RELEASE is on, we attempt to destroy the cdev for the associated tty and then rapidly re-initialise it. Since kobject destruction is not immediate, this potentially leaves us with cdev_init() calling kobject_init() on a kobject that is about to be destroyed. This turns out not to be such a good thing and this patch solves the problem by making the cdevs tty_operations->cdevs dynamically allocated. This may not be a problem in the wild (though I have some circumstantial evidence that it is), but I submit that we might want to think about fixing it anyway, since it makes debugging on systems with CONFIG_DEBUG_KOBJECT_RELEASE=y and USB resets rather difficult (guess what I have been doing lately .. ). Patch is against e26081808edadfd257c6c9d81014e3b25e9a6118 (head of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty.git ). (in fact, you will still get an oops - which is the subject of another, more controversial, patchset ..) Richard. --------------060900040608040805050000-- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/