Messages in this thread | | | Subject | Re: [PATCH v4 05/11] iommu/sva: Assign a PASID to mm on PASID allocation and free it on mm exit | From | "" <> | Date | Tue, 12 Apr 2022 23:35:32 +0800 |
| |
Hi, Fenghua
On 2022/4/12 下午9:41, Fenghua Yu wrote: > Hi, Zhangfei, > > On Tue, Apr 12, 2022 at 03:04:09PM +0800, zhangfei.gao@foxmail.com wrote: >> >> On 2022/4/11 下午10:52, Dave Hansen wrote: >>> On 4/11/22 07:44, zhangfei.gao@foxmail.com wrote: >>>> On 2022/4/11 下午10:36, Dave Hansen wrote: >>>>> On 4/11/22 07:20, zhangfei.gao@foxmail.com wrote: >>>>>>> Is there nothing before this call trace? Usually there will be at least >>>>>>> some warning text. >>>>>> I added dump_stack() in ioasid_free. >>>>> Hold on a sec, though... >>>>> >>>>> What's the *problem* here? Did something break or are you just saying >>>>> that something looks weird to _you_? >>>> After this, nginx is not working at all, and hardware reports error. >>>> Suppose the the master use the ioasid for init, but got freed. >>>> >>>> hardware reports: >>>> [ 152.731869] hisi_sec2 0000:76:00.0: qm_acc_do_task_timeout [error status=0x20] found >>>> [ 152.739657] hisi_sec2 0000:76:00.0: qm_acc_wb_not_ready_timeout [error status=0x40] found >>>> [ 152.747877] hisi_sec2 0000:76:00.0: sec_fsm_hbeat_rint [error status=0x20] found >>>> [ 152.755340] hisi_sec2 0000:76:00.0: Controller resetting... >>>> [ 152.762044] hisi_sec2 0000:76:00.0: QM mailbox operation timeout! >>>> [ 152.768198] hisi_sec2 0000:76:00.0: Failed to dump sqc! >>>> [ 152.773490] hisi_sec2 0000:76:00.0: Failed to drain out data for stopping! >>>> [ 152.781426] hisi_sec2 0000:76:00.0: QM mailbox is busy to start! >>>> [ 152.787468] hisi_sec2 0000:76:00.0: Failed to dump sqc! >>>> [ 152.792753] hisi_sec2 0000:76:00.0: Failed to drain out data for stopping! >>>> [ 152.800685] hisi_sec2 0000:76:00.0: QM mailbox is busy to start! >>>> [ 152.806730] hisi_sec2 0000:76:00.0: Failed to dump sqc! >>>> [ 152.812017] hisi_sec2 0000:76:00.0: Failed to drain out data for stopping! >>>> [ 152.819946] hisi_sec2 0000:76:00.0: QM mailbox is busy to start! >>>> [ 152.825992] hisi_sec2 0000:76:00.0: Failed to dump sqc! >>> That would have been awfully handy information to have in an initial bug report. :) >>> Is there a chance you could dump out that ioasid alloc *and* free information in ioasid_alloc/free()? This could be some kind of problem with the allocator, or with copying the ioasid at fork. >> The issue is nginx master process init resource, start daemon process, then >> master process quit and free ioasid. >> The daemon nginx process is not the original master process. >> >> master process: init resource >> driver -> iommu_sva_bind_device -> ioasid_alloc > Which code in the master process/daemon calls driver->iommu_sva_unbind_device? Our calling sequence is nginx -> openssl -> openssl engine -> kernel driver The calling entrence should be ngx_ssl_init : OPENSSL_config(NULL);
nginx: src/event/ngx_event_openssl.c ngx_ssl_init if (OPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL) == 0)
I add some print.
/usr/local/nginx$ sudo sbin/nginx ngx_ssl_init pid=2361 bind_fn ngx_openssl_create_conf pid=2361 hisi sec init Kunpeng920! ngx_ssl_create pid=2361 ngx_ssl_certificates pid=2361 ngx_ssl_certificate pid=2361 uadk_e_wd_digest_init hisi sec init Kunpeng920! ngx_ssl_ciphers pid=2361 ngx_daemon pid=2361 fork daemon master pid=2361 will exit // here master process is exit fork return 0 pid=2364 // here daemon process started ngx_daemon fork ngx_pid=2364, ngx_parent=2361 $ ps -aux | grep nginx root 2364 0.0 0.0 31324 15380 ? Ssl 15:21 0:00 nginx: master process sbin/nginx nobody 2366 0.0 0.0 32304 16448 ? Sl 15:21 0:00 nginx: worker process linaro 2371 0.0 0.0 7696 2048 pts/0 S+ 15:22 0:00 grep --color=auto nginx
nginx src/os/unix/ngx_daemon.c ngx_daemon(ngx_log_t *log) { int fd;
switch (fork()) { case -1: ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "fork() failed"); return NGX_ERROR;
case 0: // here fork daemon process break;
default: // master process directly exit, and release mm as well as ioasid exit(0); }
// only daemon process ngx_parent = ngx_pid; ngx_pid = ngx_getpid();
> >> nginx : ngx_daemon >> fork daemon, without add mm's refcount. >> >> src/os/unix/ngx_daemon.c >> ngx_daemon(ngx_log_t *log) >> { >> int fd; >> >> switch (fork()) { >> case -1: >> ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "fork() failed"); >> return NGX_ERROR; >> >> case 0: // here master process is quit directly and will be >> released. >> break; >> >> default: >> exit(0); >> } >> // here daemon process take control. >> ngx_parent = ngx_pid; >> ngx_pid = ngx_getpid(); >> >> >> fork.c >> copy_mm >> if (clone_flags & CLONE_VM) { >> mmget(oldmm); >> mm = oldmm; >> } else { >> mm = dup_mm(tsk, current->mm); // here daemon process >> handling without mmget. >> >> master process quit, mmput -> mm_pasid_drop->ioasid_free >> But this ignore driver's iommu_sva_unbind_device function, >> iommu_sva_bind_device and iommu_sva_unbind_device are not pair, So driver >> does not know ioasid is freed. >> >> Any suggestion? > ioasid is per process or per mm. A daemon process shouldn't share the same > ioasid with any other process with even its parent process. Its parent gets > an ioasid and frees it on exit. The ioasid is gone and shouldn't be used > by its child process. > > Each daemon process should call driver -> iommu_sva_bind_device -> ioasid_alloc > to get its own ioasid/PASID. On daemon quit, the ioasid is freed. > > That means nqnix needs to be changed.
Agree with Dave, I think user space should not be broken.
Thanks
| |