lkml.org 
[lkml]   [2023]   [Feb]   [15]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)
    From
    Date

    Hi, Wentong,

    On 2/15/23 5:43 PM, Laurent Pinchart wrote:
    > Hello Wentong,
    >
    > On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
    >> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
    >> companion chip designed to provide secure and low power vision capability
    >> to IA platforms. IVSC is available in existing commercial platforms from
    >> multiple OEMs.
    >>
    >> The primary use case of IVSC is to bring in context awareness. IVSC
    >> interfaces directly with the platform main camera sensor via a CSI-2 link
    >> and processes the image data with the embedded AI engine. The detected
    >> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
    >> fusion from multiple sensors. The fusion results are used to implement
    >> advanced use cases like:
    >> - Face detection to unlock screen
    >> - Detect user presence to manage backlight setting or waking up system
    >
    > Do you have plan to support these features in the ivsc driver in the
    > future ?
    >
    >> Since the Image Processing Unit(IPU) used on the host processor needs to
    >> configure the CSI-2 link in normal camera usages, the CSI-2 link and
    >> camera sensor can only be used in mutually-exclusive ways by host IPU and
    >> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
    >> driver can take ownership of the CSI-2 link and camera sensor using
    >> interfaces provided by this IVSC driver.
    >>
    >> Switching ownership requires an interface with two different hardware
    >> modules inside IVSC. The software interface to these modules is via Intel
    >> MEI (The Intel Management Engine) commands. These two hardware modules
    >> have two different MEI UUIDs to enumerate. These hardware modules are:
    >> - ACE (Algorithm Context Engine): This module is for algorithm computing
    >> when IVSC owns camera sensor. Also ACE module controls camera sensor's
    >> ownership. This hardware module is used to set ownership of camera sensor.
    >> - CSI (Camera Serial Interface): This module is used to route camera
    >> sensor data either to IVSC or to host for IPU driver and application.
    >>
    >> IVSC also provides a privacy mode. When privacy mode is turned on,
    >> camera sensor can't be used. This means that both ACE and host IPU can't
    >> get image data. And when this mode is turned on, host IPU driver is
    >> informed via a registered callback, so that user can be notified.
    >
    > How does the privacy mode work, and how can the user trust that the
    > closed-source IVSC and IME firmwares will honour the privacy settings ?
    >

    Continue with question from Laurent,

    How IVSC handle the privacy request from user? Is there some notification
    mechanism to user-space? I'd have concern if IVSC driver need private callback
    to request back-end(e.g. ISP driver) to handle stream cutting.

    >> In summary, to acquire ownership of camera by IPU driver, first ACE
    >> module needs to be informed of ownership and then to setup MIPI CSI-2
    >> link for the camera sensor and IPU.
    >>
    >> Implementation:
    >> There are two different drivers to handle ACE and CSI hardware modules
    >> inside IVSC.
    >> - mei_csi: MEI client driver to send commands and receive notifications
    >> from CSI module.
    >> - mei_ace: MEI client driver to send commands and get status from ACE
    >> module.
    >> Interface is exposed via ivsc.h to acquire and release camera sensor and
    >> CSI-2 link.
    >
    > Do I understand correctly, from your diagram below, that the
    > communication between the IME and IVSC goes through SPI ?
    >
    >> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
    >> -----------------------------------------------------------------------------
    >> | Host Processor |
    >> | |
    >> | ----------------- ----------------- --------------- |
    >> | | | | | | | I2C |
    >> | | IPU | | ISH | |camera driver|--| |
    >> | | | | | | | | |
    >> | ----------------- ----------------- --------------- | |
    >> | | | | | |
    >> | | | --------------- | |
    >> | | | | | | |
    >> | | | | IVSC driver | | |
    >> | | | | | | |
    >> | | | --------------- | |
    >> | | | | | |
    >> ----------------|-----------------------|----------------------|---------|---
    >> | CSI | I2C |SPI |
    >> | | | |
    >> ----------------|-----------------------|---------------- | |
    >> | IVSC | | | |
    >> | | | | |
    >> | ----------------- ----------------- | | |
    >> | | | | | | | |
    >> | | CSI | | ACE | |------| |
    >> | | | | | | |
    >> | ----------------- ----------------- | |
    >> | | | I2C | |
    >> ----------------|-----------------------|---------------- |
    >> | CSI | |
    >> | | |
    >> -------------------------------- |
    >> | | I2C |
    >> | camera sensor |-----------------------------|
    >> | |
    >> --------------------------------
    >>
    >> Wentong Wu (3):
    >> media: pci: intel: ivsc: Add CSI submodule
    >> media: pci: intel: ivsc: Add ACE submodule
    >> media: pci: intel: ivsc: Add acquire/release API for ivsc
    >>
    >> drivers/media/pci/Kconfig | 1 +
    >> drivers/media/pci/intel/Makefile | 2 +
    >> drivers/media/pci/intel/ivsc/Kconfig | 12 +
    >> drivers/media/pci/intel/ivsc/Makefile | 7 +
    >> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
    >> drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
    >> drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
    >> drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
    >> drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
    >> include/linux/ivsc.h | 74 ++++
    >> 10 files changed, 1090 insertions(+)
    >> create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
    >> create mode 100644 drivers/media/pci/intel/ivsc/Makefile
    >> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
    >> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
    >> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
    >> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
    >> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
    >> create mode 100644 include/linux/ivsc.h
    >

    --
    Best regards,
    Bingbu Cao

    \
     
     \ /
      Last update: 2023-03-27 00:24    [W:7.410 / U:0.156 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site