Android MultiMedia Framework

Yes

1. MediaPlayer Architecture

MediaPlayer is the server, MediaPlayerService and MediaPlayerService :: Client is the client.
MediaPlayerService realize the IMediaPlayerService, the main function is to create the right player through the url sent by MediaPlayer::setDataSource
MediaPlayerService :: Client realize the IMediaPlayer, the main function is to call the player created by MediaPlayerService to do those specific start, stop, resume, pause…

Entering NuplayerDriver means entering Android MultiMedia Framework.

2. NuPlayer

The playback video is mainly completed through Nuplayer. The figure below includes parser, decoder and render.

NuPlayer::Source is the parser module. Its interface looks like a combination of MediaExtractor and MediaSource
NuPlayer::Decoder connects to CCodec for decoding. CCodec has a state pattern and pass MediaBuffers around with messages.
NuPlayer::Render is responsible for rendering audio and also controls when to post video buffers back to NativeWindow for A/V sync.

3. Decoding Framework

3.1 Framework of i.MX8MP, i.MX8MQ and i.MX8MM

Decoding Process between the framework, vendor decoder component and kernel as follows on i.MX8MP, i.MX8MQ and i.MX8MM.

3.2 Framework of i.MX8QM

Decoding Process between the framework, vendor decoder component and kernel as follows on i.MX8QM.

stream on: start to decode/encode
stream off: stop decoding/encoding
qbuf: Queue the v4l2buffer filled in the decoder into the buffer queue created in the vpu driver so that the VPU can obtain it for decoding.
dqbuf: When the VPU decoding is completed, the buffer will be dequeueed so that the decoder/codec can continue to be used.

4. Decode Video and Display Process

4.1 i.MX8MM and i.MX8MP

For i.MX8MP and i.MX8MM, G2D is used to composite layers, and the buffer transmission is completed between Decoder and SurfaceFlinger through the BufferQueue mechanism. The process as follows:

BufferQueue: As the producer of BufferQueue, decoder provides the decoded buffer, and SurfaceFlinger, as the consumer of BufferQueue, composite the decoded video layer.
Display: Use G2D to composite Android layer and video layer on i.MX8MP. SurfaceFlinger will hand over all layers to Display HAL, and then composite them into the framebuffer by G2D.

4.2 i.MX8MQ

For i.MX8MQ, GPU3D is used to composite Android UI, DCSS is used to composite overlay and android UI. And the buffer transmission is completed between Decoder and SurfaceFlinger through the BufferQueue mechanism. The processing is as follows:

BufferQueue: Decoder provides the decoded buffer as producer. SurfaceFlinger as the cosumer of BufferQueue.
Display: Using GPU3D to composite Android UI layers. And video layer will be submitted to DCSS (Display Controller) through Display HAL.
DCSS: Responsible for combining video overlay and Android UI for display.
GLESRenderEngine: SurfaceFlinger calls the opengl interface through GLESRenderEngine to complete rendering composite.

4.3 i.MX8QM

For i.MX8QM, GPU3D or DPU are used to composite. And the buffer transmission is completed between Decoder Filter Component and SurfaceFlinger through the BufferQueue mechanism. The processing as follows:

Decoder Filter: Because GPU3D cannot directly composite TILED type layers, it needs to be converted into Linear through Decoder Filter first, and then handed over to SurfaceFlinger.
BufferQueue: Decoder Fillter will receive the outputbuffer from Decoder. And Filter will be as producer of BufferQueue to provide the buffer. SurfaceFlinger will be as consumer of BufferQueue to consume the buffers.
GLESRenderEngine: SurfaceFlinger calls the opengl interface through GLESRenderEngine to complete rendering composite.

Note: On i.MX8QM, Not all situations are composited using the GPU3D. Please refer to the table below.

4.4 DRM Widevine (Secure Decoder)

Now we have enabled DRM Widevine on i.MX8MP/i.MX8MQ/i.MX8QM so that secure video can be played.

For i.MX8MP, Use OEMCrypto trusted applicaiton to decrypt the encrypted stream. And Use RDC/CSU to protected hardware for secure pipeline. The framework is as follows:

For i.MX8MQ, Use OEMCrypto trusted applicaiton to decrypt the encrypted stream. And Use RDC/CSU to protected hardware for secure pipeline. The framework is as follows:

OEMCrypto: It is a library as tipc client to send the encrypted data to Trusty OS.
OEMCrypto Trusted Application: Used to decrypt the protected data into secure memory and send it to Media Framewrok.
Secure Framebuffer: It is allocated from secure heap through libdmabufheap.
Secure heap: Used to allocate secure memory. Resources in domain2 can read and write secure memory, but cannot write normal memory. Resources in domain0 can write to secure memory, but cannot read normal memory. When playing secure video, VPU, GPU2D, and lcdif will be in domain2. Or they are in domain0.
Resource Domain Controller (RDC): It provides support for the isolation of destination memory mapped locations such as peripherals and memory to a single core, a bus master, or set of cores and bus masters.
CSU Central Security Unit (CSU) : 1. Peripheral Access Policy - the appropriate bus master privilege and identity are required to access each peripheral. 2. Masters Privilege Policy - the CSU overrides the bus master privilege signals (secure/non-secure). Configure the VPU so that it can only be accessed by the secure world.

For i.MX8QM, also use OEMCrypto trusted applicaiton to decrypt the encrypted stream. But related hardware and memory are protected through the secure partition created by SCU. The framework is as follows:

OEMCrypto: It is a library as tipc client to send the encrypted data to Trusty OS.
OEMCrypto Trusted Application: Used to decrypt the protected data into secure memory and send it to Media Framewrok.
Secure Framebuffer: It is allocated from secure heap through libdmabufheap.
Secure heap: Used to allocate secure memory. Only Secure partition can access secure memory.
Frimware Loader: It is a trusted application in Trusty OS, It is responsible for loading the encrypted firmware into the specified memory.
Secure Partition: Secure partitions are created using SCU and all hardware that needs to be protected are moved to secure partitions to isolate it from non-secure partitions.

5. Encoding Process

5.1 Encoding Process on i.MX8MP

BufferQueue: MediaFramework will create Surface and create BufferQueue, SurfaceFlinger will serve as the producer to provide the composite layers, and the Encoder component will encode it as the consumer of the BufferQueue.
MPEG4Writer: Responsible for writing the VPU-encoded data to the output file.

5.2 Encoding Process on i.MX8QM and i.MX8MM

BufferQueue: MediaFramework will create Surface and create BufferQueue, SurfaceFlinger will serve as the producer to provide the composite layer, and the Encoder Filter component will be as the consumer of the BufferQueue.
MPEG4Writer: Responsible for writing the VPU-encoded data to the output file.

6. Buffer transfer and management

6.1 Transfer and release process of Input Buffer

Input Buffer allocation: It is allocated in CCodecBufferChannel, and it is used to allocate inputbuffer and recycled.
InputManager: Before the queue buffer into the decoder, it will be registered with the InputManager. When the VPU is finished using it, the InputManager will be notified to release it.
Use of InputBuffer: After the buffer is queued into the decoder, the input buffer information will be copied to v4l2buffer so that the VPU can use it.

Note: For DRM Secure decoder, the VPU will only use the paddr of inputbuffer.

6.2 Transfer and release process of Output Buffer

Output Buffer allocation: When use Surface for output, It is allocated through BufferQueueAllocator that is created in CCodecBufferChannel. It is used to allocate outputBuffer.
Management: When use Surface, OutputBuffer is managed through the BufferQueue mechanism. When the VPU filled the outputBuffer with data that can be displayed, it will notify the Consumer to acquire the buffer.
Use of OutputBuffer: In the decoder, the output buffer information will be copied to v4l2buffer so that the VPU can use it.

Note: For situations where OutputSurface is not used, GrallocAllocator is used by default instead of BufferQueueAllocator.

6.3 Buffer Management of Encoder

For screen recording situations, the Encoder's buffer transfer is also managed through the BufferQueue mechanism. After SurfaceFlinger is producer to fill the GraphicBuffer of BufferQueue, the encoder is as consumer to encode the composited data.