Freescale Android Media codecs and patents: is there any patent to pay before releasing to third party?

[DiegoFSL]

Hi everybody,

I'd like to ask a questions to which I can't seem to find a clear answer.

Freescale i.MX6Q SOC (this is the one we use the most, but some considerations apply also to others) has features that allow to decode media streams (audio and video) in hardware.

In particular, the "i.MX_Android_Extended_Codec_Release_Notes.pdf" document for the Freescale Android release, lists several "Freescale enhanced codecs".

The release description says that "Only codecs that have no license restriction are included in omxplayer package."

What does exactly "no license restriction" stand for? In particular I'm trying to figure out if I need to pay for patents related to some of those codecs, for example mp3, aac, H.264 and so on. I mean, is the decoding / encoding functionality implemented in hardware, so that there's no patent encumbered functionality in the software?

Can anybody give an indication, and eventually list special cases if there are any?

Thanks in advance!

[DiegoFSL]

Ok, so it all goes back to "it would be nice to have an easy option to remove proprietary CODEC decode/encode technology software from Freescale BSPs" as also asked here:

Re: How to disable i.MX6DQ video/audio codec.

[stephencox · Accepted solution]

Hello Diego,

The landscape of codecs at Freescale, their different license terms and how they need to be "handled" is indeed confusing. Let me see if I can clarify the i.MX codec/extended codec scenario:

• i.MX processors include fundmental media acceleration technology but do not include licensed hardware IP to accelerate a particular proprietary codec (Dolby, REAL or the like). Some of our DSP products do have licensed hardware acceleration IP (requiring a royalty -- paid by FSL) but not the i.MX line. Consequently there is no codec royalty required to use or distribute an i.MX  product from a hardware standpoint.
• Software Codecs at Freescale come in three basic flavors:
  • Restricted Access Proprietary Codecs (usually with restricted use & distribution terms) such as Dolby, Windows Media (Microsoft) and REAL... these proprietary codecs can ONLY be delivered to customers that are licensed by the codec company (as verified by visiting the codec company's licensee listing website). After verification Freescale can provide these codecs to customers via a separate, monitored (and licensed) download.
  • Restricted Use/distribution codecs such as MPEGLA (MP3, H.264...), AAC... these are also considered proprietary codecs BUT the licensors allow Freescale to distribute them in our standard BSP (or extended package) for internal use and demonstration purposes  -- any commercial distribution requires the customer to engage directly with the Licensor to garner a proper license. These terms are included in the appendix of the license you clicked through to take delivery of the Android BSP.
  • Open source or "unrestricted use" codecs like OPUS (many others). We include several of these codecs in our BSP/extended package (typically only requiring proper copyrighting for use and distribution) BUT as our click-through license suggests, all open source software (codec or other) that we distribute falls under the associated open source license for that code (BSD, LGPL...) and customers should find and review the salient license if they plan to distribute.
• We do not include any Restricted Access Proprietary codecs (Dolby, Windows Media, REAL) with the standard BSP or in the Android extended multimedia package -- again those codecs require a "request and license verification" maneuver before we could offer access.
• We do include Restricted use and open source/unrestricted codecs in both our BSP and the Media extensions package.
  • There is no royalty for internal use/development or demonstration (please see the click through license and appendix for details)
  • Many of these codecs fall under simple copyright restrictions (See their individual licenses if you plan to distribute an end product that includes them.)
  • Some of the Restricted use/distribution codecs like MP3, H.264 (all MPEGLA technologies) require customers to sign a commercial distribution license directly with the licensor BEFORE you ship end products that include their technology (these are identified in the license appendix).

Hopefully I answered more questions than I created!

[DiegoFSL]

Hi Stephen,

after looking deeper into Android Freescale BSP I think my question is still partly unresponded, and I'm about to explain why.

The part I'm referring to specifically is:

Stephen Cox:

• Restricted Use/distribution codecs such as MPEGLA (MP3, H.264...), AAC... these are also considered proprietary codecs BUT the licensors allow Freescale to distribute them in our standard BSP (or extended package) for internal use and demonstration purposes  -- any commercial distribution requires the customer to engage directly with the Licensor to garner a proper license. These terms are included in the appendix of the license you clicked through to take delivery of the Android BSP.

While for audio  decoders, encoders and parsers I can actually find the software pieces that do the decoding inside the Android Freescale BSP and run on the CPU, I can't find the same for video decoders and encoders, as I just find software that seems to be communicating with a hardware device (not the CPU) which actually performs decoding and encoding.

Let me detail that with two examples.

MP3 decoding

I can find the OMX component that is registered to OMX core in the file external/fsl_imx_omx/OpenMAXIL/release/registry/component_register.

The OMX component OMX.Freescale.std.audio_decoder.mp3.sw-based that has the audio_decoder.mp3 role uses the lib_omx_mp3_dec_v2_arm11_elinux.so.

As can be seen in external/fsl_imx_omx/OpenMAXIL/src/component/mp3_dec/Android.mk the lib_omx_mp3_dec_v2_arm11_elinux.so library uses and links to device/fsl-codec/lib/lib_mp3_dec_v2_arm12_elinux.so, which is the software piece, provided by Freescale, that runs on the CPU so it might require licensing as you detailed.

H.264 / MPEG4 AVC decoding

I can find the OMX component that is registered to OMX core in the file external/fsl_imx_omx/OpenMAXIL/release/registry/component_register.

The OMX component OMX.Freescale.std.video_decoder.avc.v3.hw-based that has the video_decoder.avc role uses the lib_omx_vpu_dec_v2_arm11_elinux.so.

As can be seen in external/fsl_imx_omx/OpenMAXIL/src/component/vpu_dec_v2/Android.mk the lib_omx_vpu_dec_v2_arm11_elinux.so library uses and links to libvpu.so, which is obtained by building the files in external/linux-lib/vpu/ provided by Freescale.

To me it looks like this software just opens /dev/mxc_vpu device and passes buffers so that hardware VPU can decode incoming video streams.

So, I'm back to the question: am I right supposing that no license is needed for computing (decoding) which is performed on the VPU? After all I won't be shipping any software performing video decoding, as libvpu.so seems to be just passing back and forth data.

Thanks,

Diego

P.S.: I know software fallback using libav gets installed by default, but don't take that into consideration, as I'm willing to remove the fallback and the library altogether.

[stephencox]

Yeah... it actually doesn't matter whether or how much hardware assist is leveraged in the decode pipeline.

The bottom line from a commercial distribution license standpoint is ... if the software codec enables a salient stream (MP3, MP4, h.264...) to be decoded on the system/SoC (regardless of whether some or most of that decode is performed by the VPU vs the CPU) then MPEGLA (other restricted codec licensors) consider that as royalty bearing distribution.

Audio codecs are somewhat "lighter" on resources so they are often decoded in the CPU of modern SoC's (i.MX6). Video codecs continue to require HW assist as they scale to crazy resolutions, frame rates and multiple display/planes... and FSL spends significant effort to tune the video codecs to take advantage of our VPU hardware. But, at the end of the day -- from a commercial distribution license/royalty standpoint -- they are treated the same. 

[DiegoFSL]

Thank you again Stephen,

The bottom line from a commercial distribution license standpoint is ... if the software codec enables a salient stream (MP3, MP4, h.264...) to be decoded on the system/SoC (regardless of whether some or most of that decode is performed by the VPU vs the CPU) then MPEGLA (other restricted codec licensors) consider that as royalty bearing distribution.

This somewhat puzzles me, as I always though the "equation" was:

• you ship software containing patented algorithms (where those patents are valid of course) then you have to obtain patent license;
• you don't ship software containing patented algorithms then you don't have to obtain patent license.

Following your statement the "equation" should be more like:

• you ship a system (hw + sw) which is able to play patented XYZ audio / video content out of the box then you have to obtain patent license;
• you ship a system (hw + sw) which is not able to play patented XYZ audio / video content out of the box then you don't have to obtain patent license.

As is in the product we have in mind the target is to use only one single video format with no audio of any kind (audio output won't even be connected) our target is to remove the ability to play any unused codec, to avoid obtaining and paying license for stuff that we have no use for.

I don't know if you can reply to the following question officially from Freescale's point of view, but I gladly accept even "personal" and "not a lawyer's perspective" kind of answers: would properly stripping down libvpu to allow decoding of only the desired format allow me to obtain license only for that single format?

I think the answers is "yes" or else any hardware "H.264 capable" would require obtaining patent license. But then every modern CPU would be "H.264 capable"...

Thank you Stephen, and again thank you for following me in this long thread and taking the time to provide your kind replies.

Diego

[stephencox]

There's no need to remove or de-feature general functionality that can be utilized for playback of various codecs. It is designed to be beneficial to a broad array of video codecs but not IP specific or royalty bearing.

Just because there is generic video acceleration technology in the VPU pipeline that can benefit h.264 playback (or DivX, or MPEG4) does not mean you have to pay a royalty for those codecs.

If you load proprietary CODEC decode/encode technology software on to your platform that specifically enables playback of that codec, then you must adhere to the license associated with distribution.

[DiegoFSL]

Hi Stephen,

thank you very much for your detailed and precise answer. I had a partial idea of what you explained, but your reply solved all my doubts. I hope your answer will be useful also to other people.

Again, thank you very much Stephen,

regards,

Diego

[saurabh206]

Hi,

Its better you can contact Freescale local SALES person for the clarification.

[DiegoFSL]

Thank you Saurabh for your kind reply,

while I don't expect a one size fits all reply, I think Freescale could publicly explain the relation between their multimedia decoding hardware and most common patents (e.g. H.264).

Thanks.