Raspberry Pi: Multimedia Programming
Reference: Raspberry Pi GPU Audio Video Programming
The GPU provides OpenGL ES 2.0, hardware-accelerated OpenVG, and 1080p30 H.264 high-profile encode and decode. While the CPU has changed in the different models, the GPU has remained the same.
• RPi OpenMAX forum
• RPi Graphics forum
• RPi OpenGL ES forum
• Raspberry Pi VideoCore APIs
• Linux Gizmos reports on many SoCs including the RPi
• Tutorial: VLC with hardware acceleration on Raspberry Pi
• How to draw 2D images using OpenGL, in SDL
• Hardware Accelerated Qt Multimedia Backend for Raspberry Pi and OpenGL Shaders on Video
• Decoding and Rendering Compressed Images with OpenMAX on Raspberry Pi
The Khronos Group is a not-for-profit industry consortium creating open standards including the following:
Resources
• Khronos Group
/etc/issue
/etc/issue is a text file which contains a message or system identification to be printed before the login prompt. /etc/os-release
Operating system identification. os-release contains data that is defined by the operating system vendor and should generally not be changed by the administrator.
For each language (C/C++) compiled by GCC, there is a standard GCC tries to follow.
To select a standard,
The Raspberry Pi contains a Broadcom VideoCore IV GPU providing:
This raspberrypi/userland repository contains the source code for the ARM side libraries used on Raspberry Pi:
The VideoCore IV GPU is running the ThreadX.
VCOS (VideoCore OS abstractions) is an abstraction layer which is a wrapper around the library of functions over the ThreadX and compiled for the Linux side.
VCOS tasked with performing certain system services, for example running the display and audio outputs.
The VideoCore Host Interface (VCHI) Queue is a message passing system provided to permit communication between these SoC and VC components. The VCHI driver provides an interface to allow Linux drivers to talk to the VideoCore GPU.
The camera module is connected to the SoC via a CSI-2 interface (providing 2 Gbps of bandwidth).
Before calling any of the vc_* functions, you need to initialise vcos and vchi, and make a vchi connection, then, calling the corresponding vc_*_init function for the part of the library you want to use
For ex., userland/host_applications/linux/apps/hello_pi/hello_encode/encode.c: Linking against the EGL library (-lEGL) also requires you to link against the GLESv2 library (-lGLESv2).
graphics_get_display_size() is a Broadcom-specific function.
The raspberrypi/firmware repository has short examples for these and other APIs. They can be found in /opt/vc/src/hello_pi/ on the Raspbian 'wheezy' image.
This repository contains pre-compiled binaries of the current Raspberry Pi kernel and modules, userspace libraries, and bootloader/GPU firmware.
Build the libraries first
Provided examples are based on the usage of libs/ilclient/libilclient.a and libs/vgfont/libvgfont.a To build these examples on a different PC(cross-compile)
Before running make, set:
The purpose is to test if the build environment is setup correctly.
There is an API for the Raspberry Pi called DispmanX that allows you to add layers that are displayed over other layers.
In general, people don’t write their graphics applications using Dispmanx APIs directly. Instead, they use Dispmanx to get a window that is then used by a framework such as EGL and from there by OpenGLES.
There is a program called hello_dispmanx which is a very small example of the Dispmanx windowing system.
You just use Dispmanx to get some information about the display size and then to create a window that takes up the full screen. Anything else is just a building block on the way to using the Dispmanx window within a higher-level API such as OpenGL ES. For ex.,
raspidmx is a library created by AndrewFromMelbourne that leverages some of the capabilities of the Dispmanx API.
Several programs can be used as a starting point for anyone wanting to make use of DispmanX.
Its a great starting point for anyone wishing to learn the Dispmanx API.
To install raspidmx,
Downloadthe source code then build it
Test
pngview
Usage: pngview [-b ] [-d ] [-l] [-x ] [-y ] file.png -b – set background colour 16 bit RGBA. For ex., 0x000F is opaque black -d – Raspberry Pi display number -l – DispmanX layer number -x – offset (pixels from the left) -y – offset (pixels from the top)
Showing the memory used by the GPU is trickier, and there is a Broadcom tool /opt/vc/bin/vcdbg to show that.
For ex., "gpu_mem=256" is set in /boot/config.txt:
The RPi does not support OpenMAX AL. What it does support is OpenMAX IL, the integration layer.
The OpenMAX IL (Integration Layer) API defines a standardized media component interface to enable developers and platform providers to integrate and communicate with multimedia codecs implemented in hardware or software.
The OpenMAX DL defines an API which contains a comprehensive set of audio, video, and imaging primitives that may be used to implement OpenMAX IL components.
The current version of OpenMAX IL is 1.1.2.
The DL primitives and their full relationship to the IL are specified in the OpenMAX Development Layer API specification documents.
The OpenMAX IL API is a component-based media API that consists of two main segments: the core API and the component API.
Consider a system that requires the implementation of four multimedia processing functions denoted as F1, F2, F3, and F4. Each of these functions may be from different vendors.
The OpenMAX IL API provides a means of encapsulating these functions.
The API includes a standard protocol that enables compliant components to exchange data with one another and be used interchangeably.
The OpenMAX IL API interfaces with a higher-level entity denoted as the IL client : OpenMAX AL, or an application.
The IL client uses the OpenMAX IL core for:
Each component has zero or more input and output ports. Each port has a type, such as an audio port, a video port, or a timer port. The type of ports are important: you manipulate different types with different structures. Each port can have one or more buffers that carry data either into or out of a component.
OpenMAX IL components use buffers to carry data. A component will usually process data from an input buffer and place it on an output buffer. This processing is not visible to the API, so it allows vendors to implement components in hardware or software which may be built on top of other A/V components.
The layer of software that invokes the APIs of the core or component is called IL client.
The IL client uses the OpenMAX IL core for loading and unloading components, setting up direct communication between two OpenMAX IL components, and accessing the component’s methods.
Each component can have an arbitrary number of ports for data communication. Components with a single output port are referred to as source components. Components with a single input port are referred to as sink components.
Each OpenMAX IL component can undergo a series of state transition:
A port shall support callbacks to the IL client and, when part of an interop profile component, shall support communication with ports on other components.
Data communication is specific to a port of the component:
Output ports are always called from the IL client with OMX_FillThisBuffer
The OMX_FillThisBuffer macro will send an empty buffer to an output port of a component. The OMX_FillThisBuffer macro is invoked to pass buffers containing no data when the component is in or making a transition to the OMX_StateExecuting state or is in the OMX_StatePaused state. This call is a non-blocking call since the component will queue the buffer and return immediately. The component should return from this call within five milliseconds.
Data communications with components is always directed to a specific component port. Each port has a component-defined minimum number of buffers it shall allocate or use. A port associates a buffer header with each buffer. A buffer header references data in the buffer and provides metadata associated with the contents of the buffer.
A component will usually process data from an input buffer and place it on an output buffer.
The entity that “owns” the buffer passed into a port is called "Buffer Supplier".
The set of buffer requirements for a port includes the number of buffers required and the required size of each buffer.
Any buffer used in a port should be registered on it using the functions OMX_UseBuffer()/OMX_AllocateBuffer() during the setup phase of the component (Loaded state).
A buffer can be marked in its buffer header. The IL client can send a command to mark a buffer. The mark is internally transmitted from an input buffer to an output buffer in a chain of OpenMAX IL components. The mark enables a component to send an event to the IL client when the marked buffer is encountered.
The OpenMAX IL core is used for dynamically loading and unloading components and for facilitating component communication.
Once loaded, the API allows the IL clients to communicate directly with the component, which eliminates any overhead for high commands. Similarly, the core allows a user to establish a communication tunnel between two components. Once established, the core API is no longer used and communications flow directly between components.
Each functional block (HW, SW, or a combination) that is required to implement a particular multimedia function, such as video capture and video encode, is represented as a component.
Components are represented as object oriented classes; where base functionality is inherited from the OMX_BASE class and standard APIs are extended to suit the component.
An OpenMAX IL component is a black box that receives instruction with asynchronous commands and transmit/receive data through ports. Ports represent both the connection for components to the data stream and the buffers needed to maintain the connection. Users may send data to components through input ports or receive data through output ports. Similarly, a communication tunnel between two components can be established by connecting the output port of one component to a similarly formatted input port of another component.
Buffer status, errors, and other time-sensitive data are relayed to the application via a set of callback functions.
The individual parameters of a component can be set or retrieved through a set of associated data structures, enumerations, and interfaces.
A component is base profile compliant if it supports the standard and buffer are exchanged only with an IL client.
A component is interop profile compliant if it supports the base profile and is able to setup the tunneling with other interop components.
OpenMAX IL defines a set of standard components.
OpenMAX IL establishes two constructs for the hierarchical definition of the set of standard components:
A role is defined as the behavior of component acting according to a particular standard component definition.
The name of the standard component defining the behavior identifies the role.
For example, a given component implementation named “OMX.CompanyXYZ.MyAudioDecoder” might support the following roles:
audio_decoder.mp3, audio_decoder.aac, audio_decoder.amr
Two convenient functions are provided for the IL client to query about which roles are supported by which component implementation:
OMX_STATETYPE (3.1.1.2)
used to indicate or change the component state. Figure 3-1 illustrates the transitions among states that occur as a consequence of the IL client calling OMX_SendCommand( OMX_StateSet, state), where the new state for the component is passed as a parameter. 
OMX_ERRORTYPE
defines the standard OMX Errors
OMX_EVENTTYPE
OMX_BUFFERSUPPLIERTYPE
used to dictate port supplier preference when tunneling between two ports.
OMX_EXTRADATATYPE
used to define the possible extra data payload types.
OMX_BS32
A bounded 32 bit signed quantity. OMX_BOOL
Represents a true (OMX_TRUE) or false (OMX_FALSE) value.
OMX_PTR
Used to pass pointers between the OMX applications and the OMX Core and components.
OMX_STRING
Used to pass "C" type strings between the application and the core and component. This is a pointer to a zero terminated string.
OMX_UUIDTYPE
A very long unique identifier to uniquely identify at runtime. This identifier should be generated by a component in a way that guarantees that every instance of the identifier running on the system is unique.
OMX_ALL
Used to as a wildcard to select all entities of the same type when specifying the index, or referring to an object by an index. (I.e. use OMX_ALL to indicate all N channels). When used as a port index for a config, parameter or command OMX_ALL denotes that the config, parameter or command applies to all ports on the component.
OMX_TICKS
Used to represent time or duration in microseconds. Refer to 6.2.1 for more information.
OMX_HANDLETYPE
Defines the component handle as seen by the IL client. The component handle is used by the IL core to access all of the public methods of the component. The component handle also contains pointers to the private data area of the component. The OpenMAX IL core allocates and initializes the component handle with help from the component during the process of loading the component. After the component is successfully loaded, the IL client can safely access any of the public functions of the component via the IL core macros, although some may return an error because the state of the component is inappropriate for the access. The component handle shall not be used directly by the IL client or other IL components to access the public methods of an implementation. The IL core macros shall be used instead.
The first two fields of each OpenMAX IL defined data structure are:
The OMX_INDEXTYPE enumeration is used to select a structure when either getting or setting parameters and/or configuration data.
Each entry in this enumeration maps to an OMX specified structure. OMX_Index.h defines it:
The OpenMAX IL core implements the main interface for an IL client that wants to use OpenMAX IL components.
OMX_FreeHandle()
The IL client should call OMX_FreeHandle() only when the component is in the OMX_StateLoaded and when all the ports are not connected via any tunnels.
OpenMAX IL components are defined in the OMX_Component.h header file. The structure OMX_COMPONENTTYPE holds the data fields and function entry points for a component:
SendCommand()
The IL client calls the SendCommand component method via the OMX_SendCommand() core macro. GetParameter()
The IL client or a tunneled component calls the GetParameter component method via the OMX_GetParameter() core macro. SetParameter()
The IL client or a tunneled component calls the SetParameter component method via the OMX_SetParameter() core macro. GetConfig()
The IL client calls the GetConfig component method via the OMX_GetConfig() core macro. SetConfig()
GetExtensionIndex()
GetState()
ComponentTunnelRequest()
UseBuffer()
AllocateBuffer()
FreeBuffer()
EmptyThisBuffer()
FillThisBuffer()
ComponentDeInit()
UseEGLImage()
ComponentRoleEnum()
In the context of a single port, each data buffer has a header associated with it that contains meta-information about the buffer.
The buffer headers are shared between IL client and ports.
The IL client, the OpenMAX IL core, and the components dynamically interact in a few meaningful use cases:
This raspberrypi/userland repository contains the source code for the ARM side libraries used on Raspberry Pi. These typically are installed in /opt/vc/lib and includes source for the ARM side code to interface to: EGL, mmal, GLESv2, vcos, vchiq_arm, bcm_host, WFC, OpenVG.
It appears to be a thin wrapper(userland/interface) around its GPU API.
The implementation of OpenMAX IL Core is based on VCOS APIs.
Therefore, to build the OpenMax IL application, the include and library path of VCOS need to be set:
All IL clients must initialize OpenMAX IL by calling OMX_Init().
Before you can do anything with a component , an instance of the component must be created.
OMX_GetHandle() must be invoked to create an instance of a component,
An IL client gets a component by calling OMX_GetHandle(), passing in the name of the component. The components do not have a standard name .No standardization among component names is dictated across different vendors.
A naming convention of a component is defined. OpenMAX IL component names are zero-terminated strings with the following format:
There is a simple way to get the list of the supported components by writing a simple IL application program.
The Broadcom OpenMAX IL components used on the Raspberry Pi are documented at the VMCS-X OpenMAX IL Components page.
As the visualization component as an example, it has
The notification of various completion events occur asynchronous. So the use of callback functions is implementated as part of the OpenMax framework to facilitate the asynchronous flow between the application and components.
Ports are the means by which a component communicates, where clients pass data into a port’s input buffers and receive processed data back from a port’s output buffers.
The OMX_PARAM_PORTDEFINITIONTYPE(3.1.2.12) structure contains a set of generic fields that characterize each port of the component.
The IL client uses this structure to retrieve general information from each port.
All of the OpenMAX data structures have the following 2 fields and must be initialized before using it.
Each port has buffers. Information about these buffers is stored in the next set of fields:
The filed eDomain can be used to identify the type (3.1.3) of the port and imply the specific data structure used in OMX_PARAM_PORTDEFINITIONTYPE.format:
OpenMAX has a standard way of getting and setting parameters. A handle to a component is required for getting/setting.
For each parameter data structure, there is a corresponding parameter index.
For OMX_PARAM_PORTDEFINITIONTYPE as an ex., the index is OMX_IndexParamPortDefinition.
To get the information of a generic port:
This similar pattern can be used for other parameter data structures.
Getting and setting information about port is done by the functions
OMX_SetParameter(), 3.2.2.9
The OMX_SetParameter macro will send a parameter structure to a component. The nParamIndex parameter indicates which structure is passed to the component. OMX_GetConfig()
The OMX_GetConfig macro will get a configuration structure from a component. OMX_SetConfig()
The OMX_SetConfig macro will set a component configuration value.
Try to write a program portinfo.c to get the information about a particular port on a component and print it out.
This program takes a component name and a port index as parameters.
A component uses the OMX_PORT_PARAM_TYPE structure to identify the number and starting index of ports of a particular domain.
To get the port's information:
the video port
These parameter and configuration details are specified in the omx_video.h header. The OMX_VIDEO_PORTDEFINITIONTYPE structure can query the current or default definition of a video port or set the definition of a video port for a component. xFramerate
frames per second. This value is represented in Q16 format.
eCompressionFormat
OMX_IMAGE_CodingJPEG, OMX_IMAGE_CodingJPEG2K, ... When OMX_IMAGE_CodingUnused is specified, the eColorFormat field is valid.
eColorFormat
OMX_COLOR_Formatxxx defined in "Table 4-31: Uncompressed Data Formats". For ex., OMX_COLOR_FormatYUV420PackedPlanar the image port
other ports
You can query the port information in more detail:
The OpenMAX IL includes a callback mechanism that allows a component to communicate with the IL client
Whenever a component changes state, it will invoke the event callback handler established for that component.
To accomplish a callback, the OpenMAX IL has 3 callback functions defined: 1 generic event handler(EventHandler) and 2 callbacks related to the dataflow (EmptyBufferDone and FillBufferDone).
The IL client is responsible for filling in an OMX_CALLBACKTYPE structure with its callback entry points and passing the structure to the OpenMAX IL core at initialization (init) time.
The IL Client library is designed to make it easier to use OpenMAX on the RPi.
Many of the OpenMAX IL client function calls in the Broadcom examples ( userland/host_applications/linux/apps/hello_pi ) are embedded in a Broadcom convenience functions: userland/host_applications/linux/apps/hello_pi/libs/ilclient/ilclient.c ( installed as /opt/vc/src/hello_pi/libs/ilclient/ilclient.c ).
The Il Client functions are documented in the file hello_pi/libs/ilclient/ilclient.h and defined in hello_pi/libs/ilclient/ilclient.c
Use the IL client library to to the initialzation of a component:
COMPONENT_T
The COMPONENT_T structure represents an IL component, together with the necessary extra information required by the IL Client API. This structure stores the handle to the OMX component, as well as the event list containing all events sent by this component. The component state structure also holds a pair of buffer queues, for input and output buffers returned to the client by the FillBufferDone and EmptyBufferDone callbacks. As some operations result in error callbacks that can be ignored, an error mask is maintained to allow some errors to be ignored. A pointer to the client state structure is also added. ILCLIENT_T *ilclient_init()
An IL Client structure is created by the ilclient_init() method. This structure is used when creating components and not needed in other API functions as a pointer to this structure is maintained in the COMPONENT_T structure. ilclient_create_component()
Components are created using the ilclient_create_component() function. It creates a handle to a component and does something according to the flag.
Use OpenMAX IL API to do the detail configuration on input/output ports:
Use the IL client library to bring the component in the idle state and enable its ports and the buffering :
ilclient_enable_port_buffers( COMPONENT_T *comp, int portIndex, ILCLIENT_MALLOC_T ilclient_malloc, ILCLIENT_FREE_T ilclient_free, void *userdata)
The ilclient_enable_port_buffers() function enables a port in base profile mode on a given component and allocates buffers. The port is not tunneled, so requires buffers to be allocated. userdata is the first argument to calls to ilclient_malloc and ilclient_free, if these arguments are not NULL.
Start to work:
Each component has ports, and each port has a number of buffers.
When a component is created, it has an array of pointers to buffer headers but doesn’t actually have any memory allocated either for the headers or for the buffers. There are 2 eays to allocate buffers:
The IL Client Library
The ilclient_enable_port_buffers() function enables a port in base profile mode on a given component and allocates buffers for the port ( by OMX_UseBuffer() ).
OpenMAX IL defines two means of data communication:
The IL client is entirely responsible for moving data buffers among components if data tunneling is not used.
An IL client that has a data buffer to deliver to a component input port shall issue an OMX_EmptyThisBuffer() call.
For the component output port, the IL client shall initially provide one or more empty buffers into which the component can write output data; the OMX_FillThisBuffer() call accomplishes this task. As soon as one buffer is available from the component output port, the component shall send an FillBufferDone() callback. At this point, your application will take the processed data and do something with it.
To avoid moving data buffers back and forth among the IL client and OpenMAX IL components, data tunnels can be set up so that the output buffer of one component is passed directly to the input port of the next component in the chain.
The tunneling setup and initialization based on the following steps:
In data tunneling, OpenMAX IL components directly pass data buffers among themselves without returning them to the IL client.
The following 2 processes should be done concurrently by the client:
The pseudo code:
The component needs to know when there is no input data so that it sends a signal to the client that there are no more output buffers.
The client signals this by setting the flag OMX_BUFFERFLAG_EOS in the flags field of the buffer header.
Typical code to read data from a file into input buffers, setting EOS on completion,
You may not know some information before you can configure ports correctly. These information can be got after the component processed it. For ex., the size of the buffer for the output port.
When the OpenMAX IL component detects the setting needed to be changed, it shall notify the IL client by generating the OMX_PortSettingsChanged event.
As soon as the IL client receives this callback, it shall disable the related port then read the new port settings with OMX_GetConfig().
After the setting is corrected and necessary resource is allocated according to the setting, related ports shall also be set up with OMX_SetConfig().
The IL Client library has two relevant functions to receive OMX_PortSettingsChanged events:
OMX_IMAGE_PORTDEFINITIONTYPE (4.4.3) is the data structure that is used to define an image path.
The OMX_IMAGE_PORTDEFINITIONTYPE structure is included as part of the OMX_PARAM_PORTDEFINITIONTYPE structure
The color format will be set to match that emitted by the input image format.
image_encode takes raw images in various formats ranging form 16/24/32-bit RGB with Alpha formats to variants of YUV and CrCbY. The output can be any of the supported compressed image formats
You basically only have to configure the input and output port and you are ready to go and throw data on it.
The basic steps:
YUV refers to a family of color spaces, all of which encode brightness information separately from color information.
In fact, YUV almost always refers to one particular color space named Y'CbCr. However, YUV is currently often used as a general term for any color space that works along the same principles as Y'CbCr.
The Y' component, also called luma, represents the brightness value of the color. luma is different from luminance, which is designated Y.
Luminance is derived from linear RGB values, whereas luma is derived from non-linear (gamma-corrected) RGB values.
Luma is derived from an RGB color:
HDTV(ITU-R BT.709)
Black-and-white televisions ignore the chroma and display the combined signal as a grayscale image.
主要的抽樣（subsample）格式有YCbCr 4:2:0、YCbCr 4:2:2、YCbCr 4:1:1和YCbCr 4:4:4。
They comprise an [n , m] Y plane followed by [(n/2), (m/2)] U and V planes.
Figure 4-4 depicts one possible set of components as well as the tunneling of ports for these components to implement a H.263 video encoding scheme. This use case encodes raw video into H.263 format and writes it to a file while previewing the captured video on a display.
This is an interface to a platform-specific camera hardware, and produces raw YUV images.
OMX_IndexParamISPTunerName
ISP tuner name. Allow the client to query / set the name of the camera tuner module to be used by Camplus. If tuner_name is a null string, the default tuner for the camera module is loaded. It is recommended that this parameter is set before OMX_IndexParamCameraDeviceNumber to avoid reloading the CDI and default tuner. port#73
OMX_IndexParamOtherPortFormat -> OMX_OTHER_PARAM_PORTFORMATTYPE port#70
port#71
port#72
A conformant image encode component, which takes raw pixels on its input port, and encodes the image into various compressed formats on the output port.
A conformant image writer component, which accepts a compressed image on its input port and writes a file to the filesystem.
Why to use MMAL: Good luck with OMX. I've been working on and off with it for 5 years, and still have trouble. It really isn't very good. The spec document is awful to learn from (It's a spec not a manual) and the code is a nightmare to debug. Fair point on MMAL not being docuemnted per se - but note it does have full Doxygen comments in the ARM side source code, which is published on the web somewhere.
Run the following command to install packages needed to build and run Cobalt on your Linux/Pi
Install the latest version of the standard C++ header files (libstdc++)
Install bison
clone the Cobalt source code repository
Modify your path to include the version of Clang that will be downloaded later
generate build files
To generate for Raspberry Pi
build the code
To build for Raspberry Pi
launch the built Cobalt client
Setup the build host for Pi
Setup the path for the Pi's cross-compiling toolchain: Sync remote Pi's system files to your host
Build
Install your Cobalt binary (and content) on the remote Pi
run Colbat
Using SSH will make it easy for you to quit or restart Cobalt.
Chapter 1 Introduction to the Raspberry Pi
Introduction
| Product | SoC | Speed | RAM | USB Ports | Ethernet | Wireless/Bluetooth | GPU |
|---|---|---|---|---|---|---|---|
| Raspberry Pi Model A+ | BCM2835 | 700MHz | 512MB | 1 | No | No | |
| Raspberry Pi Model B+ | BCM2835 | 700MHz | 512MB | 4 | Yes | No | |
| Raspberry Pi 2 Model B | BCM2836/7 | 900MHz | 1GB | 4 | Yes | No | |
| Raspberry Pi 3 Model B | BCM2837 | 1200MHz | 1GB | 4 | Yes | Yes | |
| Raspberry Pi Zero | BCM2835 | 1000MHz | 512MB | 1 | No | No | |
| Raspberry Pi Zero W | BCM2835 | 1000MHz | 512MB | 1 | No | Yes | |
| Raspberry Pi Zero WH | BCM2835 | 1000MHz | 512MB | 1 | No | Yes | |
| Raspberry Pi 3 Model B+ | BCM2837B0, Cortex-A53 (ARMv8) | 1400MHz | 1GB | 4 | Yes | 2.4GHz and 5GHz IEEE 802.11.b/g/n/ac wireless LAN, Bluetooth 4.2, BLE | Broadcom VideoCore IV @ 250 MHz: H.264, MPEG-4 decode (1080p30); H.264 encode (1080p30); OpenGL ES 1.1, 2.0 graphics |
| Raspberry Pi 4 B | BCM2711, Quad core Cortex-A72 (ARM v8) | 1.5 GHz | 1 GB , 2 GB, 4 GB | 2x USB3.0 + 2x USB2.0 | Gigabit | 2.4GHz and 5GHz 802.11 b/g/n/ac, Bluetooth 5.0 BLE | Broadcom VideoCore VI @ 500 MHz: H.265 (4kp60 decode), H264 (1080p60 decode, 1080p30 encode), OpenGL ES 3.0 graphics |
The GPU provides OpenGL ES 2.0, hardware-accelerated OpenVG, and 1080p30 H.264 high-profile encode and decode. While the CPU has changed in the different models, the GPU has remained the same.
RASPBERRY PI CAMERA
| RASPBERRY PI CAMERA V1.3 | RASPBERRY PI CAMERA V2.1 | |
|---|---|---|
| Image Sensor： | OmniVision OV5647 | Sony IMX219 |
| Resolution： | 2592 × 1944 pixels (5 megapixel) | 3280 × 2464 (8 megapixel) |
| Sensor Image Area: | 3.76 × 2.74 mm | 3.69 × 2.81 mm |
| Pixel Size: | 1.4 µm × 1.4 µm | 1.12 µm × 1.12 µm |
| Video： | 1920 × 1080 (1080p)30p | 1920 × 1080 (1080p)30p |
Resources
• Raspberry Pi home page• RPi OpenMAX forum
• RPi Graphics forum
• RPi OpenGL ES forum
• Raspberry Pi VideoCore APIs
• Linux Gizmos reports on many SoCs including the RPi
• Tutorial: VLC with hardware acceleration on Raspberry Pi
• How to draw 2D images using OpenGL, in SDL
• Hardware Accelerated Qt Multimedia Backend for Raspberry Pi and OpenGL Shaders on Video
• Decoding and Rendering Compressed Images with OpenMAX on Raspberry Pi
Chapter 2 Khronos Group
The Khronos Group is a not-for-profit industry consortium creating open standards including the following:
- OpenGL
- OpenGL ES OpenGL ES is a royalty-free, cross-platform API for full-function 2D and 3D graphics on embedded systems.
- OpenMAX OpenMAX is a royalty-free, cross-platform API that provides comprehensive streaming media codec and application portability by enabling accelerated multimedia components to be developed, integrated, and programmed across multiple operating systems and silicon platforms. The OpenMAX API will be shipped with processors to enable library and codec implementers to rapidly and effectively make use of the full acceleration potential of new silicon, regardless of the underlying hardware architecture.
- EGL EGL is an interface between Khronos-rendering APIs such as OpenGL ES or OpenVG and the underlying native platform window system.
- OpenVG
- OpenCL
Resources
• Khronos Group
Chapter 3 Compiling Programs for the Raspberry Pi
Generic Information
- arch Print machine hardware name.
pi@raspberrypi:~ $ arch
armv7l
Raspbian GNU/Linux 9 \n \l
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
GCC
Language Standards Supported by GCC
For each language (C/C++) compiled by GCC, there is a standard GCC tries to follow.
To select a standard,
- The original ANSI C standard (X3.159-1989) published in 1990 Use one of the options: -ansi, -std=c90 or -std=iso9899:1990
- An amendment to the 1990 standard was published in 1995 Use the option -std=iso9899:199409
- A new edition of the ISO C standard was published in 1999 as ISO/IEC 9899:1999 Use -std=c99 or -std=iso9899:1999
- A fourth version of the C standard, known as C11, was published in 2011 as ISO/IEC 9899:2011. Use the option -std=c11 or -std=iso9899:2011
- -std=gnu90 for C90 with GNU extensions
- -std=gnu99 for C99 with GNU extensions
- -std=gnu11 for C11 with GNU extensions.
GCC Command Options
Option index.- Overall Options The “overall options” allow you to stop this process at an intermediate stage. For example, the -c option says not to run the linker. Then the output consists of object files output by the assembler.
- --help Print (on the standard output) a description of the command-line options understood by gcc.
- --target-help Print (on the standard output) a description of target-specific command-line options for each tool(assembler, linker).
- C Language Options
- C++ Language Options
- Objective-C and Objective-C++ Language Options
- Diagnostic Message Formatting Options
- Warning Options
- C and Objective-C-only Warning Options
- Debugging Options
- Optimization Options
- Program Instrumentation Options
- Preprocessor Options
- Assembler Options
- Linker Options
- Directory Options
- Code Generation Options
- Developer Options
- -Q Makes the compiler print out each function name as it is compiled, and print some statistics about each pass when it finishes.
- Machine-Dependent Options
- -march=name Specify the name of the target architecture.
- -mcpu=name Specify the name of the target processor.
- Adapteva Epiphany Options
- ARC Options
- ARM Options
- -mabi=name Generate code for the specified ABI(Application Binary Interface). The ABI for the ARM Architecture is a collection of standards. The specifications to which an executable must conform in order to execute in a specific execution environment. Standardizing the inter-operation of binary files requires standardizing certain aspects of code generation itself, so this base standard is aimed principally at the authors and vendors of C and C++ compilers, linkers, and run-time libraries.
- -mfloat-abi=name Specifies which floating-point ABI to use. Permissible values are: ‘soft’, ‘softfp’ and ‘hard’. 'hard' is the default on Pi.
- -mtune=name This option specifies the name of the target ARM processor for which GCC should tune the performance of the code.
- -mfpu=name This specifies what floating-point hardware (or hardware emulation) is available on the target.
- -mstructure-size-boundary=n The sizes of all structures and unions are rounded up to a multiple of the number of bits set by this option.
- -mno-sched-prolog Prevent the reordering of instructions in the function prologue, or the merging of those instruction with the instructions in the function’s body. This means that all functions start with a recognizable set of instructions (or in fact one of a choice from a small set of different function prologues), and this information can be used to locate the start of functions inside an executable piece of code. The default is -msched-prolog.
- AVR Options
- GNU/Linux Options
- MIPS Options
- x86 Options
Chapter 4 Raspberry Pi VideoCore APIs
The Raspberry Pi contains a Broadcom VideoCore IV GPU providing:
- OpenGL ES 1.1
- OpenGL ES 2.0
- hardware-accelerated OpenVG 1.1
- Open EGL
- OpenMAX
- 1080p30 H.264 high-profile decode
- kernel and modules
- userspace libraries
- bootloader/GPU firmware
This raspberrypi/userland repository contains the source code for the ARM side libraries used on Raspberry Pi:
interface
├── khronos
│ ├── CMakeLists.txt
│ ├── common
│ ├── egl
│ ├── ext
│ ├── glxx
│ ├── include
│ ├── vg
│ └── wf
├── mmal
│ ├── client
│ ├── CMakeLists.txt
│ ├── components
│ ├── core
│ ├── mmal_buffer.h
│ ├── mmal_clock.h
│ ├── mmal_common.h
│ ├── mmal_component.h
│ ├── mmal_encodings.h
│ ├── mmal_events.h
│ ├── mmal_format.h
│ ├── mmal.h
│ ├── mmal_logging.h
│ ├── mmal_parameters_audio.h
│ ├── mmal_parameters_camera.h
│ ├── mmal_parameters_clock.h
│ ├── mmal_parameters_common.h
│ ├── mmal_parameters.h
│ ├── mmal_parameters_video.h
│ ├── mmal_pool.h
│ ├── mmal_port.h
│ ├── mmal_queue.h
│ ├── mmal_types.h
│ ├── openmaxil
│ ├── test
│ ├── util
│ └── vc
├── peer
│ └── vc_vchi_dispmanx_common.h
├── vchi
│ ├── common
│ ├── connections
│ ├── message_drivers
│ ├── vchi_cfg.h
│ ├── vchi_cfg_internal.h
│ ├── vchi_common.h
│ ├── vchi.h
│ └── vchi_mh.h
├── vchiq_arm
│ ├── CMakeLists.txt
│ ├── vchiq_cfg.h
│ ├── vchiq.h
│ ├── vchiq_if.h
│ ├── vchiq_ioctl.h
│ ├── vchiq_lib.c
│ ├── vchiq_test.c
│ ├── vchiq_test.h
│ ├── vchiq_test_if.h
│ ├── vchiq_util.c
│ └── vchiq_util.h
├── vcos
│ ├── CMakeLists.txt
│ ├── generic
│ ├── glibc
│ ├── pthreads
│ ├── user_nodefs.h
│ ├── vcos_assert.h
│ ├── vcos_atomic_flags.h
│ ├── vcos_attr.h
│ ├── vcos_blockpool.h
│ ├── vcos_build_info.h
│ ├── vcos_cfg.h
│ ├── vcos_cmd.h
│ ├── vcos_ctype.h
│ ├── vcos_dlfcn.h
│ ├── vcos_event_flags.h
│ ├── vcos_event.h
│ ├── vcos.h
│ ├── vcos_init.h
│ ├── vcos_inttypes.h
│ ├── vcos_isr.h
│ ├── vcos_legacy_isr.h
│ ├── vcos_logging_control.h
│ ├── vcos_logging.h
│ ├── vcos_lowlevel_thread.h
│ ├── vcos_mem.h
│ ├── vcos_mempool.h
│ ├── vcos_msgqueue.h
│ ├── vcos_mutex.h
│ ├── vcos_named_semaphore.h
│ ├── vcos_once.h
│ ├── vcos_queue.h
│ ├── vcos_quickslow_mutex.h
│ ├── vcos_reentrant_mutex.h
│ ├── vcos_semaphore.h
│ ├── vcos_stdbool.h
│ ├── vcos_stdint.h
│ ├── vcos_string.h
│ ├── vcos_thread_attr.h
│ ├── vcos_thread.h
│ ├── vcos_timer.h
│ ├── vcos_tls.h
│ └── vcos_types.h
├── vctypes
│ ├── vc_display_types.h
│ ├── vc_image_structs.h
│ └── vc_image_types.h
└── vmcs_host
├── CMakeLists.txt
├── khronos
├── linux
├── vc_cec.h
├── vc_cecservice_defs.h
├── vc_cecservice.h
├── vc_cma.h
├── vc_dispmanx.h
├── vc_dispmanx_types.h
├── vc_dispservice_defs.h
├── vc_dispservice_x_defs.h
├── vc_fileservice_defs.h
├── vcfilesys_defs.h
├── vcfilesys.h
├── vc_gencmd_defs.h
├── vcgencmd.h
├── vc_hdmi.h
├── vc_hdmi_property.h
├── vchost.h
├── vchost_platform_config.h
├── vcilcs.c
├── vcilcs_common.c
├── vcilcs_common.h
├── vc_ilcs_defs.h
├── vcilcs.h
├── vcilcs_in.c
├── vcilcs_out.c
├── vc_imageconv_defs.h
├── vc_sdtv.h
├── vc_service_common.c
├── vc_service_common.h
├── vc_tvservice_defs.h
├── vc_tvservice.h
├── vc_vchi_audioserv_defs.h
├── vc_vchi_bufman_defs.h
├── vc_vchi_bufman.h
├── vc_vchi_cecservice.c
├── vc_vchi_dispmanx.c
├── vc_vchi_dispmanx.h
├── vc_vchi_fileservice_defs.h
├── vc_vchi_filesys.c
├── vc_vchi_filesys.h
├── vc_vchi_gencmd.c
├── vc_vchi_gencmd.h
├── vc_vchi_gpuserv.c
├── vc_vchi_gpuserv.h
└── vc_vchi_tvservice.c
The VideoCore IV GPU is running the ThreadX.
VCOS (VideoCore OS abstractions) is an abstraction layer which is a wrapper around the library of functions over the ThreadX and compiled for the Linux side.
VCOS tasked with performing certain system services, for example running the display and audio outputs.
The VideoCore Host Interface (VCHI) Queue is a message passing system provided to permit communication between these SoC and VC components. The VCHI driver provides an interface to allow Linux drivers to talk to the VideoCore GPU.
The camera module is connected to the SoC via a CSI-2 interface (providing 2 Gbps of bandwidth).
API Guides
raspberrypi/userland
Source code for ARM side libraries for interfacing to Raspberry Pi GPU.- The Raspberry Pi requires that the bcm_host_init() function is called first before any GPU calls can be made. raspberrypi/userland/host_applications/linux/libs/bcm_host/bcm_host.c :
void bcm_host_init(void)
{
VCHIQ_INSTANCE_T vchiq_instance;
static int initted;
int success = -1;
char response[ 128 ];
if (initted)
return;
initted = 1;
vcos_init();
if (vchiq_initialise(&vchiq_instance) != VCHIQ_SUCCESS)
{
printf("* failed to open vchiq instance\n");
exit(-1);
}
vcos_log("vchi_initialise");
success = vchi_initialise( &global_initialise_instance);
vcos_assert(success == 0);
vchiq_instance = (VCHIQ_INSTANCE_T)global_initialise_instance;
global_connection = vchi_create_connection(single_get_func_table(),
vchi_mphi_message_driver_func_table());
vcos_log("vchi_connect");
vchi_connect(&global_connection, 1, global_initialise_instance);
vc_vchi_gencmd_init (global_initialise_instance, &global_connection, 1);
vc_vchi_dispmanx_init (global_initialise_instance, &global_connection, 1);
vc_vchi_tv_init (global_initialise_instance, &global_connection, 1);
vc_vchi_cec_init (global_initialise_instance, &global_connection, 1);
//vc_vchi_bufman_init (global_initialise_instance, &global_connection, 1);
if ( success == 0 )
{
success = vc_gencmd( response, sizeof(response), "set_vll_dir /sd/vlls" );
vcos_assert( success == 0 );
}
}
int
main(int argc, char **argv)
{
if (argc < 2) {
printf("Usage: %s \n", argv[0]);
exit(1);
}
bcm_host_init();
return video_encode_test(argv[1]);
}
raspberrypi/firmware
This repository contains pre-compiled binaries of the current Raspberry Pi kernel and modules, userspace libraries, and bootloader/GPU firmware.
- Download the raspberrypi/firmware Download the source code and unzip it.
wget https://github.com/raspberrypi/firmware/archive/master.zip
unzip master.zip
cd firmware-master/opt/vc/src/hello_pi
make -C libs/ilclient
make -C libs/vgfont
fatal error: ft2build.h: No such file or directory
pkg-config --cflags freetype2
sudo apt-get install libfreetype6 libfreetype6-dev
cd hello_world
make
./rebuild.sh
SDKSTAGE=path/to/firmware-directory
CC=path/to/cross-compiler
API examples
hello_world
"hello_word" is a simple program which just prints out a string "hello world".The purpose is to test if the build environment is setup correctly.
hello_triangle
This sample shows how to get a OpenGL ES context on the Raspberry Pi.hello_triangle2
hello_video
This sample decodes H.264 video using the OpenMAX APIs and the ilclient library and is distributed with a short scene of Big Buck Bunnyhello_audio
This sample plays a sine wave for ten seconds using the OpenMAX APIs.hello_font
hello_dispmanx
hello_tiger
hello_encode
This sample is a GPU-assisted H.264 encoder using the OpenMAX APIs, and currently works with raw frames. It outputs a raw H.264 file.hello_jpeg
hello_videocube
hello_teapot
hello_fft
hello_mmal_encode
DispmanX
There is an API for the Raspberry Pi called DispmanX that allows you to add layers that are displayed over other layers.
In general, people don’t write their graphics applications using Dispmanx APIs directly. Instead, they use Dispmanx to get a window that is then used by a framework such as EGL and from there by OpenGLES.
There is a program called hello_dispmanx which is a very small example of the Dispmanx windowing system.
// A simple demo using dispmanx to display an overlay
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <assert.h>
#include <unistd.h>
#include <sys/time.h>
#include "bcm_host.h"
#define WIDTH 200
#define HEIGHT 200
#ifndef ALIGN_UP
#define ALIGN_UP(x,y) ((x + (y)-1) & ~((y)-1))
#endif
typedef struct
{
DISPMANX_ELEMENT_HANDLE_T element;
DISPMANX_DISPLAY_HANDLE_T display;
DISPMANX_UPDATE_HANDLE_T update;
DISPMANX_MODEINFO_T info;
void *image;
DISPMANX_RESOURCE_HANDLE_T resource;
uint32_t vc_image_ptr;
} RECT_VARS_T;
static RECT_VARS_T gRectVars;
static void FillRect( VC_IMAGE_TYPE_T type, void *image, int pitch, int aligned_height, int x, int y, int w, int h, int val )
{
int row;
int col;
uint16_t *line = (uint16_t *)image + y * (pitch>>1) + x;
for ( row = 0; row < h; row++ )
{
for ( col = 0; col < w; col++ )
{
line[col] = val;
}
line += (pitch>>1);
}
}
int main(void)
{
RECT_VARS_T *vars;
uint32_t screen = 0;
int ret;
VC_RECT_T src_rect;
VC_RECT_T dst_rect;
VC_IMAGE_TYPE_T type = VC_IMAGE_RGB565;
int width = WIDTH, height = HEIGHT;
int pitch = ALIGN_UP(width*2, 32); // VC_IMAGE_RGB565 means the color space is 2 bytes(16-bits)
int aligned_height = ALIGN_UP(height, 16);
VC_DISPMANX_ALPHA_T alpha = { DISPMANX_FLAGS_ALPHA_FROM_SOURCE | DISPMANX_FLAGS_ALPHA_FIXED_ALL_PIXELS,
120, /*alpha 0->255*/
0 };
vars = &gRectVars;
bcm_host_init();
printf("Open display[%i]...\n", screen );
vars->display = vc_dispmanx_display_open( screen );
ret = vc_dispmanx_display_get_info( vars->display, &vars->info);
assert(ret == 0);
printf( "Display is %d x %d\n", vars->info.width, vars->info.height );
vars->image = calloc( 1, pitch * height );
assert(vars->image);
FillRect( type, vars->image, pitch, aligned_height, 0, 0, width, height, 0xFFFF );
FillRect( type, vars->image, pitch, aligned_height, 0, 0, width, height, 0xF800 );
FillRect( type, vars->image, pitch, aligned_height, 20, 20, width - 40, height - 40, 0x07E0 );
FillRect( type, vars->image, pitch, aligned_height, 40, 40, width - 80, height - 80, 0x001F );
vars->resource = vc_dispmanx_resource_create( type,
width,
height,
&vars->vc_image_ptr );
assert( vars->resource );
vc_dispmanx_rect_set( &dst_rect, 0, 0, width, height);
ret = vc_dispmanx_resource_write_data( vars->resource,
type,
pitch,
vars->image,
&dst_rect );
assert( ret == 0 );
vars->update = vc_dispmanx_update_start( 10 );
assert( vars->update );
vc_dispmanx_rect_set( &src_rect, 0, 0, width << 16, height << 16 );
vc_dispmanx_rect_set( &dst_rect, ( vars->info.width - width ) / 2,
( vars->info.height - height ) / 2,
width,
height );
vars->element = vc_dispmanx_element_add( vars->update,
vars->display,
2000, // layer
&dst_rect,
vars->resource,
&src_rect,
DISPMANX_PROTECTION_NONE,
&alpha,
NULL, // clamp
VC_IMAGE_ROT0 );
ret = vc_dispmanx_update_submit_sync( vars->update );
assert( ret == 0 );
printf( "Sleeping for 10 seconds...\n" );
sleep( 10 );
vars->update = vc_dispmanx_update_start( 10 );
assert( vars->update );
ret = vc_dispmanx_element_remove( vars->update, vars->element );
assert( ret == 0 );
ret = vc_dispmanx_update_submit_sync( vars->update );
assert( ret == 0 );
ret = vc_dispmanx_resource_delete( vars->resource );
assert( ret == 0 );
ret = vc_dispmanx_display_close( vars->display );
assert( ret == 0 );
return 0;
}
You just use Dispmanx to get some information about the display size and then to create a window that takes up the full screen. Anything else is just a building block on the way to using the Dispmanx window within a higher-level API such as OpenGL ES. For ex.,
static EGL_DISPMANX_WINDOW_T nativewindow;
nativewindow.element = vars->element;
nativewindow.width = width;
nativewindow.height = height;
vc_dispmanx_update_submit_sync( vars->display );
raspidmx
raspidmx is a library created by AndrewFromMelbourne that leverages some of the capabilities of the Dispmanx API.
Several programs can be used as a starting point for anyone wanting to make use of DispmanX.
Its a great starting point for anyone wishing to learn the Dispmanx API.
To install raspidmx,
- Install libpng
sudo apt-get install libpng12-dev
cd /home/pi/raspidmx
unzip ~/Downloads/raspidmx-master.zip
cd raspidmx-master
sudo make
export LD_LIBRART_PATH=$PWD/lib
- test_pattern The tested pattern is the same four colour square displayed when the Raspberry Pi boots.
test_pattern/test_pattern
pngview/pngview -b 0 -l 3 /home/pi/Downloads/image1.png
GPU memory
Showing the memory used by the GPU is trickier, and there is a Broadcom tool /opt/vc/bin/vcdbg to show that.
For ex., "gpu_mem=256" is set in /boot/config.txt:
$ sudo vcdbg reloc
Relocatable heap version 4 found at 0x30000000
total space allocated is 236M, with 236M relocatable, 0 legacy and 0 offline
0 legacy blocks of size 2359296
free list at 0x3e401d20
228M free memory in 1 free block(s)
largest free block is 228M bytes
0x30000000: free 228M
[ 4] 0x3e401d40: used 576 (refcount 1 lock count 0, size 512, align 4, data 0x3e401d60, d0rual) 'ILCS VC buffer pool'
[ 3] 0x3e401f80: used 8.0M (refcount 1 lock count 8, size 8355840, align 4096, data 0x3e402000, d1rual) 'ARM FB'
[ 2] 0x3ebfafa0: used 16K (refcount 1 lock count 0, size 16384, align 32, data 0x3ebfafc0, d0ruAl) 'audioplus_tmp_buf'
[ 1] 0x3ebfefe0: used 4.0K (refcount 1 lock count 0, size 0, align 4096, data 0x3ebff000, d1rual) 'camera fast alloc arena'
small allocs not requested
Chapter 5 EGL on the Raspberry Pi
Chapter 6: OpenGL ES on the Raspberry Pi
Chapter 7: OpenMAX on the Raspberry Pi Concepts
OpenMAX Overview
- OpenMAX AL (Application Layer) OpenMAX AL provides a standardized interface between an application and multimedia middleware, where multimedia middleware provides the services needed to perform expected API functionality. OpenMAX AL provides application portability with regards to the multimedia interface.
- OpenMAX IL (Integration Layer) OpenMAX IL serves as a low-level interface for audio, video, and imaging codecs used in embedded and/or mobile devices. It gives applications and media frameworks the ability to interface with multimedia codecs and supporting components (i.e., sources and sinks) in a unified manner. The codecs themselves may be any combination of hardware or software and are completely transparent to the user. Without a standardized interface of this nature, codec vendors must write to proprietary or closed interfaces to integrate into mobile devices. The principal goal of the IL is to give codecs a degree of system abstraction using a specialized arsenal of features, honed to combat the problem of portability among many vastly different media systems.
- OpenMAX DL (Development Layer) OpenMAX DL defines an API which contains a comprehensive set of audio, video and imaging functions that can be implemented and optimized on new processors by silicon vendors and then used by codec vendors to code a wide range of codec functionality. It includes audio signal processing functions such as FFTs and filters, imaging processing primitives such as color space conversion and video processing primitives to enable the optimized implementation of codecs such as MPEG-4, H.264, MP3, AAC and JPEG. OpenMAX supports acceleration concurrency via both iDL, which uses OpenMAX IL constructs, and aDL which adds asynchronous interfaces to the OpenMAX DL API.
The RPi does not support OpenMAX AL. What it does support is OpenMAX IL, the integration layer.
Resources
- Raspberry Pi Video on a Teapot Rendering OpenMAX onto an OpenGL surface
- OpenMax rendering onto OpenGL texture
- firmware/documentation/ilcomponents/
- OpenMAX IL: The Standard for Media Library Portability
- Decoding and Rendering to Texture H264 with OpenMAX on Raspberry Pi
OpenMAX IL Overview
The OpenMAX IL (Integration Layer) API defines a standardized media component interface to enable developers and platform providers to integrate and communicate with multimedia codecs implemented in hardware or software.
The OpenMAX DL defines an API which contains a comprehensive set of audio, video, and imaging primitives that may be used to implement OpenMAX IL components.
The current version of OpenMAX IL is 1.1.2.
The DL primitives and their full relationship to the IL are specified in the OpenMAX Development Layer API specification documents.
The OpenMAX IL API is a component-based media API that consists of two main segments: the core API and the component API.
Consider a system that requires the implementation of four multimedia processing functions denoted as F1, F2, F3, and F4. Each of these functions may be from different vendors.
The OpenMAX IL API provides a means of encapsulating these functions.
The API includes a standard protocol that enables compliant components to exchange data with one another and be used interchangeably.
The OpenMAX IL API interfaces with a higher-level entity denoted as the IL client : OpenMAX AL, or an application.
The IL client uses the OpenMAX IL core for:
- loading and unloading components
- setting up direct communication between two OpenMAX IL components
- accessing the component’s methods
Each component has zero or more input and output ports. Each port has a type, such as an audio port, a video port, or a timer port. The type of ports are important: you manipulate different types with different structures. Each port can have one or more buffers that carry data either into or out of a component.
OpenMAX IL components use buffers to carry data. A component will usually process data from an input buffer and place it on an output buffer. This processing is not visible to the API, so it allows vendors to implement components in hardware or software which may be built on top of other A/V components.
OpenMax IL Architectural Overview
The layer of software that invokes the APIs of the core or component is called IL client.
The IL client uses the OpenMAX IL core for loading and unloading components, setting up direct communication between two OpenMAX IL components, and accessing the component’s methods.
Each component can have an arbitrary number of ports for data communication. Components with a single output port are referred to as source components. Components with a single input port are referred to as sink components.
Each OpenMAX IL component can undergo a series of state transition:
- Every component is first considered to be unloaded. The component shall be loaded through a call to the OpenMAX IL core. All other state transitions may then be achieved by communicating directly with the component.
- A component can enter an invalid state when a state transition is made with invalid data.
- The IL client shall stop, de-initialize, unload, and reload the component when the IL client detects an invalid state.
- The only way to exit the invalid state is to unload and reload the component.
A port shall support callbacks to the IL client and, when part of an interop profile component, shall support communication with ports on other components.
Data communication is specific to a port of the component:
- Input ports are always called from the IL client with OMX_EmptyThisBuffer The OMX_EmptyThisBuffer macro will send a filled buffer to an input port of a component.
- When the buffer contains data, the value of the nFilledLen field of the buffer header will not be zero.
- If the buffer contains no data, the value of nFilledLen is 0x0.
#define OMX_EmptyThisBuffer (
hComponent,
pBuffer )
((OMX_COMPONENTTYPE*)hComponent)->EmptyThisBuffer(
hComponent,
pBuffer)
#define OMX_FillThisBuffer (
hComponent,
pBuffer )
((OMX_COMPONENTTYPE*)hComponent)->FillThisBuffer(
hComponent,
pBuffer)
- When a port is non-tunneled buffers sent to OMX_FillThisBuffer() return to the IL client with the FillBufferDone callback once they have been filled.
- When a port is tunneled buffers sent to OMX_FillThisBuffer() are sent to the tunneled port once they are filled.
Data communications with components is always directed to a specific component port. Each port has a component-defined minimum number of buffers it shall allocate or use. A port associates a buffer header with each buffer. A buffer header references data in the buffer and provides metadata associated with the contents of the buffer.
A component will usually process data from an input buffer and place it on an output buffer.
The entity that “owns” the buffer passed into a port is called "Buffer Supplier".
The set of buffer requirements for a port includes the number of buffers required and the required size of each buffer.
Any buffer used in a port should be registered on it using the functions OMX_UseBuffer()/OMX_AllocateBuffer() during the setup phase of the component (Loaded state).
A buffer can be marked in its buffer header. The IL client can send a command to mark a buffer. The mark is internally transmitted from an input buffer to an output buffer in a chain of OpenMAX IL components. The mark enables a component to send an event to the IL client when the marked buffer is encountered.
OpenMax IL Core
The OpenMAX IL core is used for dynamically loading and unloading components and for facilitating component communication.
Once loaded, the API allows the IL clients to communicate directly with the component, which eliminates any overhead for high commands. Similarly, the core allows a user to establish a communication tunnel between two components. Once established, the core API is no longer used and communications flow directly between components.
OpenMAX IL Core functions
- OMX_Init(), OMX_Deinit() Initialize and de-initialize the core environment.
- OMX_ComponentNameEnum(), OMX_GetComponentsOfRole(), OMX_GetRolesOfComponent() These functions provide information about the components available in the system, the components that support a specific role and the roles supported by a given component.
- OMX_ComponentNameEnum(cComponentName, nNameLength, nIndex) The OMX_ComponentNameEnum method will enumerate through all the names of recognized components in the system to detect all the components in the system run-time. cComponentName is a pointer to a null-terminated string with the component name.
- OMX_GetHandle(), OMX_FreeHandle() These functions are used to create and destroy an instance of a given component
- OMX_SetupTunnel() This function tries to establish a tunnel between two ports of two components.
OpenMax IL Components
Each functional block (HW, SW, or a combination) that is required to implement a particular multimedia function, such as video capture and video encode, is represented as a component.
Components are represented as object oriented classes; where base functionality is inherited from the OMX_BASE class and standard APIs are extended to suit the component.
An OpenMAX IL component is a black box that receives instruction with asynchronous commands and transmit/receive data through ports. Ports represent both the connection for components to the data stream and the buffers needed to maintain the connection. Users may send data to components through input ports or receive data through output ports. Similarly, a communication tunnel between two components can be established by connecting the output port of one component to a similarly formatted input port of another component.
Buffer status, errors, and other time-sensitive data are relayed to the application via a set of callback functions.
The individual parameters of a component can be set or retrieved through a set of associated data structures, enumerations, and interfaces.
A component is base profile compliant if it supports the standard and buffer are exchanged only with an IL client.
A component is interop profile compliant if it supports the base profile and is able to setup the tunneling with other interop components.
Standard Components
OpenMAX IL defines a set of standard components.
OpenMAX IL establishes two constructs for the hierarchical definition of the set of standard components:
- Standard component class a category of standard components that share the same ports and high level functionality.
- Standard component an instance of a standard component class that has the same ports and high level functionality as the class but that specifies the supported formats, parameters, and configs on those ports as well as the specific functionality of the component.
- the audio_decoder class This represents all components that receive encoded audio on a single audio input port and emit decoded audio on single audio output. The audio_decoder class contains a standard component definition for each audio format: audio_decoder.mp3, audio_decoder.aac, audio_decoder.amr, etc.
- the differences between components in the same audio_decoder class The difference may be the specific format of audio decoding implemented , or, in terms of their specific functionality.
Component Role
A role is defined as the behavior of component acting according to a particular standard component definition.
The name of the standard component defining the behavior identifies the role.
For example, a given component implementation named “OMX.CompanyXYZ.MyAudioDecoder” might support the following roles:
audio_decoder.mp3, audio_decoder.aac, audio_decoder.amr
Two convenient functions are provided for the IL client to query about which roles are supported by which component implementation:
- OMX_GetRolesOfComponent query all the roles fulfilled by a given a component.
- OMX_GetComponentsOfRole query the names of all installed components that support a given role.
OpenMAX IL Control API
OpenMAX IL Types
Enumerations
There are 6 enumeration types defined in OMX_Core.h:- OMX_COMMANDTYPE used to specify the action in the OMX_SendCommand macro.
typedef enum OMX_COMMANDTYPE
{
OMX_CommandStateSet, /** Change the component state */
OMX_CommandFlush, /** Flush the data queue(s) of a component */
OMX_CommandPortDisable, /** Disable a port on a component. */
OMX_CommandPortEnable, /** Enable a port on a component. */
OMX_CommandMarkBuffer, /** Mark a component/buffer for observation */
OMX_CommandKhronosExtensions = 0x6F000000, /** Reserved region for introducing Khronos Standard Extensions */
OMX_CommandVendorStartUnused = 0x7F000000, /** Reserved region for introducing Vendor Extensions */
OMX_CommandMax = 0X7FFFFFFF
} OMX_COMMANDTYPE;
- OMX_StateLoaded Component has been loaded but has no resources allocated. A component is in the OMX_StateLoaded state after it has been created via an OMX_GetHandle() call and before allocation of its resources.
- OMX_StateIdle Component has all resources but has not transferred any buffers or begun processing data. The component shall acquire all of its static resources, including buffers for all enabled ports, before completing the transition from OMX_StateLoaded to OMX_StateIdle. Note, the component does not acquire buffers for any disabled ports.
- OMX_StateExecuting Component is transferring buffers and is processing data (if data is available).
- Any port that communicates with the IL client shall call the EmptyBufferDone and FillBufferDone callbacks to return an empty or full buffer, respectively, back to the IL client.
- Any tunneling port shall call OMX_FillThisBuffer() or OMX_EmptyThisBuffer() on its corresponding tunneled port to return an empty or full buffer, respectively, back to its tunneled port.
- OMX_StateInvalid Component is corrupt or has encountered an error from which it cannot recover.
- OMX_StatePause Component data processing has been paused but may be resumed from the point it was paused.
- OMX_StateWaitFor Resources Component is waiting for a resource to become available.
Data Types
- OMX_U8 An 8 bit unsigned quantity that is byte aligned.
- OMX_S8 An 8 bit signed quantity that is byte aligned.
- OMX_U16 A 16 bit unsigned quantity that is 16 bit word aligned.
- OMX_S16 A 16 bit signed quantity that is 16 bit word aligned.
- OMX_U32 A 32 bit unsigned quantity that is 32 bit word aligned.
- OMX_S32 A 32 bit signed quantity that is 32 bit word aligned.
- OMX_U64 A 64 bit unsigned quantity that is 64 bit word aligned.
- OMX_S64 A 64 bit signed quantity that is 64 bit word aligned.
- OMX_BU32 A bounded 32 bit unsigned quantity.
typedef struct OMX_BU32 {
OMX_U32 nValue;
OMX_U32 nMin;
OMX_U32 nMax;
} OMX_BU32;
typedef struct OMX_BS32 {
OMX_S32 nValue;
OMX_S32 nMin;
OMX_S32 nMax;
} OMX_BS32;
Structures
The first two fields of each OpenMAX IL defined data structure are:
OMX_U32 nSize; /* size of the structure in bytes */
OMX_VERSIONTYPE nVersion; /* OMX specification version information */
The entity that allocates an OpenMAX IL defined structure is responsible for filling in these two values. The OMX_INDEXTYPE enumeration is used to select a structure when either getting or setting parameters and/or configuration data.
Each entry in this enumeration maps to an OMX specified structure. OMX_Index.h defines it:
typedef enum OMX_INDEXTYPE {
OMX_IndexComponentStartUnused = 0x01000000,
OMX_IndexParamPriorityMgmt, /**< reference: OMX_PRIORITYMGMTTYPE */
OMX_IndexParamAudioInit, /**< reference: OMX_PORT_PARAM_TYPE */
OMX_IndexParamImageInit, /**< reference: OMX_PORT_PARAM_TYPE */
OMX_IndexParamVideoInit, /**< reference: OMX_PORT_PARAM_TYPE */
OMX_IndexParamOtherInit, /**< reference: OMX_PORT_PARAM_TYPE */
.
OMX_IndexPortStartUnused = 0x02000000,
OMX_IndexParamPortDefinition, /**< reference: OMX_PARAM_PORTDEFINITIONTYPE */
OMX_IndexParamCompBufferSupplier, /**< reference: OMX_PARAM_BUFFERSUPPLIERTYPE */
OMX_IndexReservedStartUnused = 0x03000000,
/* Audio parameters and configurations */
OMX_IndexAudioStartUnused = 0x04000000,
OMX_IndexParamAudioPortFormat, /**< reference: OMX_AUDIO_PARAM_PORTFORMATTYPE */
.
/* Image specific parameters and configurations */
OMX_IndexImageStartUnused = 0x05000000,
OMX_IndexParamImagePortFormat, /**< reference: OMX_IMAGE_PARAM_PORTFORMATTYPE */
.
/* Video specific parameters and configurations */
OMX_IndexVideoStartUnused = 0x06000000,
OMX_IndexParamVideoPortFormat, /**< reference: OMX_VIDEO_PARAM_PORTFORMATTYPE */
.
/* Image & Video common Configurations */
OMX_IndexCommonStartUnused = 0x07000000,
OMX_IndexParamCommonDeblocking, /**< reference: OMX_PARAM_DEBLOCKINGTYPE */
.
/* Reserved Time range */
OMX_IndexTimeStartUnused = 0x09000000,
OMX_IndexConfigTimeScale, /**< reference: OMX_TIME_CONFIG_SCALETYPE */
.
OMX_IndexMax = 0x7FFFFFFF
} OMX_INDEXTYPE;
OpenMAX IL Core Methods/Macros
The OpenMAX IL core implements the main interface for an IL client that wants to use OpenMAX IL components.
Functions
- OMX_Init() The OMX_Init() method initializes the OpenMAX IL core. Each OpenMAX IL client shall use OMX_Init() as their first call into OpenMAX IL.
- OMX_Deinit() An OpenMAX IL client shall use OMX_Deinit() as their last call into OpenMAX IL, after all OpenMAX IL-related resources have been released.
- OMX_GetHandle() If the component is valid, OMX_GetHandle() will invoke the component's methods to fill the component handle and set up the callbacks.
OMX_API OMX_ERRORTYPE OMX_APIENTRY OMX_GetHandle(
OMX_OUT OMX_HANDLETYPE *pHandle,
OMX_IN OMX_STRING cComponentName,
OMX_IN OMX_PTR pAppData,
OMX_CALLBACKTYPE *pCallBacks
)
- in cComponentName, pAppData, pCallBacks.
- out pHandle
do {
OMX_GetState(hComp, &eState);
} while (OMX_StateLoaded != eState);
OMX_FreeHandle(hComp);
OpenMAX IL Component Methods and Structures
OpenMAX IL components are defined in the OMX_Component.h header file. The structure OMX_COMPONENTTYPE holds the data fields and function entry points for a component:
- pComponentPrivate The component allocates and initializes this member when the component is first loaded. The application should not access this data area.
- pApplicationPrivate The component initializes this field during the call to SetCallbacks, as this field is provided back to the IL client when the component issues callbacks.
- GetComponentVersion() The IL client calls the GetComponentVersion component method via the OMX_GetComponentVersion() core macro.
OMX_ERRORTYPE (*GetComponentVersion)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_OUT OMX_STRING pComponentName,
OMX_OUT OMX_VERSIONTYPE* pComponentVersion,
OMX_OUT OMX_VERSIONTYPE* pSpecVersion,
OMX_OUT OMX_UUIDTYPE* pComponentUUID);
OMX_ERRORTYPE (*SendCommand)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_COMMANDTYPE Cmd,
OMX_IN OMX_U32 nParam1,
OMX_IN OMX_PTR pCmdData);
OMX_ERRORTYPE (*GetParameter)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_INDEXTYPE nParamIndex,
OMX_INOUT OMX_PTR pComponentParameterStructure);
OMX_ERRORTYPE (*SetParameter)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_INDEXTYPE nIndex,
OMX_IN OMX_PTR pComponentParameterStructure);
OMX_ERRORTYPE (*GetConfig)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_INDEXTYPE nIndex,
OMX_INOUT OMX_PTR pComponentConfigStructure);
SetConfig
OMX_ERRORTYPE (*SetConfig)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_INDEXTYPE nIndex,
OMX_IN OMX_PTR pComponentConfigStructure);
OMX_ERRORTYPE (*GetExtensionIndex)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_STRING cParameterName,
OMX_OUT OMX_INDEXTYPE* pIndexType);
OMX_ERRORTYPE (*GetState)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_OUT OMX_STATETYPE* pState);
OMX_ERRORTYPE (*ComponentTunnelRequest)(
OMX_IN OMX_HANDLETYPE hComp,
OMX_IN OMX_U32 nPort,
OMX_IN OMX_HANDLETYPE hTunneledComp,
OMX_IN OMX_U32 nTunneledPort,
OMX_INOUT OMX_TUNNELSETUPTYPE* pTunnelSetup);
OMX_ERRORTYPE (*UseBuffer)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_INOUT OMX_BUFFERHEADERTYPE** ppBufferHdr,
OMX_IN OMX_U32 nPortIndex,
OMX_IN OMX_PTR pAppPrivate,
OMX_IN OMX_U32 nSizeBytes,
OMX_IN OMX_U8* pBuffer);
OMX_ERRORTYPE (*AllocateBuffer)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_INOUT OMX_BUFFERHEADERTYPE** ppBuffer,
OMX_IN OMX_U32 nPortIndex,
OMX_IN OMX_PTR pAppPrivate,
OMX_IN OMX_U32 nSizeBytes);
OMX_ERRORTYPE (*FreeBuffer)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_U32 nPortIndex,
OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
OMX_ERRORTYPE (*EmptyThisBuffer)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
OMX_ERRORTYPE (*FillThisBuffer)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
SetCallbacks()
OMX_ERRORTYPE (*SetCallbacks)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_CALLBACKTYPE* pCallbacks,
OMX_IN OMX_PTR pAppData);
OMX_ERRORTYPE (*ComponentDeInit)(
OMX_IN OMX_HANDLETYPE hComponent);
OMX_ERRORTYPE (*UseEGLImage)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_INOUT OMX_BUFFERHEADERTYPE** ppBufferHdr,
OMX_IN OMX_U32 nPortIndex,
OMX_IN OMX_PTR pAppPrivate,
OMX_IN void* eglImage);
OMX_ERRORTYPE (*ComponentRoleEnum)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_OUT OMX_U8 *cRole,
OMX_IN OMX_U32 nIndex);
OMX_BUFFERHEADERTYPE, 3.1.2.7
In the context of a single port, each data buffer has a header associated with it that contains meta-information about the buffer.
The buffer headers are shared between IL client and ports.
typedef struct OMX_BUFFERHEADERTYPE
{
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U8* pBuffer;
OMX_U32 nAllocLen;
OMX_U32 nFilledLen;
OMX_U32 nOffset;
OMX_PTR pAppPrivate;
OMX_PTR pPlatformPrivate;
OMX_PTR pInputPortPrivate;
OMX_PTR pOutputPortPrivate;
OMX_HANDLETYPE hMarkTargetComponent;
OMX_PTR pMarkData;
OMX_U32 nTickCount;
OMX_TICKS nTimeStamp;
OMX_U32 nFlags;
OMX_U32 nOutputPortIndex;
OMX_U32 nInputPortIndex;
} OMX_BUFFERHEADERTYPE;
- pBuffer a pointer to the actual buffer where data is stored but not necessarily the start of valid data; for more information, see the description of nOffset below.
- nAllocLen the total size of the allocated buffer in bytes, including valid and unused byte.
- nFilledLen the total size of valid bytes currently in the buffer starting from the location specified by pBuffer and nOffset.
- nOffset the start offset of valid data in bytes from the start of the buffer.
- pAppPrivate a pointer to an IL client private structure.
- pPlatformPrivate a pointer to a private platform-specific structure.
- pOutputPortPrivate a private pointer of the output port that uses the buffer.
- pInputPortPrivate a private pointer of the input port that uses the buffer.
- hMarkTargetComponent the handle of the component that should emit an OMX_EventMark event upon processing this buffer.
- pMarkData pointer refers to IL client-specific data associated with the mark that is sent on OMX_EventMark when emitted.
- nTickCount an optional entry
- nTimeStamp a timestamp corresponding to the sample starting at the first logical sample boundary in the buffer.
- nFlags
- OMX_BUFFERFLAG_EOS A source component (e.g. a demuxer) sets OMX_BUFFERFLAG_EOS when it has reached the end of the stream content on a particular output port.
- OMX_BUFFERFLAG_STARTTIME The source of a stream (e.g., a de-multiplexing component) sets the OMX_BUFFERFLAG_STARTTIME flag on the buffer that contains the starting timestamp for the stream. The starting timestamp corresponds to the first data that should be displayed at startup or after a seek operation.
- OMX_BUFFERFLAG_DECODEONLY The source of a stream (e.g., a de-multiplexing component) sets the OMX_BUFFERFLAG_DECODEONLY flag on any buffer that should be decoded but not rendered.
- OMX_BUFFERFLAG_DECODEONLY The source of a stream (e.g., a de-multiplexing component) sets the OMX_BUFFERFLAG_DECODEONLY flag on any buffer that should be decoded but not rendered.
- OMX_BUFFERFLAG_DATACORRUPT set when the IL client identifies the data in the associated buffer as corrupt.
- OMX_BUFFERFLAG_ENDOFFRAME set by an output port when the last byte that a buffer payload contains is an end-of-frame.
- OMX_BUFFERFLAG_SYNCFRAME set by an output port to indicate that the buffer content contains a coded synchronization frame.
- OMX_BUFFERFLAG_EXTRADATA set by an output port when the buffer payload contains additional information appended to the end of the buffer payload.
- OMX_BUFFERFLAG_CODECCONFIG set by an output port when all bytes in the buffer form part or all of a set of codec specific configuration data.
- nOutputPortIndex the port index of the output port that uses the buffer. The value of nOutputPortIndex is undefined for the input port's buffer.
- nInputPortIndex the port index of the input port that uses the buffer. The value of nInputPortIndex is undefined for the output port's buffer.
Calling Sequences
The IL client, the OpenMAX IL core, and the components dynamically interact in a few meaningful use cases:
- initialization
- de-initialization
- data flow
- data tunneling setup
- data flow in the case of data tunneling
- dynamic port reconfiguration
OpenMAX IL Data API
Chapter 8: OpenMAX Components on the Raspberry Pi
Broadcom's Implementation of OpenMAX IL
This raspberrypi/userland repository contains the source code for the ARM side libraries used on Raspberry Pi. These typically are installed in /opt/vc/lib and includes source for the ARM side code to interface to: EGL, mmal, GLESv2, vcos, vchiq_arm, bcm_host, WFC, OpenVG.
It appears to be a thin wrapper(userland/interface) around its GPU API.
The implementation of OpenMAX IL Core is based on VCOS APIs.
Therefore, to build the OpenMax IL application, the include and library path of VCOS need to be set:
-I /opt/vc/include/IL -I /opt/vc/include -I /opt/vc/include/interface/vcos/pthreads -L /opt/vc/lib -l openmaxil -l bcm_host
Component Initialization
All IL clients must initialize OpenMAX IL by calling OMX_Init().
err = OMX_Init();
if(err != OMX_ErrorNone) {
fprintf(stderr, "OMX_Init() failed\n", 0);
exit(1);
}
Before you can do anything with a component , an instance of the component must be created.
OMX_GetHandle() must be invoked to create an instance of a component,
OMX_HANDLETYPE handle;
OMX_CALLBACKTYPE callbacks;
OMX_ERRORTYPE err;
OMX_CALLBACKTYPE callbacks = { .EventHandler = NULL,
.EmptyBufferDone = NULL,
.FillBufferDone = NULL
};
err = OMX_GetHandle(&handle, componentName, NULL, &callbacks);
An IL client gets a component by calling OMX_GetHandle(), passing in the name of the component. The components do not have a standard name .No standardization among component names is dictated across different vendors.
A naming convention of a component is defined. OpenMAX IL component names are zero-terminated strings with the following format:
“OMX.<vendor_name>.<vendor_specified_convention>"
There is a simple way to get the list of the supported components by writing a simple IL application program.
#include <stdio.h>
#include <stdlib.h>
#include <OMX_Core.h>
#ifdef RASPBERRY_PI
#include <bcm_host.h>
#endif
OMX_ERRORTYPE err;
void listroles(char *name) {
int n;
OMX_U32 numRoles;
OMX_U8 *roles[32];
/* get the number of roles by passing in a NULL roles param */
err = OMX_GetRolesOfComponent(name, &numRoles, NULL);
if (err != OMX_ErrorNone) {
fprintf(stderr, "Getting roles failed\n", 0);
exit(1);
}
printf(" Num roles is %d\n", numRoles);
if (numRoles > 32) {
printf("Too many roles to list\n");
return;
}
/* now get the roles */
for (n = 0; n < numRoles; n++) {
roles[n] = malloc(OMX_MAX_STRINGNAME_SIZE);
}
err = OMX_GetRolesOfComponent(name, &numRoles, roles);
if (err != OMX_ErrorNone) {
fprintf(stderr, "Getting roles failed\n", 0);
exit(1);
}
for (n = 0; n < numRoles; n++) {
printf(" role: %s\n", roles[n]);
free(roles[n]);
}
/* This is in version 1.2
for (i = 0; OMX_ErrorNoMore != err; i++) {
err = OMX_RoleOfComponentEnum(role, name, i);
if (OMX_ErrorNone == err) {
printf(" Role of omponent is %s\n", role);
}
}
*/
}
int main(int argc, char** argv) {
int i;
unsigned char name[OMX_MAX_STRINGNAME_SIZE];
# ifdef RASPBERRY_PI
bcm_host_init();
# endif
err = OMX_Init();
if (err != OMX_ErrorNone) {
fprintf(stderr, "OMX_Init() failed\n", 0);
exit(1);
}
err = OMX_ErrorNone;
for (i = 0; OMX_ErrorNoMore != err; i++) {
err = OMX_ComponentNameEnum(name, OMX_MAX_STRINGNAME_SIZE, i);
if (OMX_ErrorNone == err) {
printf("Component is %s\n", name);
listroles(name);
}
}
printf("No more components\n");
/*
i= 0 ;
while (1) {
printf("Component %s\n", OMX_ComponentRegistered[i++]);
}
*/
exit(0);
}
cc -g -DRASPBERRY_PI -I /opt/vc/include/IL -I /opt/vc/include -I /opt/vc/include/interface/vcos/pthreads -L /opt/vc/lib -l openmaxil -l bcm_host -o listcomponents listcomponents.c
$ ./listcomponents
- OMX.broadcom.audio_capture
- OMX.broadcom.audio_decode
- OMX.broadcom.audio_encode
- OMX.broadcom.audio_render
- OMX.broadcom.audio_mixer
- OMX.broadcom.audio_splitter
- OMX.broadcom.audio_processor
- OMX.broadcom.camera
- OMX.broadcom.clock
- OMX.broadcom.coverage
- OMX.broadcom.egl_render
- OMX.broadcom.image_fx
- OMX.broadcom.image_decode
- OMX.broadcom.image_encode
- OMX.broadcom.image_read
- OMX.broadcom.image_write
- OMX.broadcom.read_media
- OMX.broadcom.resize
- OMX.broadcom.source
- OMX.broadcom.text_scheduler
- OMX.broadcom.transition
- OMX.broadcom.video_decode
- OMX.broadcom.video_encode
- OMX.broadcom.video_render
- OMX.broadcom.video_scheduler
- OMX.broadcom.video_splitter
- OMX.broadcom.visualisation
- OMX.broadcom.write_media
- OMX.broadcom.write_still
The Broadcom OpenMAX IL components used on the Raspberry Pi are documented at the VMCS-X OpenMAX IL Components page.
As the visualization component as an example, it has
- two audio ports (green) an input port #140 and an output port #141;
- a timer port (yellow) an input port #143
- a video output port (red) an output port #142
The notification of various completion events occur asynchronous. So the use of callback functions is implementated as part of the OpenMax framework to facilitate the asynchronous flow between the application and components.
Port Information ( Data Structure)
Ports are the means by which a component communicates, where clients pass data into a port’s input buffers and receive processed data back from a port’s output buffers.
The OMX_PARAM_PORTDEFINITIONTYPE(3.1.2.12) structure contains a set of generic fields that characterize each port of the component.
The IL client uses this structure to retrieve general information from each port.
typedef struct OMX_PARAM_PORTDEFINITIONTYPE {
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPortIndex;
OMX_DIRTYPE eDir;
OMX_U32 nBufferCountActual;
OMX_U32 nBufferCountMin;
OMX_U32 nBufferSize;
OMX_BOOL bEnabled;
OMX_BOOL bPopulated;
OMX_PORTDOMAINTYPE eDomain;
union {
OMX_AUDIO_PORTDEFINITIONTYPE audio;
OMX_VIDEO_PORTDEFINITIONTYPE video;
OMX_IMAGE_PORTDEFINITIONTYPE image;
OMX_OTHER_PORTDEFINITIONTYPE other;
} format;
OMX_BOOL bBuffersContiguous;
OMX_U32 nBufferAlignment;
} OMX_PARAM_PORTDEFINITIONTYPE;
The 1st field is the size of the structure , the 2nd field is the version of OpenMAX that is being used.All of the OpenMAX data structures have the following 2 fields and must be initialized before using it.
struct OMX_PARAM_PORTDEFINITIONTYPE portdef;
memset(&portdef, 0, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
portdef.nVersion.nVersion = OMX_VERSION;
The 3rd filed field nPortIndex is a read-only field that identifies the port. One way of finding this is from the documentation. Each port has buffers. Information about these buffers is stored in the next set of fields:
OMX_U32 nBufferCountActual; // the number of buffers that are required on this port before it is populated. >= nBufferCountMin
OMX_U32 nBufferCountMin; // read-only, the minimum number of buffers that the port requires
OMX_U32 nBufferSize; // read-only, the minimum size in bytes for buffers that are allocated for this port
OMX_BOOL bPopulated; // read-only, indicate if a port is populated( all of the buffers have been allocated on the port).
The filed eDomain can be used to identify the type (3.1.3) of the port and imply the specific data structure used in OMX_PARAM_PORTDEFINITIONTYPE.format:
- OMX_PortDomainAudio Specifies that the field format is a structure of the OMX_AUDIO_PORTDEFINITIONTYPE type.
- OMX_PortDomainVideo Specifies that the field format is a structure of the OMX_VIDEO_PORTDEFINITIONTYPE type.
- OMX_PortDomainImage Specifies that the field format is a structure of the OMX_IMAGE_PORTDEFINITIONTYPE type.
Getting and Setting Ports
OpenMAX has a standard way of getting and setting parameters. A handle to a component is required for getting/setting.
For each parameter data structure, there is a corresponding parameter index.
For OMX_PARAM_PORTDEFINITIONTYPE as an ex., the index is OMX_IndexParamPortDefinition.
To get the information of a generic port:
OMX_PARAM_PORTDEFINITIONTYPE portdef;
err = OMX_GetParameter(handle, OMX_IndexParamPortDefinition, &portdef)
This similar pattern can be used for other parameter data structures.
Getting and setting information about port is done by the functions
- OMX_GetParameter(), 3.2.2.8 The OMX_GetParameter macro will get a parameter setting from a component.
#define OMX_ GetParameter ( hComponent, nParamIndex, pComponentParameterStructure)
((OMX_COMPONENTTYPE*)hComponent)->GetParameter(
hComponent,
nParamIndex,
pComponentParameterStructure)
- The caller shall provide memory for the structure and populate the nSize and nVersion fields before invoking this macro.
- If the parameter settings are for a port, the caller shall also provide a valid port number in the nPortIndex field before invoking this macro.
- This call is a blocking call. The component should return from this call within 20
milliseconds.
/* disable every audio port of a component*/
OMX_GetParameter(hComp, OMX_IndexParamAudioInit, &oParam);
for (i=0; i< oParam.nPorts; i++) {
OMX_SendCommand(
hComp,
OMX_CommandPortDisable,
oParam.nStartPortNumber + i,
0);
}
#define OMX_SetParameter (
hComponent,
nParamIndex,
pComponentParameterStructure)
((OMX_COMPONENTTYPE*)hComponent)->SetParameter(
hComponent,
nParamIndex,
pComponentParameterStructure)
- The caller shall provide the memory for the correct structure and shall fill in the structure nSize and nVersion fields in addition to all other fields before invoking this macro.
- The caller is free to dispose of this structure after the call, as the component is required to copy any data it shall retain.
- This call is a blocking call. The component should return from this call within 20 milliseconds.
/* force a port to be the supplier */
OMX_GetParameter(hComp, OMX_IndexParamPortDefinition , &oPortDef);
if (oPortDef.eDir == OMX_DirInput){
oSupplier.eBufferSupplier = OMX_BufferSupplyInput;
} else {
oSupplier.eBufferSupplier = OMX_BufferSupplyOutput;
}
oSupplier.nPortIndex = nPortIndex;
OMX_SetParameter(hComp, OMX_IndexParamCompBufferSupplier, &oSupplier);
#define OMX_GetConfig (
hComponent,
nConfigIndex,
pComponentConfigStructure)
((OMX_COMPONENTTYPE*)hComponent)->GetConfig(
hComponent,
nConfigIndex,
pComponentConfigStructure)
- This macro can be invoked at any time after the component has been loaded.
- The caller shall provide the memory for the structure and populate the
nSize and nVersion fields before invoking this macro. - The nConfigIndex parameter indicates which structure is being requested from the component.
- Audio Table 4-2: [ OMX_IndexConfigAudioMidiImmediateEvent, ... , OMX_IndexConfigAudioNoiseReduction ]
- Video Table 4-32: [ OMX_IndexConfigCommonColorFormatConversion, ..., OMX_IndexConfigCommonFocusStatus ] Table 4-45: [ OMX_IndexConfigVideoBitrate, ..., OMX_IndexConfigVideoNalSize ]
- Image Table 4-65: [ OMX_IndexConfigFlashControl, ..., OMX_IndexConfigFocusControl ]
- Other Table 4-71: [ OMX_IndexConfigTimeScale, ..., OMX_IndexConfigTimeSeekMode]
- If the configuration settings are for a port, the caller shall also provide a valid port number in the nPortIndex field before invoking this macro.
/* Wait until a certain playback position */
do {
OMX_GetConfig(hClockComp, OMX_IndexConfigTimeCurrentMediaTime, oMediaTime);
} while (oMediaStamp.nTimeStamp < nTargetTimeStamp);
#define OMX_SetConfig (
hComponent,
nConfigIndex,
pComponentConfigStructure
)
((OMX_COMPONENTTYPE*)hComponent)->SetConfig(
hComponent,
nConfigIndex,
pComponentConfigStructure)
- Some configuration structures contain read-only fields. The OMX_SetConfig method will preserve read-only fields in configuration structures that contain them, and shall not generate an error when the caller attempts to change the value of a read-only field.
- This call is a blocking call.
/* Change the time scale of the clock component*/
oScale.xScale = 0x00020000; /*2x*/
OMX_SetConfig(hClockComp, OMX_IndexConfigTimeScale, (OMX_PTR)&oScale);
Getting Information About a Port
Try to write a program portinfo.c to get the information about a particular port on a component and print it out.
This program takes a component name and a port index as parameters.
#include <stdio.h>
#include <stdlib.h>
#include <OMX_Core.h>
#include <OMX_Component.h>
#include <bcm_host.h>
OMX_ERRORTYPE get_port_info(OMX_HANDLETYPE handle, OMX_PARAM_PORTDEFINITIONTYPE *portdef) {
// assign the index to use the structure passed
return OMX_GetParameter(handle, OMX_IndexParamPortDefinition, portdef);
}
void print_port_info(OMX_PARAM_PORTDEFINITIONTYPE *portdef) {
char *domain;
printf("Port %d\n", portdef->nPortIndex);
if (portdef->eDir == OMX_DirInput) {
printf(" is input port\n");
} else {
printf(" is output port\n");
}
switch (portdef->eDomain) {
case OMX_PortDomainAudio:
domain = "Audio";
break;
case OMX_PortDomainVideo:
domain = "Video";
break;
case OMX_PortDomainImage:
domain = "Image";
break;
case OMX_PortDomainOther:
domain = "Other";
break;
}
printf("Domain is %s\n", domain);
printf("Buffer count %d\n", portdef->nBufferCountActual);
printf("Buffer minimum count %d\n", portdef->nBufferCountMin);
printf("Buffer size %d bytes\n", portdef->nBufferSize);
};
OMX_CALLBACKTYPE callbacks= {
.EventHandler = NULL,
.EmptyBufferDone = NULL,
.FillBufferDone = NULL
};
int main(int argc, char** argv) {
int i;
char componentName[128]; // min space required see /opt/vc/include/IL/OMX_Core.h
OMX_ERRORTYPE err;
OMX_HANDLETYPE handle;
OMX_PARAM_PORTDEFINITIONTYPE portdef;
OMX_VERSIONTYPE specVersion, compVersion;
OMX_UUIDTYPE uid;
int portindex;
if (argc < 3) {
fprintf(stderr, "Usage: %s component-name port-index\n", argv[0]);
exit(1);
}
strncpy(componentName, argv[1], 128);
portindex = atoi(argv[2]);
bcm_host_init();
err = OMX_Init();
if (err != OMX_ErrorNone) {
fprintf(stderr, "OMX_Init() failed\n", 0);
exit(1);
}
/** Ask the core for a handle to the component
*/
err = OMX_GetHandle(&handle, componentName, NULL, &callbacks);
if (err != OMX_ErrorNone) {
fprintf(stderr, "OMX_GetHandle failed\n", 0);
exit(1);
}
// Get some version info
err = OMX_GetComponentVersion(handle, componentName, &compVersion, &specVersion, &uid);
if (err != OMX_ErrorNone) {
fprintf(stderr, "OMX_GetComponentVersion failed\n", 0);
exit(1);
}
printf("Component name: %s version %d.%d, Spec version %d.%d\n",
componentName, compVersion.s.nVersionMajor,
compVersion.s.nVersionMinor,
specVersion.s.nVersionMajor,
specVersion.s.nVersionMinor);
memset(&portdef, 0, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
portdef.nVersion.nVersion = OMX_VERSION;
portdef.nPortIndex = portindex;
if (get_port_info(handle, &portdef) == OMX_ErrorNone) {
print_port_info(&portdef);
}
exit(0);
}
Build the code:
$ cc -I /opt/vc/include/IL -I /opt/vc/include -I /opt/vc/include/interface/vcos/pthreads -L /opt/vc/lib -l openmaxil -l bcm_host portinfo.c -o portinfo
Then, run it:
$ ./portinfo OMX.broadcom.image_decode 320
Component name: OMX.broadcom.image_decode:0 version 0.0, Spec version 1.1
Port 320
is input port
Domain is Image
Buffer count 3
Buffer minimum count 2
Buffer size 81920 bytes
Getting Information About Specific Type of Ports
A component uses the OMX_PORT_PARAM_TYPE structure to identify the number and starting index of ports of a particular domain.
typedef struct OMX_PORT_PARAM_TYPE {
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPorts;
OMX_U32 nStartPortNumber;
} OMX_PORT_PARAM_TYPE;
- nPorts is the number of ports of a given port domain (audio, video, image, or other) for the component.
- nStartPortNumber is the index of the first port of a given port domain (audio, video, image, or other) for the component . Subsequent ports of the given domain are numbered sequentially from nStartPortNumber.
- OMX_IndexParamAudioInit
- OMX_IndexParamImageInit
- OMX_IndexParamVideoInit
- OMX_IndexParamOtherInit
OMX_PORT_PARAM_TYPE param;
err = OMX_GetParameter(handle, OMX_IndexParamVideoInit, ¶m);
Detailed Information for the Audio/Video/Image/Other Ports
To get the port's information:
OMX_GetParameter(handle, OMX_IndexParamPortDefinition, &portdef)
Each type of port hase its own specific information, to get the specific information for each type of port:- the audio port portdef.format.audio is the specific information with the type OMX_AUDIO_PORTDEFINITIONTYPE (4.1.5). This tructure is used to define all of the parameters necessary for the compliant component to set up an input or an output audio path:
typedef struct OMX_AUDIO_PORTDEFINITIONTYPE {
OMX_STRING cMIMEType;
OMX_NATIVE_DEVICETYPE pNativeRender;
OMX_BOOL bFlagErrorConcealment;
OMX_AUDIO_CODINGTYPE eEncoding;
} OMX_AUDIO_PORTDEFINITIONTYPE;
- cMIMEType is the MIME type of data for the port. If a MIME type string buffer is not supplied this parameter shall be set to NULL.
- pNativeRender is the platform-specific reference for an output device; otherwise this field is 0.
- bFlagErrorConcealment turns on error concealment if it is supported by the OpenMAX IL component.
- eEncoding is the type of data expected for this port (e.g., PCM, AMR, MP3, and so forth). These types are also defined in the header file /opt/vc/include/IL/OMX_Audio.h .
typedef struct OMX_AUDIO_PARAM_PCMMODETYPE {
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPortIndex;
OMX_U32 nChannels;
OMX_NUMERICALDATATYPE eNumData;
OMX_ENDIANTYPE eEndian;
OMX_BOOL bInterleaved;
OMX_U32 nBitPerSample;
OMX_U32 nSamplingRate;
OMX_AUDIO_PCMMODETYPE ePCMMode;
OMX_AUDIO_CHANNELTYPE eChannelMapping[OMX_AUDIO_MAXCHANNELS];
} OMX_AUDIO_PARAM_PCMMODETYPE;
OMX_AUDIO_PARAM_PCMMODETYPE sPCMMode;
err = OMX_GetParameter(handle, OMX_IndexParamAudioPcm, &sPCMMode);
typedef struct OMX_VIDEO_PORTDEFINITIONTYPE {
OMX_STRING cMIMEType;
OMX_NATIVE_DEVICETYPE pNativeRender;
OMX_U32 nFrameWidth;
OMX_U32 nFrameHeight;
OMX_S32 nStride; // buffer parameter
OMX_U32 nSliceHeight; // buffer parameter
OMX_U32 nBitrate;
OMX_U32 xFramerate;
OMX_BOOL bFlagErrorConcealment;
OMX_VIDEO_CODINGTYPE eCompressionFormat;
OMX_COLOR_FORMATTYPE eColorFormat;
OMX_NATIVE_WINDOWTYPE pNativeWindow;
} OMX_VIDEO_PORTDEFINITIONTYPE;
- nStride is a field containing the number of bytes per span of an image, which indicates the number of bytes to get from span N to span N+1.
- nSliceHeight is a read-only field containing the slice height parameter used when processing uncompressed image data. This describes the plane height of a multi-planar (YUV) video bytebuffer layout. "4.2.3 Buffer Payload Requirements for Uncompressed Data":
For each of these uncompressed formats(OMX_COLOR_FormatYUVxxx), each buffer payload contains a slice of the Y, U, and V planes. The nSliceHeight refers to the slice height of the Y plane. The slice height of the U and V planes are derived from the slice height for the Y plane based upon for the format."4.2.2 Minimum Buffer Payload Size for Uncompressed Data":
Uncompressed image and video data have a minimum buffer payload size. The minimum buffer payload size is determined by the nSliceHeight and nStride fields. nStride indicates the width of a span in bytes; when negative, it indicates the data is bottom-up instead of the top-down). nSliceHeight indicates the number of spans in a slice. The minimum buffer payload size can be easily calculated as the absolute value of ( nSliceHeight * nStride )For example, for OMX_COLOR_FormatYUV420PackedPlanar with a nSliceHeight of 16, a buffer payload shall contain 16 spans of Y followed by 8 spans of U (half the stride) and 8 spans of V (half the stride). This enables ports that process planar data in slices to operate on all three planes simultaneously, instead of forcing the ports to buffer the entire image before processing can begin.
OMX_Types.h:
/** The OMX_PTR type is intended to be used to pass pointers between the OMX
applications and the OMX Core and components. This is a 32 bit pointer and
is aligned on a 32 bit boundary.
*/
typedef void* OMX_PTR;
You can query the port information in more detail:
static void setStructureSizeVer(OMX_PTR header, OMX_U32 size) {
/* header->nVersion */
OMX_VERSIONTYPE* ver = (OMX_VERSIONTYPE*)(header + sizeof(OMX_U32));
/* header->nSize */
*((OMX_U32*)header) = size;
ver->s.nVersionMajor = specVersion.s.nVersionMajor;
ver->s.nVersionMinor = specVersion.s.nVersionMinor;
ver->s.nRevision = specVersion.s.nRevision;
ver->s.nStep = specVersion.s.nStep;
}
OMX_ERRORTYPE setAudioPortNumEncoding(int portNumber, OMX_AUDIO_CODINGTYPE encoding) {
OMX_PARAM_PORTDEFINITIONTYPE sPortDef;
setStructureSizeVer(&sPortDef, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
sPortDef.nPortIndex = portNumber;
sPortDef.nPortIndex = portNumber;
err = OMX_GetParameter(handle, OMX_IndexParamPortDefinition, &sPortDef);
if(err != OMX_ErrorNone){
fprintf(stderr, "Error in getting OMX_PORT_DEFINITION_TYPE parameter\n",0);
exit(1);
}
sPortDef.format.audio.eEncoding = encoding;
sPortDef.nBufferCountActual = sPortDef.nBufferCountMin;
err = OMX_SetParameter(handle, OMX_IndexParamPortDefinition, &sPortDef);
return err;
}
void getPCMInformation(int portNumber) {
/* assert: PCM is a supported mode */
OMX_AUDIO_PARAM_PCMMODETYPE sPCMMode;
/* set it into PCM format before asking for PCM info */
if (setAudioPortNumEncoding(portNumber, OMX_AUDIO_CodingPCM) != OMX_ErrorNone) {
fprintf(stderr, "Error in setting coding to PCM\n");
return;
}
setStructureSizeVer(&sPCMMode, sizeof(OMX_AUDIO_PARAM_PCMMODETYPE));
/* read back the port's setting */
sPCMMode.nPortIndex = portNumber;
err = OMX_GetParameter(handle, OMX_IndexParamAudioPcm, &sPCMMode);
if(err != OMX_ErrorNone){
printf("PCM mode unsupported\n");
} else {
printf(" PCM default sampling rate %d\n", sPCMMode.nSamplingRate);
printf(" PCM default bits per sample %d\n", sPCMMode.nBitPerSample);
printf(" PCM default number of channels %d\n", sPCMMode.nChannels);
}
}
void getSupportedAudioFormats(int indentLevel, int portNumber) {
OMX_AUDIO_PARAM_PORTFORMATTYPE sAudioPortFormat;
setStructureSizeVer(&sAudioPortFormat, sizeof(OMX_AUDIO_PARAM_PORTFORMATTYPE));
sAudioPortFormat.nIndex = 0;
sAudioPortFormat.nPortIndex = portNumber;
printf("Supported audio formats are:\n");
for(;;) {
err = OMX_GetParameter(handle, OMX_IndexParamAudioPortFormat, &sAudioPortFormat);
if (err == OMX_ErrorNoMore) {
printf("No more formats supported\n");
return;
}
/* This shouldn't occur, but does with Broadcom library */
if (sAudioPortFormat.eEncoding == OMX_AUDIO_CodingUnused) {
printf("No coding format returned\n");
return;
}
switch (sAudioPortFormat.eEncoding) {
case OMX_AUDIO_CodingPCM:
printf("Supported encoding is PCM\n");
getPCMInformation(portNumber);
break;
case OMX_AUDIO_CodingVORBIS:
printf("Supported encoding is Ogg Vorbis\n");
break;
case OMX_AUDIO_CodingMP3:
printf("Supported encoding is MP3\n");
getMP3Information(portNumber);
break;
.
.
.
default:
printf("Supported encoding is not PCM or MP3 or Vorbis, is 0x%X\n", sAudioPortFormat.eEncoding);
}
sAudioPortFormat.nIndex++;
}
}
void getPortMediaInformation(int nPort, OMX_PARAM_PORTDEFINITIONTYPE sPortDef) {
printf("Port %d requires %d buffers\n", nPort, sPortDef.nBufferCountMin);
printf("Port %d has min buffer size %d bytes\n", nPort, sPortDef.nBufferSize);
if (sPortDef.eDir == OMX_DirInput) {
printf("Port %d is an input port\n", nPort);
} else {
printf("Port %d is an output port\n", nPort);
}
switch (sPortDef.eDomain) {
case OMX_PortDomainAudio:
printf("Port %d is an audio port\n", nPort);
printf("Port mimetype %s\n",
sPortDef.format.audio.cMIMEType);
switch (sPortDef.format.audio.eEncoding) {
case OMX_AUDIO_CodingPCM:
printf("Port encoding is PCM\n");
break;
case OMX_AUDIO_CodingVORBIS:
printf("Port encoding is Ogg Vorbis\n");
break;
case OMX_AUDIO_CodingMP3:
printf("Port encoding is MP3\n");
break;
default:
printf("Port encoding is not PCM or MP3 or Vorbis, is %d\n",
sPortDef.format.audio.eEncoding);
}
getSupportedAudioFormats(nPort);
break;
/* could put other port types here */
case OMX_PortDomainVideo:
.
.
break
case OMX_PortDomainImage:
.
.
break
default:
printf("Port %d is not an audio port\n", nPort);
}
}
void getAllPortsInformation() {
OMX_PORT_PARAM_TYPE param;
OMX_PARAM_PORTDEFINITIONTYPE sPortDef;
int startPortNumber;
int nPorts;
int n;
int index_param_media =0;
for ( index_param_media= OMX_IndexParamAudioInit; i<= OMX_IndexParamOtherInit; i ++) {
if ( index_param_media == OMX_IndexParamAudioInit)
printf("Audio ports:\n");
else if ( index_param_media == OMX_IndexParamVideoInit)
printf("Video ports:\n");
else if ( index_param_media == OMX_IndexParamImageInit)
printf("Image ports:\n");
else if ( index_param_media == OMX_IndexParamOtherInit)
printf("Other ports:\n");
setStructureSizeVer(¶m, sizeof(OMX_PORT_PARAM_TYPE));
err = OMX_GetParameter(handle, index_param_media, ¶m);
if(err != OMX_ErrorNone){
fprintf(stderr, "Error in getting audio OMX_PORT_PARAM_TYPE parameter\n", 0);
return;
}
startPortNumber = param.nStartPortNumber;
nPorts = param.nPorts;
if (nPorts == 0) {
printf("No ports of this type\n");
return;
}
printf("Ports start on %d\n", startPortNumber);
printf("There are %d open ports\n", nPorts);
for (n = 0; n < nPorts; n++) {
setStructureSizeVer(&sPortDef, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
sPortDef.nPortIndex = startPortNumber + n;
err = OMX_GetParameter(handle, OMX_IndexParamPortDefinition, &sPortDef);
if(err != OMX_ErrorNone){
fprintf(stderr, "Error in getting OMX_PORT_DEFINITION_TYPE parameter\n", 0);
exit(1);
}
printf("Port %d has %d buffers of size %d\n",
sPortDef.nPortIndex,
sPortDef.nBufferCountActual,
sPortDef.nBufferSize);
printf("Direction is %s\n", (sPortDef.eDir == OMX_DirInput ? "input" : "output"));
getPortMediaInformation(startPortNumber + n, sPortDef);
}
}
}
Chapter 9: OpenMAX on the Raspberry Pi State
The OpenMAX IL includes a callback mechanism that allows a component to communicate with the IL client
Whenever a component changes state, it will invoke the event callback handler established for that component.
To accomplish a callback, the OpenMAX IL has 3 callback functions defined: 1 generic event handler(EventHandler) and 2 callbacks related to the dataflow (EmptyBufferDone and FillBufferDone).
The IL client is responsible for filling in an OMX_CALLBACKTYPE structure with its callback entry points and passing the structure to the OpenMAX IL core at initialization (init) time.
typedef struct OMX_CALLBACKTYPE {
OMX_ERRORTYPE (*EventHandler)( OMX_IN OMX_HANDLETYPE hComponent, OMX_IN OMX_PTR pAppData, OMX_IN OMX_EVENTTYPE eEvent, OMX_IN OMX_U32 nData1, OMX_IN OMX_U32 nData2, OMX_IN OMX_PTR pEventData);
OMX_ERRORTYPE (*EmptyBufferDone)( OMX_IN OMX_HANDLETYPE hComponent, OMX_IN OMX_PTR pAppData, OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
OMX_ERRORTYPE (*FillBufferDone)( OMX_IN OMX_HANDLETYPE hComponent, OMX_IN OMX_PTR pAppData, OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
} OMX_CALLBACKTYPE;
- EventHandler A component uses the EventHandler method to notify the IL client when an event of interest occurs within the component. The parameters are as follows.
- hComponent The handle of the component that calls this function.
- eEvent The event that the component is communicating to the IL client.
- nData1 The first integer event-specific parameter.
- nData2 The second integer event-specific parameter.
- pEventData A pointer to additional event-specific data.
- EmptyBufferDone A component uses the EmptyBufferDone callback to pass a buffer from an input port back to the IL client. The EmptyBufferDone call is a blocking call that should return from within five milliseconds. The parameters are as follows.
- hComponent The handle of the component that is calling this function.
- pAppData A pointer to IL client-defined data.
- pBuffer A pointer to an OMX_BUFFERHEADERTYPE structure that was consumed or returned.
- FillBufferDone A component uses the FillBufferDone callback to pass a buffer from an output port back to the IL client.
Chapter 10: OpenMAX IL Client Library on the Raspberry Pi
The IL Client library is designed to make it easier to use OpenMAX on the RPi.
Many of the OpenMAX IL client function calls in the Broadcom examples ( userland/host_applications/linux/apps/hello_pi ) are embedded in a Broadcom convenience functions: userland/host_applications/linux/apps/hello_pi/libs/ilclient/ilclient.c ( installed as /opt/vc/src/hello_pi/libs/ilclient/ilclient.c ).
The Il Client functions are documented in the file hello_pi/libs/ilclient/ilclient.h and defined in hello_pi/libs/ilclient/ilclient.c
The 1st Step
bcm_host_init();
The 2nd Step
Use the IL client library to to the initialzation of a component:
COMPONENT_T *list[5];
ILCLIENT_T *client;
OMX_ERRORTYPE r;
COMPONENT_T *video_encode = NULL;
memset(list, 0, sizeof(list));
if ((client = ilclient_init()) == NULL) {
return -3;
}
if (OMX_Init() != OMX_ErrorNone) {
ilclient_destroy(client);
return -4;
}
// create video_encode
r = ilclient_create_component(client, &video_encode, "video_encode",
ILCLIENT_DISABLE_ALL_PORTS |
ILCLIENT_ENABLE_INPUT_BUFFERS |
ILCLIENT_ENABLE_OUTPUT_BUFFERS);
if (r != 0) {
printf("ilclient_create_component() for video_encode failed with %x!\n",r);
exit(1);
}
list[0] = video_encode;
- ILCLIENT_T The ILCLIENT_T structure encapsulates the state needed for the IL Client API. It contains a set of callback functions used to communicate with the user. It also includes a linked list of free event structures.
typedef struct _ILCLIENT_T ILCLIENT_T;
struct _ILCLIENT_T {
ILEVENT_T *event_list;
VCOS_SEMAPHORE_T event_sema;
ILEVENT_T event_rep[NUM_EVENTS];
ILCLIENT_CALLBACK_T port_settings_callback;
void *port_settings_callback_data;
ILCLIENT_CALLBACK_T eos_callback;
void *eos_callback_data;
ILCLIENT_CALLBACK_T error_callback;
void *error_callback_data;
ILCLIENT_BUFFER_CALLBACK_T fill_buffer_done_callback;
void *fill_buffer_done_callback_data;
ILCLIENT_BUFFER_CALLBACK_T empty_buffer_done_callback;
void *empty_buffer_done_callback_data;
ILCLIENT_CALLBACK_T configchanged_callback;
void *configchanged_callback_data;
};
typedef struct _COMPONENT_T COMPONENT_T;
struct _COMPONENT_T {
OMX_HANDLETYPE comp;
ILCLIENT_CREATE_FLAGS_T flags;
VCOS_SEMAPHORE_T sema;
VCOS_EVENT_FLAGS_T event;
struct _COMPONENT_T *related;
OMX_BUFFERHEADERTYPE *out_list;
OMX_BUFFERHEADERTYPE *in_list;
char name[32];
char bufname[32];
unsigned int error_mask;
unsigned int private;
ILEVENT_T *list;
ILCLIENT_T *client;
};
i = vcos_semaphore_create(&st->event_sema, "il:event", 1);
ilclient_lock_events(st);
st->event_list = NULL;
for (i=0; i < NUM_EVENTS; i++)
{
st->event_rep[i].eEvent = -1; // mark as unused
st->event_rep[i].next = st->event_list;
st->event_list = st->event_rep+i;
}
ilclient_unlock_events(st);
int ilclient_create_component(ILCLIENT_T *client, COMPONENT_T **comp, char *name, ILCLIENT_CREATE_FLAGS_T flags)
- handle The client handle
- comp On successful creation, the component structure pointer will be written back into comp.
- name The name of the component to be created. Component names as provided are automatically prefixed with "OMX.broadcom." before passing to the IL core.
- flags The client can specify some creation behaviour by using the flags field. The meaning of each flag is defined by the ILCLIENT_CREATE_FLAGS_T type.
- ILCLIENT_DISABLE_ALL_PORTS disable all ports on creation
- ILCLIENT_ENABLE_INPUT_BUFFERS allow the client to communicate with input ports via buffer communication, rather than tunneling with another component.
- ILCLIENT_ENABLE_OUTPUT_BUFFERS allow the client to communicate with output ports via buffer communication, rather than tunneling with another component.
- return 0 on success, -1 on failure
*comp = vcos_malloc(sizeof(COMPONENT_T), "il:comp");
status = vcos_event_flags_create(&(*comp)->event,"il:comp");
(*comp)->flags = flags;
callbacks.EventHandler = ilclient_event_handler;
callbacks.EmptyBufferDone = flags & ILCLIENT_ENABLE_INPUT_BUFFERS ? ilclient_empty_buffer_done : ilclient_empty_buffer_done_error;
callbacks.FillBufferDone = flags & ILCLIENT_ENABLE_OUTPUT_BUFFERS ? ilclient_fill_buffer_done : ilclient_fill_buffer_done_error;
error = OMX_GetHandle(&(*comp)->comp, component_name, *comp, &callbacks);
if (error == OMX_ErrorNone) {
if(flags & (ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_OUTPUT_ZERO_BUFFERS)) {
// disable ports
}
} else
{
vcos_event_flags_delete(&(*comp)->event);
vcos_semaphore_delete(&(*comp)->sema);
vcos_free(*comp);
*comp = NULL;
return -1;
}
ilclient_create_component() will install the buffer handler callbacks according to the fags. The 3rd Step
Use OpenMAX IL API to do the detail configuration on input/output ports:
OMX_PARAM_PORTDEFINITIONTYPE def;
// get current settings of video_encode component from port 200
memset(&def, 0, sizeof(OMX_PARAM_PORTDEFINITIONTYPE));
def.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
def.nVersion.nVersion = OMX_VERSION;
def.nPortIndex = 200;
if ( OMX_GetParameter( ILC_GET_HANDLE(video_encode), OMX_IndexParamPortDefinition, &def) != OMX_ErrorNone) {
printf("%s:%d: OMX_GetParameter() for video_encode port 200 failed!\n",__FUNCTION__, __LINE__);
exit(1);
}
// Port 200: in 1/1 115200 16 enabled,not pop.,not cont. 320x240 320x240 @1966080 20
def.format.video.nFrameWidth = 640;
def.format.video.nFrameHeight = ((640)*9/16);
def.format.video.xFramerate = 30 << 16;
def.format.video.nSliceHeight = ALIGN_UP(def.format.video.nFrameHeight, 16);
def.format.video.nStride = def.format.video.nFrameWidth;
def.format.video.eColorFormat = OMX_COLOR_FormatYUV420PackedPlanar;
printf("Port %u: %s %u/%u %u %u %s,%s,%s %ux%u %ux%u @%u %u\n",
def.nPortIndex,
def.eDir == OMX_DirInput ? "in" : "out",
def.nBufferCountActual,
def.nBufferCountMin,
def.nBufferSize,
def.nBufferAlignment,
def.bEnabled ? "enabled" : "disabled",
def.bPopulated ? "populated" : "not pop.",
def.bBuffersContiguous ? "contig." : "not cont.",
def.format.video.nFrameWidth,
def.format.video.nFrameHeight,
def.format.video.nStride,
def.format.video.nSliceHeight,
def.format.video.xFramerate, def.format.video.eColorFormat);
r = OMX_SetParameter( ILC_GET_HANDLE(video_encode), OMX_IndexParamPortDefinition, &def);
if (r != OMX_ErrorNone) {
printf("%s:%d: OMX_SetParameter() for video_encode port 200 failed with %x!\n", __FUNCTION__, __LINE__, r);
exit(1);
}
OMX_VIDEO_PARAM_PORTFORMATTYPE format;
memset(&format, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));
format.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);
format.nVersion.nVersion = OMX_VERSION;
format.nPortIndex = 201;
format.eCompressionFormat = OMX_VIDEO_CodingAVC;
printf("OMX_SetParameter for video_encode:201...\n");
r = OMX_SetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamVideoPortFormat, &format);
if (r != OMX_ErrorNone) {
printf
("%s:%d: OMX_SetParameter() for video_encode port 201 failed with %x!\n",
__FUNCTION__, __LINE__, r);
exit(1);
}
OMX_VIDEO_PARAM_BITRATETYPE bitrateType;
// set current bitrate to 1Mbit
memset(&bitrateType, 0, sizeof(OMX_VIDEO_PARAM_BITRATETYPE));
bitrateType.nSize = sizeof(OMX_VIDEO_PARAM_BITRATETYPE);
bitrateType.nVersion.nVersion = OMX_VERSION;
bitrateType.eControlRate = OMX_Video_ControlRateVariable;
bitrateType.nTargetBitrate = 1000000;
bitrateType.nPortIndex = 201;
r = OMX_SetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamVideoBitrate, &bitrateType);
if (r != OMX_ErrorNone) {
printf("%s:%d: OMX_SetParameter() for bitrate for video_encode port 201 failed with %x!\n",__FUNCTION__, __LINE__, r);
exit(1);
}
// get current bitrate
memset(&bitrateType, 0, sizeof(OMX_VIDEO_PARAM_BITRATETYPE));
bitrateType.nSize = sizeof(OMX_VIDEO_PARAM_BITRATETYPE);
bitrateType.nVersion.nVersion = OMX_VERSION;
bitrateType.nPortIndex = 201;
if (OMX_GetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamVideoBitrate, &bitrateType) != OMX_ErrorNone) {
printf("%s:%d: OMX_GetParameter() for video_encode for bitrate port 201 failed!\n",__FUNCTION__, __LINE__);
exit(1);
}
printf("Current Bitrate=%u\n",bitrateType.nTargetBitrate);
The 4-th Step
Use the IL client library to bring the component in the idle state and enable its ports and the buffering :
printf("encode to idle...\n");
if (ilclient_change_component_state(video_encode, OMX_StateIdle) == -1) {
printf("%s:%d: ilclient_change_component_state(video_encode, OMX_StateIdle) failed",__FUNCTION__, __LINE__);
}
printf("enabling port buffers for 200...\n");
if (ilclient_enable_port_buffers(video_encode, 200, NULL, NULL, NULL) != 0) {
printf("enabling port buffers for 200 failed!\n");
exit(1);
}
printf("enabling port buffers for 201...\n");
if (ilclient_enable_port_buffers(video_encode, 201, NULL, NULL, NULL) != 0) {
printf("enabling port buffers for 201 failed!\n");
exit(1);
}
- ilclient_change_component_state(COMPONENT_T *comp, OMX_STATETYPE state) The ilclient_change_component_state() function changes the state of an individual component. This will trigger the state change, and also wait for that state change to be completed. It should not be called if this state change has dependencies on other components (if the component is tunnelled, requires connected components to also change state.) also changing states.
error = OMX_SendCommand(comp->comp, OMX_CommandStateSet, state, NULL);
ilclient_wait_for_command_complete(comp, OMX_CommandStateSet, state)
OMX_SendCommand(comp->comp, OMX_CommandPortEnable, portIndex, NULL);
for (i=0; i != portdef.nBufferCountActual; i++)
{
unsigned char *buf;
if(ilclient_malloc)
buf = ilclient_malloc(private, portdef.nBufferSize, portdef.nBufferAlignment, comp->bufname);
else
buf = vcos_malloc_aligned(portdef.nBufferSize, portdef.nBufferAlignment, comp->bufname);
if(!buf)
break;
error = OMX_UseBuffer(comp->comp, end, portIndex, NULL, portdef.nBufferSize, buf);
if(error != OMX_ErrorNone)
{
if(ilclient_free)
ilclient_free(private, buf);
else
vcos_free(buf);
break;
}
end = (OMX_BUFFERHEADERTYPE **) &((*end)->pAppPrivate);
}
OMX_UseBuffer() ( 3.2.2.14) requests the component to use a buffer already allocated by the IL client or a buffer already supplied by a tunneled component.
#define OMX_UseBuffer (\
hComponent,\
ppBufferHdr,\
nPortIndex,\
pAppPrivate,\
nSizeBytes,\
pBuffer)\
((OMX_COMPONENTTYPE*)hComponent->UseBuffer(\
hComponent,\
ppBufferHdr,\
nPortIndex,\
pAppPrivate,\
nSizeBytes,\
pBuffer)
- hComponent [in] The handle of that component that executes the call.
- ppBufferHdr [out] A pointer to a pointer of an OMX_BUFFERHEADERTYPE structure that receives the pointer to the buffer header.
- nPortIndex [in] The index of the port that will use the specified buffer. This index is relative to the component that owns the port
- pAppPrivate [in] A pointer that refers to an implementation-specific memory area that is under responsibility of the supplier of the buffer.
- nSizeBytes [in] The buffer size in bytes.
- pBuffer [in] A pointer to the memory buffer area to be used.
for (i=0;inBufferCount;i++) {
pPort->pBuffer[i] = malloc(pPort->nBufferSize);
OMX_UseBuffer( pPort->hTunnelComponent,
&pPort->pBufferHdr[i],
pPort->nTunnelPort,
pPort,
pPort->nBufferSize,
pPort->pBuffer[j]);
}
The 5-th Step
Start to work:
printf("encode to executing...\n");
ilclient_change_component_state(video_encode, OMX_StateExecuting);
outf = fopen(outputfilename, "w");
if (outf == NULL) {
printf("Failed to open '%s' for writing video\n", outputfilename);
exit(1);
}
printf("looping for buffers...\n");
do {
buf = ilclient_get_input_buffer(video_encode, 200, 1);
if (buf == NULL) {
printf("Doh, no buffers for me!\n");
} else {
/* A pseudo card to fill the raw frame according to def.format.video.eColorFormat */
generate_test_card(buf->pBuffer, &buf->nFilledLen, framenumber++, &def);
if (OMX_EmptyThisBuffer(ILC_GET_HANDLE(video_encode), buf) != OMX_ErrorNone) {
printf("Error emptying buffer!\n");
}
out = ilclient_get_output_buffer(video_encode, 201, 1);
if (out != NULL) {
if (out->nFlags & OMX_BUFFERFLAG_CODECCONFIG) {
int i;
for (i = 0; i < out->nFilledLen; i++)
printf("%x ", out->pBuffer[i]);
printf("\n");
}
r = fwrite(out->pBuffer, 1, out->nFilledLen, outf);
if (r != out->nFilledLen) {
printf("fwrite: Error emptying buffer: %d!\n", r);
} else {
printf("Writing frame %d/%d, len %u\n", framenumber, NUMFRAMES, out->nFilledLen);
}
out->nFilledLen = 0;
} else {
printf("Not getting it :(\n");
}
r = OMX_FillThisBuffer(ILC_GET_HANDLE(video_encode), out);
if (r != OMX_ErrorNone) {
printf("Error sending buffer for filling: %x\n", r);
}
}
}
while (framenumber < NUMFRAMES);
fclose(outf);
- OMX_BUFFERHEADERTYPE ilclient_get_input_buffer(COMPONENT_T *comp, int portIndex, int block) The ilclient_get_input_buffer() function returns a buffer that was sent to an input port and that has been returned from a component using the OMX_EmptyBufferDone callback. If block non-zero, the function will block until a buffer is available. A pointer to buffer will be returned if available, otherwise NULL.
- ilclient_get_output_buffer(COMPONENT_T *comp, int portIndex, int block) The ilclient_get_output_buffer() function returns a buffer that was sent to an output port and that has been returned from a component using the OMX_FillBufferDone callback.
Chapter 11: OpenMAX Buffers on the Raspberry Pi
Each component has ports, and each port has a number of buffers.
When a component is created, it has an array of pointers to buffer headers but doesn’t actually have any memory allocated either for the headers or for the buffers. There are 2 eays to allocate buffers:
- OMX_AllocateBuffer()
#define OMX_AllocateBuffer
( hComponent, pBuffer, nPortIndex, pAppPrivate, nSizeBytes ) \
((OMX_COMPONENTTYPE*) hComponent)->AllocateBuffer
( hComponent, ppBuffer, nPortIndex, pAppPrivate, nSizeBytes) pAppPrivate,nSizeBytes, pBuffer)
The OMX_AllocateBuffer macro will request that the component allocate a new buffer and buffer header. The component will allocate the buffer and the buffer header and return a pointer to the buffer header.
for (i = 0; i < portDef->nBufferCount; i++) {
OMX_AllocateBuffer(hComp, &portDef->pBufferHdr[i], portDef->nPortIndex, portDef, portDef->nBufferSize);
}
For ex., IL client asks component to allocate buffers:
for (i=0;i<pClient->nBufferCount;i++) {
OMX_AllocateBuffer(hComp,
&pClient->pBufferHdr[i],
pClient->nPortIndex,
pClient,
pClient->nBufferSize);
}
OpenMAX IL defines two means of data communication:
- Non-tunneled communication a port exchanges data only with the IL client
- Tunneled communication a port exchanges data directly with a port on another component
Non-tunneled Data Flow, 3.4.2.1
The IL client is entirely responsible for moving data buffers among components if data tunneling is not used.
An IL client that has a data buffer to deliver to a component input port shall issue an OMX_EmptyThisBuffer() call.
For the component output port, the IL client shall initially provide one or more empty buffers into which the component can write output data; the OMX_FillThisBuffer() call accomplishes this task. As soon as one buffer is available from the component output port, the component shall send an FillBufferDone() callback. At this point, your application will take the processed data and do something with it.
Tunneled Initialization, 3.4.1.2
To avoid moving data buffers back and forth among the IL client and OpenMAX IL components, data tunnels can be set up so that the output buffer of one component is passed directly to the input port of the next component in the chain.
The tunneling setup and initialization based on the following steps:
- The components are constructed with the calls to OMX_GetHandle.
- The components are tunneled, linking an output port of the first component to an input port of the second component. The port that shall supply the buffer is decided in this phase.
- The IL client may override the input ports’ choice of buffer supplier after OMX_SetupTunnel has completed by setting the buffer supplier into the input port, which in turn will reprogram the supplier to the output port.
Tunneled Data Flow, 3.4.2.2
In data tunneling, OpenMAX IL components directly pass data buffers among themselves without returning them to the IL client.
Writing to and Reading from Buffers
The following 2 processes should be done concurrently by the client:
- whenever there is an empty input buffer: put data into the input buffer ( get by ilclient_get_input_buffer() ) then call OMX_EmptyThisBuffer() to ask the component to process it
- Whenever an output buffer is full: process the buffered output ( get by ilclient_get_output_buffer(), data is received by the FillBufferDone callback ) data then call OMX_FillThisBuffer() to inform the component
The pseudo code:
create component in loaded state with all ports disabled
move the component to idle state
enable input ports (creating input port buffers)
enable output ports (creating output port buffers)
move the component to executing state
while there is more input data:
if there is an empty input buffer
put data into the input buffer
call EmptyThis Buffer
if there is a full output buffer
process its data
call FillThisBuffer
while there is another full output buffer:
process its data
call FillThisBuffer
EOS Flag
The component needs to know when there is no input data so that it sends a signal to the client that there are no more output buffers.
The client signals this by setting the flag OMX_BUFFERFLAG_EOS in the flags field of the buffer header.
Typical code to read data from a file into input buffers, setting EOS on completion,
OMX_ERRORTYPE read_into_buffer_and_empty(FILE *fp, COMPONENT_T *component, OMX_BUFFERHEADERTYPE *buff_header, int *toread) {
OMX_ERRORTYPE r;
int buff_size = buff_header->nAllocLen;
int nread = fread(buff_header->pBuffer, 1, buff_size, fp);
printf("Read %d\n", nread);
buff_header->nFilledLen = nread;
*toread -= nread;
if (*toread <= 0) {
printf("Setting EOS on input\n");
buff_header->nFlags |= OMX_BUFFERFLAG_EOS;
}
r = OMX_EmptyThisBuffer(ilclient_get_handle(component), buff_header);
if (r != OMX_ErrorNone) {
fprintf(stderr, "Empty buffer error %s\n", err2str(r));
}
return r;
}
When getting information from the output buffers, the client will be watching for that flag,
OMX_ERRORTYPE save_info_from_filled_buffer(COMPONENT_T *component, OMX_BUFFERHEADERTYPE * buff_header) {
OMX_ERRORTYPE r;
printf("Got a filled buffer with %d, allocated %d\n", buff_header->nFilledLen, buff_header->nAllocLen);
// do something here, like save the data - do nothing this time
// quit if no more data coming
if (buff_header->nFlags & OMX_BUFFERFLAG_EOS) {
printf("Got EOS on output\n");
exit(0);
}
// otherwise refill it
r = OMX_FillThisBuffer(ilclient_get_handle(component), buff_header);
if (r != OMX_ErrorNone) {
fprintf(stderr, "Fill buffer error %s\n", err2str(r));
}
return r;
}
Dynamic Port Reconfiguration,3.4.5
You may not know some information before you can configure ports correctly. These information can be got after the component processed it. For ex., the size of the buffer for the output port.
When the OpenMAX IL component detects the setting needed to be changed, it shall notify the IL client by generating the OMX_PortSettingsChanged event.
As soon as the IL client receives this callback, it shall disable the related port then read the new port settings with OMX_GetConfig().
After the setting is corrected and necessary resource is allocated according to the setting, related ports shall also be set up with OMX_SetConfig().
The IL Client library has two relevant functions to receive OMX_PortSettingsChanged events:
- ilclient_remove_event(COMPONENT_T *st, OMX_EVENTTYPE eEvent, OMX_U32 nData1, int ignore1, OMX_IN OMX_U32 nData2, int ignore2) The ilclient_remove_event() function queries the event list for the given component, matching against the given criteria. If a matching event is found, the event is removed and added to the free event list.
- ilclient_wait_for_event(COMPONENT_T *comp, OMX_EVENTTYPE event, OMX_U32 nData1, int ignore1, OMX_IN OMX_U32 nData2, int ignore2, int event_flag, int suspend) The ilclient_wait_for_event() function is similar to ilclient_remove_event(), but allows the caller to block until that event arrives.
Chapter 12: Image Processing on the Raspberry Pi
OMX_IMAGE_PORTDEFINITIONTYPE (4.4.3) is the data structure that is used to define an image path.
The OMX_IMAGE_PORTDEFINITIONTYPE structure is included as part of the OMX_PARAM_PORTDEFINITIONTYPE structure
typedef struct OMX_PARAM_PORTDEFINITIONTYPE {
...
union {
OMX_AUDIO_PORTDEFINITIONTYPE audio;
OMX_VIDEO_PORTDEFINITIONTYPE video;
OMX_IMAGE_PORTDEFINITIONTYPE image;
OMX_OTHER_PORTDEFINITIONTYPE other;
} format;
...
} OMX_PARAM_PORTDEFINITIONTYPE;
typedef struct OMX_IMAGE_PORTDEFINITIONTYPE {
OMX_STRING cMIMEType;
OMX_NATIVE_DEVICETYPE pNativeRender;
OMX_U32 nFrameWidth;
OMX_U32 nFrameHeight;
OMX_S32 nStride;
OMX_U32 nSliceHeight;
OMX_BOOL bFlagErrorConcealment;
OMX_IMAGE_CODINGTYPE eCompressionFormat;
OMX_COLOR_FORMATTYPE eColorFormat;
OMX_NATIVE_WINDOWSTYPE pNativeWindow;
} OMX_IMAGE_PORTDEFINITIONTYPE;
OMX_IMAGE_PARAM_PORTFORMATTYPE (4.4.4) is used to enumerate the various data input/output format supported by the port.
typedef struct OMX_IMAGE_PARAM_PORTFORMATTYPE {
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPortIndex;
OMX_U32 nIndex;
OMX_IMAGE_CODINGTYPE eCompressionFormat;
OMX_COLOR_FORMATTYPE eColorFormat;
} OMX_IMAGE_PARAM_PORTFORMATTYPE;
- eCompressionFormat This is the enumeration describing the compression format used on the port. Table 4-66 shows the supported image compression formats. When OMX_IMAGE_CodingUnused is specified, the eColorFormat field is valid. On the RPi, the only value given is OMX_IMAGE_CodingUnused .
- eColorFormat This is the decompressed color format used for the port. This field is valid only when the eCompressionFormat field is set to OMX_IMAGE_CodingUnused. For enumerations on OMX_COLOR_FORMATTYPE, see section 4.2. /opt/vc/include/IL/OMX_IVCommon.h defines it.
The color format will be set to match that emitted by the input image format.
JPEG Encoding
image_encode takes raw images in various formats ranging form 16/24/32-bit RGB with Alpha formats to variants of YUV and CrCbY. The output can be any of the supported compressed image formats
You basically only have to configure the input and output port and you are ready to go and throw data on it.
The basic steps:
- Get handle with callbacks for EmptyBufferDone and FillBufferDone.
- Disable ports 340 and 341.
- Switch handle to idle.
- Configure, enable and allocate bufferX for port 340.
- Configure, enable and allocate bufferY for port 341.
- Switch handle to executing.
- Loop over the data:
- Copy compressed data from bufferY.
- Fill bufferY.
- Copy your image data to bufferX.
- Empty bufferX.
- Switch handle to idle.
- Disable and deallocate bufferX for port 340.
- Disable and deallocate bufferY for port 341.
- Switch handle to loaded.
- Free handle.
Chapter 13: OpenMAX Video Processing on the Raspberry Pi
Video Basics
About YUV Video
YUV refers to a family of color spaces, all of which encode brightness information separately from color information.
In fact, YUV almost always refers to one particular color space named Y'CbCr. However, YUV is currently often used as a general term for any color space that works along the same principles as Y'CbCr.
The Y' component, also called luma, represents the brightness value of the color. luma is different from luminance, which is designated Y.
Luminance is derived from linear RGB values, whereas luma is derived from non-linear (gamma-corrected) RGB values.
Luma is derived from an RGB color:
- SDTV(ITU-R BT.601)
Y' = 0.299R + 0.587G + 0.114B
Y' = 0.2125R + 0.7154G + 0.0721B
The U and V components, also called chroma values or color difference values, are derived by subtracting the Y value from the red and blue components of the original RGB color:
U = B - Y'
V = R - Y'
YUV contains enough information to recover the original RGB value.Black-and-white televisions ignore the chroma and display the combined signal as a grayscale image.
Y'CbCr: YUV in Computer Video
Pb = (0.5 / (1 - 0.114)) × (B - Y')
Pr = (0.5 / (1 - 0.299)) × (R - Y')
Y' = 16 + 219 × Y'
Cb = 128 + 224 × Pb
Cr = 128 + 224 × Pr
These scaling and offset factors produce the range of values listed in the following table:| Component | Range |
| Y' | 16–235 |
| Cb/Cr | 16–240, with 128 representing zero |
YUV Formats
- 緊縮格式（packed formats）：將Y、U、V值儲存成Macro Pixels，和RGB的存放方式類似。 In packed formats, you usually have only one plane, with all the luma and chroma data interleaved. This is similar to RGB pixel formats. Packet formats cannot normally deal with vertical sub-sampling. Otherwise scan lines would have different sizes. So generally, packed formats are horizontally subsampled, especially by a factor of 2 (i.e., YUV 4:2:2).
- 平面格式（planar formats）：將Y、U、V的三個分量分別存放在不同的矩陣中。 Planar formats use separate matrices for each of the 3 color components. In other words, there is one table of luminance pixel values, and two separate tables for the chrominance components. This segregated representation in memory of pixels is more convenient for video coding. Y份量，U份量和V份量都是以獨立的平面組織的，也就是說所有的U份量必須在Y分量後面，而V份量在所有的U份量後面
YUV Sub-sampling
為節省頻寬起見，大多數YUV格式平均使用的每像素位數都少於24位元。主要的抽樣（subsample）格式有YCbCr 4:2:0、YCbCr 4:2:2、YCbCr 4:1:1和YCbCr 4:4:4。
- 4:4:4表示完全取樣。
- 4:2:2表示2:1的水平取樣，垂直完全採樣。
- 4:2:0表示2:1的水平取樣，垂直2：1採樣。 DVD-Video是以YUV 4:2:0的方式採樣記錄，也就是我們俗稱的I420
- 4:1:1表示4:1的水平取樣，垂直完全採樣。
I420 planar formats
Most video decoders output raw pictures in I420 format. They comprise an [n , m] Y plane followed by [(n/2), (m/2)] U and V planes. Image Stride
When a video image is stored in memory, the memory buffer might contain extra padding bytes after each row of pixels. The padding bytes affect how the image is stored in memory, but do not affect how the image is displayed. The stride is the number of bytes from one row of pixels in memory to the next row of pixels in memory. Stride is also called pitch. Generate a Video image
Generate an video image changing with frame number in OMX_COLOR_FormatYUV420PackedPlanar format:
/*
A video frame is divided into multiple slices which are to be saved in buffers.
nStride
+------------------+
| slice#1 | nSliceHeight
+------------------+
| |
| |
+------------------+
Each buffer contains a slice of the Y, U, and V planes.
buf_start
+-----------------------------+
| Y |
| |
+--------------+--------------+
| U | V |
+--------------+--------------+
Fetch the output buffer until a complete frame is read
*/
def->format.video.nFrameWidth = WIDTH;
def->format.video.nFrameHeight = HEIGHT;
def->format.video.xFramerate = 30 << 16;
def->format.video.nSliceHeight = ALIGN_UP(def.format.video.nFrameHeight, 16);
def->format.video.nStride = def.format.video.nFrameWidth;
def->format.video.eColorFormat = OMX_COLOR_FormatYUV420PackedPlanar;
int frame=0;
OMX_PARAM_PORTDEFINITIONTYPE *def;
OMX_U32 * filledLen;
int i, j;
OMX_VIDEO_PORTDEFINITIONTYPE *vid = &def->format.video;
// locate the starting position of each plane in Y'CrCb
char *y = buf;
char *u = y + vid->nStride * vid->nSliceHeight;
char *v = u + (vid->nStride >> 1) * (vid->nSliceHeight >> 1);
for (j = 0; j < vid->nFrameHeight / 2; j++) {
// move the starting position for each scan
char *py = y + 2 * j * vid->nStride;
char *pu = u + j * (vid->nStride >> 1);
char *pv = v + j * (vid->nStride >> 1);
for (i = 0; i < vid->nFrameWidth / 2; i++) {
int z = (((i + frame) >> 4) ^ ((j + frame) >> 4)) & 15;
py[0] = py[1] = py[vid->nStride] = py[vid->nStride + 1] = 0x80 + z * 0x8;
pu[0] = 0x00 + z * 0x10;
pv[0] = 0x80 + z * 0x30;
py += 2;
pu++;
pv++;
}
}
*filledLen = (vid->nStride * vid->nSliceHeight * 3) >> 1;
frame++;
Video Use Case Examples, 4.3.3
Figure 4-4 depicts one possible set of components as well as the tunneling of ports for these components to implement a H.263 video encoding scheme. This use case encodes raw video into H.263 format and writes it to a file while previewing the captured video on a display.
Chapter 14: OpenMAX Audio on the Raspberry Pi
Chapter 15: Rendering OpenMAX to OpenGL on the Raspberry Pi
Chapter 16: Playing Multimedia Files on the Raspberry Pi
Chapter 17: Basic OpenVG on the Raspberry Pi
Chapter 18: Text Processing in OpenVG on the Raspberry Pi
Chapter 19: Overlays on the Raspberry Pi
VMCS-X OpenMAX IL Components
Video Domain Components
OMX.broadcom.camera
| camera |
|
- port#73 This port supports only the clock format type. This is a clock input port for correctly assigning timestamps to frames.
- port#70 The preview port, it will emit images whenever the component is in the executing state and the port is enabled.
- port#71 The video capture port, it will emit images only when the capturing configuration parameter OMX_IndexConfigPortCapturing is set for that port, and either the clock port is disabled or the connected clock is in the Running state. When a connected clock is transitioned to stop the component will stop recording data at the end of the current image, and then transmit a packet with no data but with the EOS flag set.
- port#72 The still image capture port, it will only capture an image when the port is executing and OMX_IndexConfigPortCapturing is set. If the bOneShot flag is set, an EOS will be emited from the port after each complete capture.
- OMX_COLOR_FormatYUV420PackedPlanar This format differs from OMX_COLOR_FormatYUV420Planar in that each slice of data shall contain a plane of Y, U, and V data in this order, whereas the OMX_COLOR_FormatYUV420Planarr format transfers each plane in its entirety. (planar的YUV格式，先連續儲存所有畫素點的Y，緊接著儲存所有畫素點的U，隨後是所有畫素點的V。Y，U，V分別用三個獨立的陣列表示。)
- OMX_COLOR_FormatYUV420PackedSemiPlanar
- OMX_COLOR_FormatYUV422PackedPlanar
- Create component.
- Use OMX_IndexConfigRequestCallback to request callbacks on OMX_IndexParamCameraDeviceNumber.
- Set OMX_IndexParamISPTunerName.
- Set OMX_IndexParamCameraFlashType.
- Set OMX_IndexParamCameraDeviceNumber.
- Wait for the callback that OMX_IndexParamCameraDeviceNumber has changed. At this point, all the drivers have been loaded. Other settings can be applied whilst waiting for this event.
- Query for OMX_IndexConfigCameraSensorModes as required.
- Change state to IDLE, and proceed as required.
- component
- OMX_IndexParamCameraDeviceNumber This index uses the standard IL structure OMX_PARAM_U32TYPE . When this is set, the value will be checked to ensure that the camera number specified is defined for the platform. Setting this parameter also triggers loading of the relevant drivers to speed up queries for OMX_IndexConfigCameraSensorModes, which require the drivers to be loaded. This parameter is supported by OMX_IndexConfigRequestCallback, so the client can be notified once that driver is loaded, rather than having to poll the settingss.
- OMX_IndexConfigRequestCallback This config implements IL416c to allow clients to request notification of when a config or parameter is changed. Enable config change notifications. When the parameter specified in nIndex for port nPortIndex changes, an OMX_EventParamOrConfigChanged event is generated for the client. OMX_Broadcom.h:
typedef struct OMX_CONFIG_REQUESTCALLBACKTYPE
{
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPortIndex;
OMX_INDEXTYPE nIndex;
OMX_BOOL bEnable;
} OMX_CONFIG_REQUESTCALLBACKTYPE;
typedef struct OMX_PARAM_CAMERAISPTUNERTYPE {
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U8 tuner_name[64];
} OMX_PARAM_CAMERAISPTUNERTYPE;
typedef struct OMX_OTHER_PARAM_PORTFORMATTYPE {
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPortIndex;
OMX_U32 nIndex; /** Indicates the enumeration index for the format from 0x0 to N-1 */
OMX_OTHER_FORMATTYPE eFormat; /** Type of data expected for this channel */
} OMX_OTHER_PARAM_PORTFORMATTYPE;
- OMX_IndexConfigPortCapturing Query / set whether images are to be emitted on the specified port or not. Changing to enabling capture when the component is not executing will reset the internal counter used for time stamping frames when not connected to a clock component.
typedef struct OMX_CONFIG_PORTBOOLEANTYPE{
OMX_U32 nSize;
OMX_VERSIONTYPE nVersion;
OMX_U32 nPortIndex;
OMX_BOOL bEnabled;
} OMX_CONFIG_PORTBOOLEANTYPE;
Image Domain Components
OMX.broadcom.image_encode
| image_encode |
|
OMX.broadcom.image_write
| image_write |
Raspberry Pi Camera Module
There are four applications provided: raspistill, raspivid, raspiyuv and raspividyuv. raspistill and raspiyuv are very similar and are intended for capturing images; raspivid and raspvidyuv are for capturing video. All the applications are driven from the command line, and written to take advantage of the MMAL API which runs over OpenMAX. The MMAL API provides an easier to use system than that presented by OpenMAX. Note that MMAL is a Broadcom-specific API used only on Videocore 4 systems. The applications use up to four OpenMAX (MMAL) components: camera, preview, encoder, and null_sink.Why to use MMAL: Good luck with OMX. I've been working on and off with it for 5 years, and still have trouble. It really isn't very good. The spec document is awful to learn from (It's a spec not a manual) and the code is a nightmare to debug. Fair point on MMAL not being docuemnted per se - but note it does have full Doxygen comments in the ARM side source code, which is published on the web somewhere.
Cobalt
Set up Cobalt environment
The Cobalt Authors originally maintained a port of Chromium called H5VCC, the HTML5 Video Container for Consoles, ported to each of the major game consoles was a dangerous task. Cobalt is a lightweight application container (i.e. an application runtime, like a JVM or the Flash Player) that is compatible with a subset of the W3C HTML5 specifications. If you author a single-page web application (SPA) that complies with the Cobalt Subset of W3C standards, it will run as well as possible on all the devices that Cobalt supports. These instructions explain how to set up Cobalt for your workstation and your Raspberry Pi device.Set up your environment - Linux
These instructions explain how Linux users set up their Cobalt development environment,- clone the tools used by Cobalt
cd ~/
install -d cobalt
cd cobalt
git clone https://cobalt.googlesource.com/depot_tools
export PATH=${PATH}:~/cobalt/depot_tools
sudo apt-get install build-essential coreutils git gperf \
libasound2-dev libavformat-dev libavresample-dev \
libdirectfb-dev libdirectfb-extra libpulse-dev \
libgl1-mesa-dev libgles2-mesa-dev libx11-dev \
libxcomposite-dev libxcomposite1 libxrender-dev libxrender1 \
libxpm-dev m4 ruby tar xserver-xephyr xz-utils yasm
sudo apt-get install ruby
sudo apt-get install libstdc++-4.8-dev
- For old Cobalt, use bison-2.7.1
- For new Cobalt, use bison-3.0.5
sudo apt-get remove bison
cd /tmp
wget http://ftp.gnu.org/gnu/bison/bison-x.y.z.tar.gz
tar zxf bison-x.y.z.tar.gz
cd bison-x.y.z
sh configure && make && sudo make install
cd ~/cobalt
git clone https://cobalt.googlesource.com/cobalt
export PATH="~/cobalt/cobalt/src/third_party/llvm-build/Release+Asserts/bin:${PATH}"
cd ~/cobalt/cobalt/src
cobalt/build/gyp_cobalt [-C <build_type>] <platform>
- To generate for Linux x64
cobalt/build/gyp_cobalt linux-x64x11
cobalt/build/gyp_cobalt raspi-1
cd ~/cobalt/cobalt/src
ninja -C out/<platform>_<build_type> <target_name>
| Type | Optimizations | Logging | Asserts | Debug Info | Console |
|---|---|---|---|---|---|
| debug | None | Full | Full | Full | Enabled |
| devel | Full | Full | Full | Full | Enabled |
| qa | Full | Limited | None | None | Enabled |
| gold | Full | None | None | None | Disabled |
- To build for Linux 64
ninja -C out/linux-x64x11_debug cobalt
ninja -C out/raspi-1_debug cobalt
out/<platform>_<build_type>/<target_name> --allow_http --ignore_certificate_errors [--url=<url>]
out/linux-x64x11_debug/cobalt --allow_http --ignore_certificate_errors
./cobalt --help
Options:
--video_playback_rate_multiplier
Specifies the multiplier of video playback rate. Set to a value greater than 1.0 to play video faster. Set to a value less than 1.0 to play video slower.
--viewport
Specifies the viewport size: width ['x' height]
--version
Prints the current version of Cobalt
--fallback_splash_screen_url
Setting this switch defines the splash screen URL that Cobalt will use in absence of a web cache. The referenced url should be a content file (for example file:///foobar.html) or an embedded file (for example h5vcc-embedded://foobar.html) and all files referenced must be content or embedded files as well. If none is passed (case-insensitive), no splash screen will be constructed. If no value is set, the URL in gyp_configuration.gypi or base.gypi will be used.
--software_surface_cache_size_in_bytes
Only relevant if you are using the Blitter API. Determines the capacity of the software surface cache, which is used to cache all surfaces that are rendered via a software rasterizer to avoid re-rendering them.
--skia_atlas_texture_dimensions
Specifies the dimensions of the Skia caching texture atlases (e.g. 2048x2048).
--skia_cache_size_in_bytes
Determines the capacity of the skia cache. The Skia cache is maintained within Skia and is used to cache the results of complicated effects such as shadows, so that Skia draw calls that are used repeatedly across frames can be cached into surfaces. This setting is only relevant when using the hardware-accelerated Skia rasterizer. While it depends on the platform, this setting may affect GPU memory usage.
--scratch_surface_cache_size_in_bytes
Determines the capacity of the scratch surface cache. The scratch surface cache facilitates the reuse of temporary offscreen surfaces within a single frame. This setting is only relevant when using the hardware-accelerated Skia rasterizer. While it depends on the platform, this setting may affect GPU memory usage.
--retain_remote_typeface_cache_during_suspend
Causes the remote typeface cache to be retained when Cobalt is suspended, so that they don't need to be re-downloaded when Cobalt is resumed.
--remote_typeface_cache_size_in_bytes
Determines the capacity of the remote typefaces cache which manages all typefaces downloaded from a web page.
--reduce_gpu_memory_by
Reduces the gpu-memory of the system by this amount. This causes AutoMem to reduce the runtime size of the GPU-Memory caches.
--reduce_cpu_memory_by
Reduces the cpu-memory of the system by this amount. This causes AutoMem to reduce the runtime size of the CPU-Memory caches.
--qr_code_overlay
Display QrCode based overlay information. These information can be used for performance tuning or playback quality check.
--offscreen_target_cache_size_in_bytes
Determines the amount of GPU memory the offscreen target atlases will use. This is specific to the direct-GLES rasterizer and caches any render tree nodes which require skia for rendering. Two atlases will be allocated from this memory or multiple atlases of the frame size if the limit allows. It is recommended that enough memory be reserved for two RGBA atlases about a quarter of the frame size.
--max_cobalt_gpu_usage
Specifies the maximum GPU usage of the cobalt.
--max_cobalt_cpu_usage
Specifies the maximum CPU usage of the cobalt.
--local_storage_partition_url
Overrides the default storage partition with a custom partition URL to use for local storage. The provided URL is canonicalized.
--javascript_gc_threshold_in_bytes
Specifies the javascript gc threshold. When this amount of garbage has collected then the garbage collector will begin running.
--url
Setting this switch defines the startup URL that Cobalt will use. If no value is set, a default URL will be used.
--image_cache_size_in_bytes
Determines the capacity of the image cache which manages image surfaces300 downloaded from a web page. While it depends on the platform, often (and ideally) these images are cached within GPU memory.
--help
Prints help information of cobalt command
--fps_overlay
If toggled, framerate statistics will be displayed in an on-screen overlay and updated after each animation completes, or after a maximum number of frames has been collected.
--fps_stdout
If toggled, framerate statistics will be printed to stdout after each animation completes, or after a maximum number of frames has been collected.
--force_migration_for_storage_partitioning
Overrides the default storage migration policy when upgrading to partitioned storage and forces data migration regardless of theinitial app url. The default policy is to migrate data only forhttps://www.youtube.com/tv.
--enable_map_to_mesh_rectangular
If toggled and map-to-mesh is supported on this platform, this allows it to accept the 'rectangular' keyword. Useful to get rectangular stereo video on platforms that do not support stereoscopy natively, letting the client apply a stereo mesh projection (one that differs for each eye).
--disable_timer_resolution_limit
By default, window.performance.now() will return values at a clamped minimum resolution of 20us. By specifying this flag, the limit will be removed and the resolution will be 1us (or larger depending on the platform.
--disable_javascript_jit
Specifies that javascript jit should be disabled.
Set up your environment - Raspberry Pi
- Set up your Pi Update the package configuration on your Raspberry Pi so that Cobalt can run properly
sudo apt-get remove -y --purge --auto-remove libgl1-mesa-dev libegl1-mesa-dev libgles2-mesa libgles2-mesa-dev
sudo apt-get install -y libpulse-dev libasound2-dev libavformat-dev libavresample-dev
export PATH=${PATH}:~/cobalt/depot_tools
export PATH="~/cobalt/cobalt/src/third_party/llvm-build/Release+Asserts/bin:${PATH}"
cd ~/cobalt/cobalt/src
export RASPI_HOME=~/pi
install -d $RASPI_HOME/sysroot
cd ~
install -d pi/sysroot
rsync -avzh --safe-links \
--delete-after pi@$RASPI_ADDR:/{opt,lib,usr} \
--exclude="lib/firmware" --exclude="lib/modules" \
--include="usr/lib" --include="usr/include" \
--include="usr/local/include" --include="usr/local/lib" \
--exclude="usr/*" --include="opt/vc" --exclude="opt/*" \
~/pi/sysroot
- --delete-before receiver deletes before transfer (sync) files
- --delete-after receiver deletes after transfer (sync) files
cobalt/build/gyp_cobalt raspi-2
ninja -C out/raspi-2_debug cobalt
- Build host
- Pi
- ~/pi/sysroot/usr/lib/arm-linux-gnueabihf All were built with GLIBCXX_3.4.21
- ~/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/arm-linux-gnueabihf/lib/libstdc++.so Support until GLIBCXX_3.4.19
...
In file included from ../../starboard/types.h:33:0,
from ../../starboard/system.h:25,
from ../../starboard/log.h:33,
from ../../starboard/client_porting/poem/assert_poem.h:22,
from ../../third_party/libwebp/src/utils/bit_reader_utils.c:19:
/home/jerry/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/lib/gcc/arm-linux-gnueabihf/4.8.3/include/stdint.h:9:26: fatal error: stdint.h: No such file or directory
# include_next <stdint.h>
ninja: Entering directory `out/raspi-2_debug'
[1/1] LINK cobalt
FAILED: /home/jerry/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-g++ @cobalt.rsp
collect2: error: ld terminated with signal 9 [Killed]
ninja: build stopped: subcommand failed.
^
[4817/5744] CXX obj/cobalt/renderer/ra...dware_rasterizer.hardware_rasterizer.o
../../cobalt/renderer/rasterizer/skia/hardware_rasterizer.cc: In constructor ‘cobalt::renderer::rasterizer::skia::HardwareRasterizer::Impl::Impl(cobalt::renderer::backend::GraphicsContext*, int, int, int, int, bool, bool)’:
../../cobalt/renderer/rasterizer/skia/hardware_rasterizer.cc:589:65: warning: passing NULL to non-pointer argument 2 of ‘static GrContext* GrContext::Create(GrBackend, GrBackendContext, const GrContextOptions&)’ [-Wconversion-null]
GrContext::Create(kOpenGL_GrBackend, NULL, context_options));
^
[5744/5744] LINK cobalt
FAILED: /home/jerry/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-g++ @cobalt.rsp
/home/jerry/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/../lib/gcc/arm-linux-gnueabihf/4.8.3/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lEGL
/home/jerry/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/../lib/gcc/arm-linux-gnueabihf/4.8.3/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lGLESv2
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
-lbrcmEGL -lbrcmGLESv2
cd ~/pi/sysroot/usr/lib/arm-linux-gnueabihf
sudo mv libEGL.so.1.0.0 libEGL.so.1.0.0_backup
sudo mv libGLESv2.so.2.0.0 libGLESv2.so.2.0.0_backup
sudo ln -s ../../../opt/vc/lib/libEGL.so libEGL.so.1.0.0
sudo ln -s ../../../opt/vc/lib/libGLESv2.so libGLESv2.so.2.0.0
cd ~/pi/sysroot/opt/vc/lib
sudo ln -s libbrcmEGL.so libEGL.so
sudo ln -s libEGL.so libEGL.so.1
sudo ln -s libbrcmGLESv2.so libGLESv2.so
sudo ln -s libGLESv2.so libGLESv2.so.2
...
/home/jerry/pi/sysroot/usr/lib/arm-linux-gnueabihf/libopenmpt.so.0: undefined reference to `VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21'
/home/jerry/pi/sysroot/usr/lib/arm-linux-gnueabihf/libopenmpt.so.0: undefined reference to `std::range_error::range_error(char const*)@GLIBCXX_3.4.21'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
`std::range_error::range_error(char const*)@GLIBCXX_3.4.21'
$ /home/jerry/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-g++ -v
Using built-in specs.
COLLECT_GCC=/home/jerry/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-g++
COLLECT_LTO_WRAPPER=/home/jerry/pi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/../libexec/gcc/arm-linux-gnueabihf/4.8.3/lto-wrapper
Target: arm-linux-gnueabihf
...
gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1+bzr2650 - Linaro GCC 2014.03)
pi@raspberrypi:~ $ gcc --version
gcc (Raspbian 6.3.0-18+rpi1+deb9u1) 6.3.0 20170516
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
pi@raspberrypi:~ $ g++ --version
g++ (Raspbian 6.3.0-18+rpi1+deb9u1) 6.3.0 20170516
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
pi@raspberrypi:~ $ ldd --version
ldd (Debian GLIBC 2.24-11+deb9u3) 2.24
cd ~/pi/tools/arm-bcm2708
wget https://releases.linaro.org/components/toolchain/binaries/6.3-2017.05/arm-linux-gnueabihf/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf.tar.xz
tar -xJf gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf.tar.xz
toolchain = os.path.realpath(
os.path.join(
raspi_home,
'tools/arm-bcm2708/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf'))
# 'tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64'))
cobalt/build/gyp_cobalt raspi-2
ninja -C out/raspi-2_devel cobalt
#0 install required packages
sudo apt-get install g++ make gawk
#1 create a working folder in your home:
cd ~
mkdir gcc_all && cd gcc_all
#2 download the source code,
# we do need the kernel header files in order to build the target system’s standard C library.
wget https://ftpmirror.gnu.org/binutils/binutils-2.28.tar.bz2
wget https://ftpmirror.gnu.org/gcc/gcc-6.3.0/gcc-6.3.0.tar.gz
wget https://ftpmirror.gnu.org/glibc/glibc-2.24.tar.bz2
wget https://ftpmirror.gnu.org/gcc/gcc-8.1.0/gcc-8.1.0.tar.gz
git clone --depth=1 https://github.com/raspberrypi/linux
#3 extract the archives and erase them:
tar xf binutils-2.28.tar.bz2
tar xf glibc-2.24.tar.bz2
tar xf gcc-6.3.0.tar.gz
tar xf gcc-8.1.0.tar.gz
rm *.tar.*
#4 some prerequisites needed by GCC
cd gcc-6.3.0
contrib/download_prerequisites
rm *.tar.*
cd ..
cd gcc-8.1.0
contrib/download_prerequisites
rm *.tar.*
#5 create a folder in which we’ll put the cross compiler and add it to the path
cd ..
sudo mkdir -p /opt/cross-pi-gcc
sudo chown $USER /opt/cross-pi-gcc
export PATH=/opt/cross-pi-gcc/bin:$PATH
#6 Copy the kernel headers in the above folder
cd linux
KERNEL=kernel7
make ARCH=arm INSTALL_HDR_PATH=/opt/cross-pi-gcc/arm-linux-gnueabihf headers_install
#7 build Binutils
cd ..
mkdir build-binutils && cd build-binutils
../binutils-2.28/configure --prefix=/opt/cross-pi-gcc --target=arm-linux-gnueabihf --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-multilib
make -j 8
make install
#8 change a line in ubsan.c from gcc-6.3.0/gcc/ , find line 1474:
|| xloc.file == '\0' || xloc.file[0] == '\xff'
with
|| xloc.file[0] == '\0' || xloc.file[0] == '\xff'
#8 a partial build of Glibc
cd ..
mkdir build-gcc && cd build-gcc
../gcc-6.3.0/configure --prefix=/opt/cross-pi-gcc --target=arm-linux-gnueabihf --enable-languages=c,c++,fortran --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-multilib
make -j8 all-gcc
make install-gcc
#9 partially build Glibc , install Glibc’s standard C library headers to:
cd ..
mkdir build-glibc && cd build-glibc
../glibc-2.24/configure --prefix=/opt/cross-pi-gcc/arm-linux-gnueabihf --build=$MACHTYPE --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf --with-arch=armv6 --with-fpu=vfp --with-float=hard --with-headers=/opt/cross-pi-gcc/arm-linux-gnueabihf/include --disable-multilib libc_cv_forced_unwind=yes
make install-bootstrap-headers=yes install-headers
make -j8 csu/subdir_lib
install csu/crt1.o csu/crti.o csu/crtn.o /opt/cross-pi-gcc/arm-linux-gnueabihf/lib
arm-linux-gnueabihf-gcc -nostdlib -nostartfiles -shared -x c /dev/null -o /opt/cross-pi-gcc/arm-linux-gnueabihf/lib/libc.so
touch /opt/cross-pi-gcc/arm-linux-gnueabihf/include/gnu/stubs.h
#10 Back to GCC with 2nd build
cd ..
cd build-gcc
make -j8 all-target-libgcc
make install-target-libgcc
#11 Finish building Glibc, the standard C library is installed to /opt/cross-pi-gcc/arm-linux-gnueabihf/lib:
cd ..
cd build-glibc
make -j8
make install
#12 Finish building GCC 6.3.0, the standard C++ library is installed to /opt/cross/arm-linux-gnueabihf/lib/ :
cd ..
cd build-gcc
make -j8
make install
cd ..
#13 Test
# test.cpp:
#include <iostream>
int main()
{
auto lambda = [](auto x){ return x; };
std::cout << lambda("Hello generic lambda!\n");
return 0;
}
arm-linux-gnueabihf-g++ test.cpp -o test -v
file test
#14 Make a backup to the working space:
sudo cp -r /opt/cross-pi-gcc /opt/arm-linux-gnueabihf-pi-6.3.0
###################################################################################################################
# Next, we are going to use the above built Glibc to build a more modern cross compiler that will overwrite gcc-6.3
###################################################################################################################
cd ..
mkdir build-gcc8 && cd build-gcc8
../gcc-8.1.0/configure --prefix=/opt/cross-pi-gcc --target=arm-linux-gnueabihf --enable-languages=c,c++,fortran --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-multilib
make -j8
make install
Or, to build a new GCC on the Pi device:
sudo apt-get install libgmp-dev libmpfr-dev libmpc-dev
wget https://ftp.gnu.org/gnu/gcc/gcc-6.3.0/gcc-6.3.0.tar.gz
tar xzf gcc-6.3.0.tar.gz
cd gcc-6.3.0/
contrib/download_prerequisites
mkdir ../gcc-build
cd ../gcc-build
../gcc-6.3.0/configure -v \
# for Pi3 and new Pi2
../configure -v --enable-languages=c,c++ --with-cpu=cortex-a53 \
--with-fpu=neon-fp-armv8 --with-float=hard --build=arm-linux-gnueabihf \
--host=arm-linux-gnueabihf --target=arm-linux-gnueabihf --enable-checking=release
# ARM 64-bit
#../configure --enable-languages=c,c++ --with-cpu=cortex-a53 --enable-checking=release
# x86_64
#../configure --disable-multilib --enable-languages=c,c++ --enable-checking=release
# Pi Zero (best use -j2 or less)
#../configure -v --enable-languages=c,c++ --with-cpu=arm1176jzf-s \
# --with-fpu=vfp --with-float=hard --build=arm-linux-gnueabihf \
# --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf --enable-checking=release
# Old Pi2, the Pi3 config should work for the new Pi2
#../configure -v --enable-languages=c,c++ --with-cpu=cortex-a7 \
# --with-fpu=neon-vfpv4 --with-float=hard --build=arm-linux-gnueabihf \
# --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf --enable-checking=release
sudo dd if=/dev/zero of=/swapfile1GB bs=1M count=1024
sudo chmod 0600 /swapfile1GB
sudo mkswap /swapfile1GB
sudo swapon /swapfile1GB
make -j5
# sudo make install
../../starboard/shared/ffmpeg/ffmpeg_common.h:20:32: fatal error: libavcodec/avcodec.h: No such file or directory
../../starboard/shared/alsa/alsa_util.h:18:28: fatal error: alsa/asoundlib.h: No such file or directory
'-I<(sysroot)/opt/vc/include/interface/vmcs_host/linux',
# and for alsa, ffmpeg
'-I<(sysroot)/usr/include',
'-I<(sysroot)/usr/include/arm-linux-gnueabihf',
],
'linker_flags': [
../../starboard/shared/libevent/socket_waiter_internal.cc:73:6: error: ‘void {anonymous}::GetSocketPipe(SbSocketPrivate**, SbSocketPrivate**)’ defined but not used [-Werror=unused-function]
void GetSocketPipe(SbSocket* client_socket, SbSocket* server_socket) {
^~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
../../starboard/shared/libevent/socket_waiter_internal.cc:41:17: error: ‘SbSocketAddress {anonymous}::GetIpv4Localhost()’ defined but not used [-Werror=unused-function]
SbSocketAddress GetIpv4Localhost() {
^~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
../../starboard/shared/libevent/socket_waiter_internal.cc:50:10: error: ‘SbSocketPrivate* {anonymous}::AcceptBySpinning(SbSocket, SbTime)’ defined but not used [-Werror=unused-function]
SbSocket AcceptBySpinning(SbSocket server_socket, SbTime timeout) {
^~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
../../starboard/shared/starboard/application.cc:41:6: error: ‘void starboard::shared::starboard::{anonymous}::Dispatch(SbEventType, void*, SbEventDataDestructor)’ defined but not used [-Werror=unused-function]
void Dispatch(SbEventType type, void* data, SbEventDataDestructor destructor) {
^~~~~~~~
cc1plus: all warnings being treated as errors
../../starboard/shared/signal/suspend_signals.cc:73:6: error: ‘void starboard::shared::signal::{anonymous}::Ignore(int)’ defined but not used [-Werror=unused-function]
void Ignore(int signal_id) {
^~~~~~
cc1plus: all warnings being treated as errors
../../base/logging.h:598:26: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
cobalt/css_parser/position_parse_structures.cc
cobalt/debug/debug_dispatcher.cc
cobalt/layout/used_style.cc
cobalt/media_capture/encoders/linear16_audio_encoder.cc
cobalt/media_capture/media_recorder.cc
cobalt/speech/audio_encoder_flac.cc
cobalt/websocket/web_socket_impl.cc
cobalt/xhr/xml_http_request.cc
[1/1] LINK cobalt
FAILED: /home/jerry/pi/tools/arm-bcm2708/cross-pi-gcc/bin/arm-linux-gnueabihf-g++ @cobalt.rsp
/home/jerry/pi/tools/arm-bcm2708/cross-pi-gcc/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lasound
/home/jerry/pi/tools/arm-bcm2708/cross-pi-gcc/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lavcodec
/home/jerry/pi/tools/arm-bcm2708/cross-pi-gcc/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lavformat
/home/jerry/pi/tools/arm-bcm2708/cross-pi-gcc/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find -lavutil
/home/jerry/pi/tools/arm-bcm2708/cross-pi-gcc/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find /opt/cross-pi-gcc/arm-linux-gnueabihf/lib/libpthread.so.0
/home/jerry/pi/tools/arm-bcm2708/cross-pi-gcc/bin/../lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../arm-linux-gnueabihf/bin/ld: cannot find /opt/cross-pi-gcc/arm-linux-gnueabihf/lib/libpthread_nonshared.a
collect2: error: ld returned 1 exit status
--- a/src/starboard/raspi/shared/gyp_configuration.gypi
+++ b/src/starboard/raspi/shared/gyp_configuration.gypi
@@ -55,6 +55,9 @@
'-I<(sysroot)/opt/vc/include',
'-I<(sysroot)/opt/vc/include/interface/vcos/pthreads',
'-I<(sysroot)/opt/vc/include/interface/vmcs_host/linux',
+ # and for flsa, fmpeg
+ '-I<(sysroot)/usr/include',
+ '-I<(sysroot)/usr/include/arm-linux-gnueabihf',
],
'linker_flags': [
'--sysroot=<(sysroot)',
@@ -62,6 +65,11 @@
# libraries.
'-L<(sysroot)/opt/vc/lib',
'-Wl,-rpath=<(sysroot)/opt/vc/lib',
+ '-L<(sysroot)/usr/lib/arm-linux-gnueabihf',
+ '-Wl,-rpath-link=<(sysroot)/usr/lib/arm-linux-gnueabihf',
+ '-L<(sysroot)/lib/arm-linux-gnueabihf',
+ '-Wl,-rpath-link=<(sysroot)/lib/arm-linux-gnueabihf',
+ '-v',
# We don't wrap these symbols, but this ensures that they aren't
# linked in.
@@ -103,6 +111,7 @@
'-fno-rtti',
],
'platform_libraries': [
+ '-lm',
'-lasound',
'-lavcodec',
'-lavformat',
@@ -115,6 +124,7 @@
'-lbcm_host',
'-lvcos',
'-lvchiq_arm',
+ '-lmmal_core -lmmal_util -lcontainers -lmmal_components -lmmal_vc_client -lvcsm',
],
'conditions': [
['cobalt_fastbuild==0', {
rsync -avzh --exclude="obj*" out/raspi-2_devel pi@10.0.0.45:~/
ssh pi@$RASPI_ADDR
cd raspi-2_debug
./cobalt
Cobalt for Raspberry Pi
Source
src/starboard/raspi/1/gyp_configuration.gypi
src/starboard/raspi/1/starboard_platform.gyp
src/starboard/raspi/shared/configuration_public.h
src/starboard/raspi/shared/open_max/decode_target_create.cc
src/starboard/raspi/shared/open_max/decode_target_destroy.cc
src/starboard/raspi/shared/open_max/decode_target_get_format.cc
src/starboard/raspi/shared/open_max/decode_target_get_plane.cc
src/starboard/raspi/shared/open_max/decode_target_get_size.cc
src/starboard/raspi/shared/open_max/decode_target_internal.h
src/starboard/raspi/shared/open_max/decode_target_is_opaque.cc
src/starboard/raspi/shared/open_max/dispmanx_resource_pool.cc
src/starboard/raspi/shared/open_max/dispmanx_resource_pool.h
src/starboard/raspi/shared/open_max/image_decode.cc
src/starboard/raspi/shared/open_max/image_is_decode_supported.cc
src/starboard/raspi/shared/open_max/open_max_component.cc
src/starboard/raspi/shared/open_max/open_max_component.h
src/starboard/raspi/shared/open_max/open_max_component_base.cc
src/starboard/raspi/shared/open_max/open_max_component_base.h
src/starboard/raspi/shared/open_max/open_max_egl_render_component.cc
src/starboard/raspi/shared/open_max/open_max_egl_render_component.h
src/starboard/raspi/shared/open_max/open_max_image_decode_component.cc
src/starboard/raspi/shared/open_max/open_max_image_decode_component.h
src/starboard/raspi/shared/open_max/open_max_video_decode_component.cc
src/starboard/raspi/shared/open_max/open_max_video_decode_component.h
src/starboard/raspi/shared/open_max/video_decoder.cc
src/starboard/raspi/shared/open_max/video_decoder.h
留言