MediaCodec class can be used to access low-level media codecs, i.e. encoder/decoder components. It is part of the Android low-level multimedia support infrastructure (normally used together with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack.)

MediaCodec 类可以处理低层级的多媒体编解码器,如:encoder/decoder。它是Android低层级的多媒体支持基础的一部分(通常和MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack 一起使用)

Screen Shot 2017-05-23 at 10.55.42 AM.png

In broad terms, a codec processes input data to generate output data. It processes data asynchronously and uses a set of input and output buffers. At a simplistic level, you request (or receive) an empty input buffer, fill it up with data and send it to the codec for processing. The codec uses up the data and transforms it into one of its empty output buffers. Finally, you request (or receive) a filled output buffer, consume its contents and release it back to the codec.
总的来说,codec 处理输入的数据,生成输出数据。异步处理数据,并且使用一系列的输入输出缓冲(buffer). 在一个简化的层次, 你请求(或者接收)一个空的输入buffer, 填充数据后把它交给codec进行处理。codec用完数据后把buffer转换成它的里面的众多空的输出buffer中的一个。最后,你请求(或者接收)一个充满数据的输出buffer,提取出它里面的数据,然后把它释放回codec。

Data Types

Codecs operate on three kinds of data: compressed data, raw audio data and raw video data. All three kinds of data can be processed using ByteBuffers, but you should use a Surface for raw video data to improve codec performance. Surface uses native video buffers without mapping or copying them to ByteBuffers; thus, it is much more efficient. You normally cannot access the raw video data when using a Surface, but you can use the ImageReader class to access unsecured decoded (raw) video frames. This may still be more efficient than using ByteBuffers, as some native buffers may be mapped into direct ByteBuffers. When using ByteBuffer mode, you can access raw video frames using the Image class and getInput/OutputImage(int).


Codec对三种数据进行处理:压缩数据,音频原始数据和视频原始数据。三种数据都能够用ByteBuffers进行处理,但是为了提高codec处理能力,在处理视频原始数据时,你应该使用Surface。Surface使用native的视频buffer,不需要映射或都复制到ByteBuffers;因此,效率更高。通常使用Surface时,你无法接获取原始视频数据,但是你可以使用ImageReader类来获取未加密的解码后的(原始)视频帧。即使这样也比使用ByteBuffers的效率更高,因为一些native buffers可以被映射到ByteBuffers. 当使用ByteBuffer模式时,你可以用Image, getInput/OutputImage(int)获取到原始视频帧。

Compressed Buffers

Input buffers (for decoders) and output buffers (for encoders) contain compressed data according to the format's type. For video types this is a single compressed video frame. For audio data this is normally a single access unit (an encoded audio segment typically containing a few milliseconds of audio as dictated by the format type), but this requirement is slightly relaxed in that a buffer may contain multiple encoded access units of audio. In either case, buffers do not start or end on arbitrary byte boundaries, but rather on frame/access unit boundaries.

压缩 Buffers

输入buffers(解码器使用)和输出buffers(编码器使用)包含根据格式压缩的数据。如果是视频,那么就是压缩后的一帧。如果是音频,通常是一个access单位(一个编码过的音频片段,一般来说包含几毫秒的音频),但也可能几个access单位。不论是哪种情况,buffers 不会在随意的byte边界开始或结束,而是以frame/acess为单位来划分起始和结束位置。

Raw Audio Buffers

Raw audio buffers contain entire frames of PCM audio data, which is one sample for each channel in channel order. Each sample is a 16-bit signed integer in native byte order.



 short[] getSamplesForChannel(MediaCodec codec, int bufferId, int channelIx) {
   ByteBuffer outputBuffer = codec.getOutputBuffer(bufferId);
   MediaFormat format = codec.getOutputFormat(bufferId);
   ShortBuffer samples = outputBuffer.order(ByteOrder.nativeOrder()).asShortBuffer();
   int numChannels = formet.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
   if (channelIx < 0 || channelIx >= numChannels) {
     return null;
   short[] res = new short[samples.remaining() / numChannels];
   for (int i = 0; i < res.length; ++i) {
     res[i] = samples.get(i * numChannels + channelIx);
   return res;

Raw Video Buffers

In ByteBuffer mode video buffers are laid out according to their color format. You can get the supported color formats as an array from getCodecInfo().getCapabilitiesForType(…).colorFormats. Video codecs may support three kinds of color formats:



  • ****native raw video format:**** This is marked by COLOR_FormatSurface and it can be used with an input or output Surface.

  • ****flexible YUV buffers (such as COLOR_FormatYUV420Flexible):**** These can be used with an input/output Surface, as well as in ByteBuffer mode, by using getInput/OutputImage(int).

  • ****other, specific formats:**** These are normally only supported in ByteBuffer mode. Some color formats are vendor specific. Others are defined in MediaCodecInfo.CodecCapabilities. For color formats that are equivalent to a flexible format, you can still use getInput/OutputImage(int).
    All video codecs support flexible YUV 4:2:0 buffers since LOLLIPOP_MR1.

  • ****native原始视频格式**** 由COLOR_FormatSurface进行标记,可以在输入或者输出的Surface中进行使用。

  • ****可变的YUV buffers(例如COLOR_FormatYUV420Flexible) :****这些buffers可以在输入/输出Surface中使用,同样在ByteBuffer模式下通过getInput/OutputImage(int)也可以使用.

  • ****其他,特定的格式:****这些格式通常只在ByteBuffer模式下支持。一些颜色格式是厂商定制的。其他的格式在MediaCodecInfo.CodecCapabilities中进行定义。对于和可变的格式相同的颜色格式,你仍然可以使用getInput/OutputImage(int).从LOLLIPOP_MR1(API Level 22)开始所有的视频codecs支持可变的YUV 4:2:0 buffers。

Accessing Raw Video ByteBuffers on Older Devices

Prior to LOLLIPOP and Image support, you need to use the KEY_STRIDE and KEY_SLICE_HEIGHT output format values to understand the layout of the raw output buffers.



Note that on some devices the slice-height is advertised as 0. This could mean either that the slice-height is the same as the frame height, or that the slice-height is the frame height aligned to some value (usually a power of 2). Unfortunately, there is no standard and simple way to tell the actual slice height in this case. Furthermore, the vertical stride of the U plane in planar formats is also not specified or defined, though usually it is half of the slice height.

注意在一些设备上,slice-height值是0。这是说,要么 slice-height和帧高度一致,要么是帧高度与某个值(通常是2的指数)对齐之后的值。不幸的是,在这种情况下,没有一个标准简单的方法来告实际的slice height。Furthermore, the vertical stride of the U plane in planar formats is also not specified or defined, though usually it is half of the slice height.(不知道这句什么意思)

The KEY_WIDTH and KEY_HEIGHT keys specify the size of the video frames; however, for most encodings the video (picture) only occupies a portion of the video frame. This is represented by the 'crop rectangle'.
KEY_WIDTH和KEY_HEIGHT指定了视频的帧宽高;但是,对于大多数的encodings,视频(图片)只占了视频帧的一部分。这部分就是'crop rectangle'.

You need to use the following keys to get the crop rectangle of raw output images from the output format. If these keys are not present, the video occupies the entire video frame.The crop rectangle is understood in the context of the output frame before applying any rotation.
获取输出格式的原始输出图片的crop rectangle,需要用到以下的keys。如果没有提供这些keys,视频占据全部的视频帧。在旋转之前,crop rectangle在输出帧的上下文中进行‘理解’。

Format Key Type Description
"crop-left" Integer The left-coordinate (x) of the crop rectangle
"crop-top" Integer The top-coordinate (y) of the crop rectangle
"crop-right" Integer The right-coordinate (x) MINUS 1 of the crop rectangle
"crop-bottom" Integer The bottom-coordinate (y) MINUS 1 of the crop rectangle

The right and bottom coordinates can be understood as the coordinates of the right-most valid column/bottom-most valid row of the cropped output image.

The size of the video frame (before rotation) can be calculated as such:

 MediaFormat format = decoder.getOutputFormat(…);
 int width = format.getInteger(MediaFormat.KEY_WIDTH);
 if (format.containsKey("crop-left") && format.containsKey("crop-right")) {
     width = format.getInteger("crop-right") + 1 - format.getInteger("crop-left");
 int height = format.getInteger(MediaFormat.KEY_HEIGHT);
 if (format.containsKey("crop-top") && format.containsKey("crop-bottom")) {
     height = format.getInteger("crop-bottom") + 1 - format.getInteger("crop-top");

Also note that the meaning of BufferInfo.offset was not consistent across devices. On some devices the offset pointed to the top-left pixel of the crop rectangle, while on most devices it pointed to the top-left pixel of the entire frame.
另外需要注意的是,BufferInfo.offset 代表的意义在不同的设备上并不一样。在一些设备上,offset指crop rectangle最左上方的像素值,而在另一些设备上是指整个帧的最左上方的像素值。


During its life a codec conceptually exists in one of three states: Stopped, Executing or Released. The Stopped collective state is actually the conglomeration of three states: Uninitialized, Configured and Error, whereas the Executing state conceptually progresses through three sub-states: Flushed, Running and End-of-Stream.


在codec的生命周期中,逻辑上只处于三种状态中的一种:停止,执行和释放。停止状态包含三种状态:Uninitialized, Configured和Error;执行状态也含三种状态:Flushed, Running和End-of-Stream.

Screen Shot 2017-05-23 at 11.07.16 AM.png

When you create a codec using one of the factory methods, the codec is in the Uninitialized state. First, you need to configure it via configure(…), which brings it to the Configured state, then call start() to move it to the Executing state. In this state you can process data through the buffer queue manipulation described above.

The Executing state has three sub-states: Flushed, Running and End-of-Stream. Immediately after start() the codec is in the Flushed sub-state, where it holds all the buffers. As soon as the first input buffer is dequeued, the codec moves to the Running sub-state, where it spends most of its life. When you queue an input buffer with the end-of-stream marker, the codec transitions to the End-of-Stream sub-state. In this state the codec no longer accepts further input buffers, but still generates output buffers until the end-of-stream is reached on the output. You can move back to the Flushed sub-state at any time while in the Executing state using flush().
执行状态含三种状态:Flushed, Running和End-of-Stream.调用start()方法后,code立即处于Flushed sub-state, 持有所有buffers。当第一个输入buffer被dequeued,codec进入Running sub-state, 这个state会占据大部分生命周期。当你用一个end-of-stream标志queue一个输入buffer时,codec进入End-of-Stream状态。在这个状态,codec不再接收输入buffers,但是仍然在生成输出buffers,直到输出至end-of-stream。在Executing状态 ,任何时候你都可以调用flush()返回Flushed sub-state.

Call stop() to return the codec to the Uninitialized state, whereupon it may be configured again. When you are done using a codec, you must release it by calling release().

On rare occasions the codec may encounter an error and move to the Error state. This is communicated using an invalid return value from a queuing operation, or sometimes via an exception. Call reset() to make the codec usable again. You can call it from any state to move the codec back to the Uninitialized state. Otherwise, call release() to move to the terminal Released state.
极少数的情况下,codec会因为错误而进入Error状态 。这种况情下,queue操作会返回一个非法值,或者有时候抛出异常。调用reset()方法重置codec。在任何状态下都可以调用reset()方法使codec回到Uninitialized状态。


Use MediaCodecList to create a MediaCodec for a specific MediaFormat. When decoding a file or a stream, you can get the desired format from MediaExtractor.getTrackFormat. Inject any specific features that you want to add using MediaFormat.setFeatureEnabled, then call MediaCodecList.findDecoderForFormat to get the name of a codec that can handle that specific media format. Finally, create the codec using createByCodecName(String).



Note: On LOLLIPOP, the format to MediaCodecList.findDecoder/EncoderForFormat must not contain a frame rate. Use format.setString(MediaFormat.KEY_FRAME_RATE, null) to clear any existing frame rate setting in the format.
注意:在LOLLIPOP, MediaCodecList.findDecoder/EncoderForFormat中的format不包含frame rate. 使用format.setString(MediaFormat.KEY_FRAME_RATE, null)来清空已经format存在的frame rate.

You can also create the preferred codec for a specific MIME type using createDecoder/EncoderByType(String). This, however, cannot be used to inject features, and may create a codec that cannot handle the specific desired media format.

Creating secure decoders

On versions KITKAT_WATCH and earlier, secure codecs might not be listed in MediaCodecList, but may still be available on the system. Secure codecs that exist can be instantiated by name only, by appending ".secure" to the name of a regular codec (the name of all secure codecs must end in ".secure".) createByCodecName(String) will throw an IOException if the codec is not present on the system.


在KITKAT_WATCH(API Level 20)和之前,加密的codecs可能不在MediaCodecList里,但是仍然是可用的。在正常的codec名字后面添加".secure", 加密的codec只可以通过名字来初始化。createByCodecName(String)会抛出异常,如果codec在系统中不存在。

From LOLLIPOP onwards, you should use the FEATURE_SecurePlayback feature in the media format to create a secure decoder.
LOLLIPOP之后,你应该用通过media format 的FEATURE_SecurePlayback来创建一个加密的解码器。


After creating the codec, you can set a callback using setCallback if you want to process data asynchronously. Then, configure the codec using the specific media format. This is when you can specify the output Surface for video producers – codecs that generate raw video data (e.g. video decoders). This is also when you can set the decryption parameters for secure codecs (see MediaCrypto). Finally, since some codecs can operate in multiple modes, you must specify whether you want it to work as a decoder or an encoder.


创建codec之后,如果你想异步处理数据,你可以设置一个callback。然后,使用特定的media format来配置codec。这时你可以为video生产者(生生原始视频数据的codec, 例如:视频解码器)指定输出Surface。在这时也可以为加密的codec设置解密参数(查看:MediaCrypto).最后,因为一些codec能在几种模式下工作,你必须指定它是做为解码器还是编码器。

Since LOLLIPOP, you can query the resulting input and output format in the Configured state. You can use this to verify the resulting configuration, e.g. color formats, before starting the codec.
从LOLLIPOP开始,在Configured状态,你可以查询输入输出格式。你可以使用这个功能来验证配置结果,例如:颜色格式,在还没有starting codec之前。

If you want to process raw input video buffers natively with a video consumer – a codec that processes raw video input, such as a video encoder – create a destination Surface for your input data using createInputSurface() after configuration. Alternately, set up the codec to use a previously created persistent input surface by calling setInputSurface(Surface).
如果你想用一个codec(比如一个视频编码器)来处理原始的视频输入buffer,在配置之后,调用createInputSurface() 来为输入数据创建一个目的地Surface。或者,调用setInputSurface(Surface)来设置一个之前已经创建好的输入surface.

Codec-specific Data

Some formats, notably AAC audio and MPEG4, H.264 and H.265 video formats require the actual data to be prefixed by a number of buffers containing setup data, or codec specific data. When processing such compressed formats, this data must be submitted to the codec after start() and before any frame data. Such data must be marked using the flag BUFFER_FLAG_CODEC_CONFIG in a call to queueInputBuffer.


一些格式,如AAC音频和MPEG4, H.264, H.265视频格式需要在实际数据之前添加一些含设置数据或者指定编码的数据的buffers做为前缀。在处理这种压缩格式时,这些数据必须在start()之后、在任何帧数据之前提交给codec。在queueInputBuffer时,这种数据必须用BUFFER_FLAG_CODEC_CONFIG标志。

Codec-specific data can also be included in the format passed to configure in ByteBuffer entries with keys "csd-0", "csd-1", etc. These keys are always included in the track MediaFormat obtained from the MediaExtractor. Codec-specific data in the format is automatically submitted to the codec upon start(); you MUST NOT submit this data explicitly. If the format did not contain codec specific data, you can choose to submit it using the specified number of buffers in the correct order, according to the format requirements. In case of H.264 AVC, you can also concatenate all codec-specific data and submit it as a single codec-config buffer.

Android uses the following codec-specific data buffers. These are also required to be set in the track format for proper MediaMuxer track configuration. Each parameter set and the codec-specific-data sections marked with (*) must start with a start code of "\x00\x00\x00\x01".

Screen Shot 2017-05-23 at 11.09.26 AM.png

Note: care must be taken if the codec is flushed immediately or shortly after start, before any output buffer or output format change has been returned, as the codec specific data may be lost during the flush. You must resubmit the data using buffers marked with BUFFER_FLAG_CODEC_CONFIG after such flush to ensure proper codec operation.

Encoders (or codecs that generate compressed data) will create and return the codec specific data before any valid output buffer in output buffers marked with the codec-config flag. Buffers containing codec-specific-data have no meaningful timestamps.

Data Processing

Each codec maintains a set of input and output buffers that are referred to by a buffer-ID in API calls. After a successful call to start() the client "owns" neither input nor output buffers. In synchronous mode, call dequeueInput/OutputBuffer(…) to obtain (get ownership of) an input or output buffer from the codec. In asynchronous mode, you will automatically receive available buffers via the MediaCodec.Callback.onInput/OutputBufferAvailable(…) callbacks.


每个codec维护着一批输入和输出buffers,在API请求中, 这些buffer可以通过buffer-ID来指向。在成功调用start()方法后,客户端既没有‘拥有’输出也没有‘拥有’输入buffers。在同步模式中,调用dequeueInput/OutputBuffer(…)从codec中获取输入或者输出buffer。在异步模式中,通过MediaCodec.Callback.onInput/OutputBufferAvailable(…) 回调,你会自动收到可用的buffers。

Upon obtaining an input buffer, fill it with data and submit it to the codec using queueInputBuffer – or queueSecureInputBuffer if using decryption. Do not submit multiple input buffers with the same timestamp (unless it is codec-specific data marked as such).
收到输入buffer时,填入数据后用queueInputBuffer交给codec, 或者,如果是加密的,用queueSecureInputBuffer。不要同时提交多个有同要的时间戳的输入buffers(除非是指定codec的data).

The codec in turn will return a read-only output buffer via the onOutputBufferAvailable callback in asynchronous mode, or in response to a dequeuOutputBuffer call in synchronous mode. After the output buffer has been processed, call one of the releaseOutputBuffer methods to return the buffer to the codec.
在异步模式下,通过onOutputBufferAvailable 回调codec会返回一个只读的输出buffer,或者在同步模式下,响应dequeuOutputBuffer。输出buffer被处理后,调用releaseOutputBuffer来释放buffer给codec。

While you are not required to resubmit/release buffers immediately to the codec, holding onto input and/or output buffers may stall the codec, and this behavior is device dependent. Specifically, it is possible that a codec may hold off on generating output buffers until all outstanding buffers have been released/resubmitted. Therefore, try to hold onto to available buffers as little as possible.
当你没有被要求马上提交或者释放buffers给codec, 持有输入或者输出buffers可能使codec停止工作,最终结果可能与设备有关。特别地,codec可能取消生成输出buffers直到所有未完成的buffers被释放/提交。所以,尽可能地少持有buffers.

Depending on the API version, you can process data in three ways:

Screen Shot 2017-05-23 at 11.11.21 AM.png

Asynchronous Processing using Buffers

Since LOLLIPOP, the preferred method is to process data asynchronously by setting a callback before calling configure. Asynchronous mode changes the state transitions slightly, because you must call start() after flush() to transition the codec to the Running sub-state and start receiving input buffers. Similarly, upon an initial call to start the codec will move directly to the Running sub-state and start passing available input buffers via the callback.


从LOLLIPOP开始,较好的方式是在配置之前设置回调,异步处理数据。异步模式稍微改变了状态的变换,因为你必须在flush()之后调用start(),使codec处于Running 状态,接收输入buffers.同样, codec初始化开始之后会直接进入Running状态, 并通过回调传递可用的输入buffers.

Screen Shot 2017-05-23 at 11.11.59 AM.png

MediaCodec is typically used like this in asynchronous mode:

MediaCodec codec = MediaCodec.createByCodecName(name);
 MediaFormat mOutputFormat; // member variable
 codec.setCallback(new MediaCodec.Callback() {
   void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
     ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
     // fill inputBuffer with valid data
     codec.queueInputBuffer(inputBufferId, …);

   void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
     MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
     // bufferFormat is equivalent to mOutputFormat
     // outputBuffer is ready to be processed or rendered.
     codec.releaseOutputBuffer(outputBufferId, …);

   void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
     // Subsequent data will conform to new format.
     // Can ignore if using getOutputFormat(outputBufferId)
     mOutputFormat = format; // option B

   void onError(…) {
 codec.configure(format, …);
 mOutputFormat = codec.getOutputFormat(); // option B
 // wait for processing to complete

Synchronous Processing using Buffers

Since LOLLIPOP, you should retrieve input and output buffers using getInput/OutputBuffer(int) and/or getInput/OutputImage(int) even when using the codec in synchronous mode. This allows certain optimizations by the framework, e.g. when processing dynamic content. This optimization is disabled if you call getInput/OutputBuffers().



Note: do not mix the methods of using buffers and buffer arrays at the same time. Specifically, only call getInput/OutputBuffers directly after start() or after having dequeued an output buffer ID with the value of INFO_OUTPUT_FORMAT_CHANGED.

注意:不要在同时使用buffers和buffer arrays的方法。特别是在start()之后马上调用getInput/OutputBuffers,或者是dequeued一个ID为INFO_OUTPUT_FORMAT_CHANGED的输出buffer.

MediaCodec is typically used like this in synchronous mode:

MediaCodec codec = MediaCodec.createByCodecName(name);
 codec.configure(format, …);
 MediaFormat outputFormat = codec.getOutputFormat(); // option B
 for (;;) {
   int inputBufferId = codec.dequeueInputBuffer(timeoutUs);
   if (inputBufferId >= 0) {
     ByteBuffer inputBuffer = codec.getInputBuffer(…);
     // fill inputBuffer with valid data
     codec.queueInputBuffer(inputBufferId, …);
   int outputBufferId = codec.dequeueOutputBuffer(…);
   if (outputBufferId >= 0) {
     ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
     MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
     // bufferFormat is identical to outputFormat
     // outputBuffer is ready to be processed or rendered.
     codec.releaseOutputBuffer(outputBufferId, …);
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
     // Subsequent data will conform to new format.
     // Can ignore if using getOutputFormat(outputBufferId)
     outputFormat = codec.getOutputFormat(); // option B

Synchronous Processing using Buffer Arrays (deprecated)

In versions KITKAT_WATCH and before, the set of input and output buffers are represented by the ByteBuffer[] arrays. After a successful call to start(), retrieve the buffer arrays using getInput/OutputBuffers(). Use the buffer ID-s as indices into these arrays (when non-negative), as demonstrated in the sample below. Note that there is no inherent correlation between the size of the arrays and the number of input and output buffers used by the system, although the array size provides an upper bound.

同步模式下Buffer Arrays的使用(deprecated)

在KITKAT_WATCH(API Level 20)和之前,输入和输出buffers都是用ByteBuffer[] 表示的。成功调用start()之后,通过getInput/OutputBuffers()来获取Buffer数组。像下面的例子,采用buffer IDs来做为数组索引。注意,数组长度和系统中使用的输入和输入buffers数量并没有内在关联,虽然数组有一个上限。

 MediaCodec codec = MediaCodec.createByCodecName(name);
 codec.configure(format, …);
 ByteBuffer[] inputBuffers = codec.getInputBuffers();
 ByteBuffer[] outputBuffers = codec.getOutputBuffers();
 for (;;) {
   int inputBufferId = codec.dequeueInputBuffer(…);
   if (inputBufferId >= 0) {
     // fill inputBuffers[inputBufferId] with valid data
     codec.queueInputBuffer(inputBufferId, …);
   int outputBufferId = codec.dequeueOutputBuffer(…);
   if (outputBufferId >= 0) {
     // outputBuffers[outputBufferId] is ready to be processed or rendered.
     codec.releaseOutputBuffer(outputBufferId, …);
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
     outputBuffers = codec.getOutputBuffers();
   } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
     // Subsequent data will conform to new format.
     MediaFormat format = codec.getOutputFormat();

End-of-stream Handling

When you reach the end of the input data, you must signal it to the codec by specifying the BUFFER_FLAG_END_OF_STREAM flag in the call to queueInputBuffer. You can do this on the last valid input buffer, or by submitting an additional empty input buffer with the end-of-stream flag set. If using an empty buffer, the timestamp will be ignored.


当输入数据结束时,你必须通知codec, 在调用queueInputBuffer时标志BUFFER_FLAG_END_OF_STREAM.你可以在最后一个输入buffer进行标志,或者增加一个设置了end-of-stream标志的额外的空buffer。如果使用空buffer,时间戳会被忽略。

The codec will continue to return output buffers until it eventually signals the end of the output stream by specifying the same end-of-stream flag in the MediaCodec.BufferInfo set in dequeueOutputBuffer or returned via onOutputBufferAvailable. This can be set on the last valid output buffer, or on an empty buffer after the last valid output buffer. The timestamp of such empty buffer should be ignored.
Codec会持续返回输入buffers直到最后在dequeueOutputBuffer 或者onOutputBufferAvailable 返回的MediaCodec.BufferInfo中标记同样的end-of-stream来说明输出流结束。可能在最后一个输出buffer中标记或者在最后一个buffer后面添加一个空的buffer进记标记。空的buffer上面的时间戳会被忽略。

Do not submit additional input buffers after signaling the end of the input stream, unless the codec has been flushed, or stopped and restarted.

Using an Output Surface

The data processing is nearly identical to the ByteBuffer mode when using an output Surface; however, the output buffers will not be accessible, and are represented as null values. E.g. getOutputBuffer/Image(int) will return null and getOutputBuffers() will return an array containing only null-s.


使用一个输出Surface的数据处理与ByteBuffer模式下的数据处理几乎是一样的;但是,在Surface模式下,不能获取输出buffers(值都为null).例如:getOutputBuffer/Image(int) 会返回null, getOutputBuffers()会返回一个只包含null的数组。

When using an output Surface, you can select whether or not to render each output buffer on the surface. You have three choices:

  • Do not render the buffer: Call releaseOutputBuffer(bufferId, false).

  • 不绘制buffer:调用releaseOutputBuffer(bufferId, false).

  • Render the buffer with the default timestamp: Call releaseOutputBuffer(bufferId, true).

  • 绘制有缺省时间戳的buffer:调用releaseOutputBuffer(bufferId, true)

  • Render the buffer with a specific timestamp: Call releaseOutputBuffer(bufferId, timestamp).
  • 绘制指定时间戳的buffer: 调用releaseOutputBuffer(bufferId, timestamp).

Since M, the default timestamp is the presentation timestamp of the buffer (converted to nanoseconds). It was not defined prior to that.
从M(API Level23)开始,缺省时间戳是presentation时间戳(转换成nanoseconds)。在M之前是没有定义的。

Also since M, you can change the output Surface dynamically using setOutputSurface.

Transformations When Rendering onto Surface

If the codec is configured into Surface mode, any crop rectangle, rotation and video scaling mode will be automatically applied with one exception:



Prior to the M release, software decoders may not have applied the rotation when being rendered onto a Surface. Unfortunately, there is no standard and simple way to identify software decoders, or if they apply the rotation other than by trying it out.

There are also some caveats.

Note that the pixel aspect ratio is not considered when displaying the output onto the Surface. This means that if you are using VIDEO_SCALING_MODE_SCALE_TO_FIT mode, you must position the output Surface so that it has the proper final display aspect ratio. Conversely, you can only use VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING mode for content with square pixels (pixel aspect ratio or 1:1).

Note also that as of N release, VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING mode may not work correctly for videos rotated by 90 or 270 degrees.
注意,当N(API Level 24)发布 ,videos旋转了90或者270度时,VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING模式可能不能正常工作。

When setting the video scaling mode, note that it must be reset after each time the output buffers change. Since the INFO_OUTPUT_BUFFERS_CHANGED event is deprecated, you can do this after each time the output format changes.

Using an Input Surface

When using an input Surface, there are no accessible input buffers, as buffers are automatically passed from the input surface to the codec. Calling dequeueInputBuffer will throw an IllegalStateException, and getInputBuffers() returns a bogus ByteBuffer[] array that MUST NOT be written into.



Call signalEndOfInputStream() to signal end-of-stream. The input surface will stop submitting data to the codec immediately after this call.

Seeking & Adaptive Playback Support

Video decoders (and in general codecs that consume compressed video data) behave differently regarding seek and format change whether or not they support and are configured for adaptive playback. You can check if a decoder supports adaptive playback via CodecCapabilities.isFeatureSupported(String). Adaptive playback support for video decoders is only activated if you configure the codec to decode onto a Surface.



Stream Boundary and Key Frames

It is important that the input data after start() or flush() starts at a suitable stream boundary: the first frame must a key frame. A key frame can be decoded completely on its own (for most codecs this means an I-frame), and no frames that are to be displayed after a key frame refer to frames before the key frame.


start()和flush()之后,输入数据从一个合适的stream边界开始是非常重要的:第一个帧必须是关键帧。一个关键帧可以被完全的解码(对于大多数的codecs来说是一个I-frame), 而且关键帧之后显示的帧不能指向关键帧之前的帧。

The following table summarizes suitable key frames for various video formats.

Screen Shot 2017-05-23 at 11.17.57 AM.png

For decoders that do not support adaptive playback (including when not decoding onto a Surface)

In order to start decoding data that is not adjacent to previously submitted data (i.e. after a seek) you MUST flush the decoder. Since all output buffers are immediately revoked at the point of the flush, you may want to first signal then wait for the end-of-stream before you call flush. It is important that the input data after a flush starts at a suitable stream boundary/key frame.


为了解码和之前提交的数据(比如,快进后)不相临的数据,你必须flsuh解码器。因为所有的输出buffers在flush时立刻被废除了,所以在你调用flush前,你可能需要先通知,然后等待end-of-stream 。flush之后,输入数据从一个合适的stream边界/关键帧开始是非常重要的。

Note: the format of the data submitted after a flush must not change; flush() does not support format discontinuities; for that, a full stop() - configure(…) - start() cycle is necessary.

Also note: if you flush the codec too soon after start() – generally, before the first output buffer or output format change is received – you will need to resubmit the codec-specific-data to the codec. See the codec-specific-data section for more info.

For decoders that support and are configured for adaptive playback

In order to start decoding data that is not adjacent to previously submitted data (i.e. after a seek) it is not necessary to flush the decoder; however, input data after the discontinuity must start at a suitable stream boundary/key frame.



For some video formats - namely H.264, H.265, VP8 and VP9 - it is also possible to change the picture size or configuration mid-stream. To do this you must package the entire new codec-specific configuration data together with the key frame into a single buffer (including any start codes), and submit it as a regular input buffer.
对于一些视频格式,比如:H.264, H.265, VP8 and VP9,是可能改变图片大小或者mid-stream的结构。要实现这一步,你必须对整个新的特定编码的配置数据和关键帧进行打包成一个新buffer(包括任何start codes),然作为一个普通的输入buffer进行提交。

You will receive an INFO_OUTPUT_FORMAT_CHANGED return value from dequeueOutputBuffer or a onOutputFormatChanged callback just after the picture-size change takes place and before any frames with the new size have been returned.

Note: just as the case for codec-specific data, be careful when calling flush() shortly after you have changed the picture size. If you have not received confirmation of the picture size change, you will need to repeat the request for the new picture size.

Error handling

The factory methods createByCodecName and createDecoder/EncoderByType throw IOException on failure which you must catch or declare to pass up. MediaCodec methods throw IllegalStateException when the method is called from a codec state that does not allow it; this is typically due to incorrect application API usage. Methods involving secure buffers may throw MediaCodec.CryptoException, which has further error information obtainable from getErrorCode().


工厂方法createByCodecName和createDecoder/EncoderByType 在遇错的时候会抛出IOException,你必须进行catch或者声明异常。当codec在一个不允许的状态被调用时会抛出IllegalStateException;这是典型的API错误引起的。与加密的buffers有关的方法可能会抛出MediaCodec.CryptoException,使用getErrorCode()方法可以获取进一步的错误信息。

Internal codec errors result in a MediaCodec.CodecException, which may be due to media content corruption, hardware failure, resource exhaustion, and so forth, even when the application is correctly using the API. The recommended action when receiving a CodecException can be determined by calling isRecoverable() and isTransient():

  • recoverable errors: If isRecoverable() returns true, then call stop(), configure(…), and start() to recover.

  • 能恢复的错误: 如果isRecoverable() 返回true, 那么调用stop(),configure(…), 和start() 来恢复。

  • transient errors: If isTransient() returns true, then resources are temporarily unavailable and the method may be retried at a later time.

  • 短暂的错误:如果isTransient()返回true,那么暂时资源不可用,可以晚点再试这个方法。

  • fatal errors: If both isRecoverable() and isTransient() return false, then the CodecException is fatal and the codec must be reset or released.

  • 致命的错误:如果isRecoverable()和isTransient()都返回false,那么CodecException是致命的,codec必须被reset或者released.

Both isRecoverable() and isTransient() do not return true at the same time.

Valid API Calls and API History

This sections summarizes the valid API calls in each state and the API history of the MediaCodec class. For API version numbers, see Build.VERSION_CODES.



Screen Shot 2017-05-25 at 11.14.34 AM.png


  • 本篇文章是基于谷歌有关Graphic的一篇概览文章的翻译:
    lee_3do阅读 3,103评论 3 14
  • **2014真题Directions:Read the following text. Choose the be...
    又是夜半惊坐起阅读 3,578评论 0 5
  • 关于我 特别羡慕那些有自己个性的人,而我是否是落在人群里挖都挖不出来的人。很长一段时间因为自己不够倾城,不够鹤立鸡...
    土豆和紫苏阅读 22评论 0 0
  • 这是一部励志的电影,卡尔在父亲的鼓励下参加了美国海军,父亲跟他说不要过像他一样的生活,不管遇到什么困难都...
    阿敏m阅读 44评论 0 0
  • 周遭一片乌暗 摊在娃娃床的那颓肉体瞪大双目 一张血肉模糊的面具 映现在了那对白珠间 没有半点响声 徒留两张面面相觑...
    花予辰阅读 73评论 0 5