Android P 图形显示系统(十二) BufferQueue(三)

Buffer状态

对于生产者这边,BufferQueue的流程基本讲完了。简单说来,首先提需求,告诉BufferQueue需要什么样的Buffer,大小,格式,usage等等;然后dequeue Buffer出来,往Buffer里面绘制显示数据;绘制完成后,queue到BufferQueue里面,并通知消费者进行消费。如此不断的的dequeue,绘制,queue。

消费者这边的流程,我们还没有讲到。对于消费者来说,收到通知后,将从BufferQueue里面取queue过来的Buffer进行合成,合成完的Buffer再释放掉,这里的释放,是概念上的,并没有真正释放内存,只是让其返回队列,可以被再次dequeue。消费者这边也是不断的接通知,取buffer合成,然后释放,不断循环。
此图是Android官网对BufferQueue通信过程的描述,这很好的描述这个过程。


BufferQueue数据流

在Android 6.0及之前的版本,在这些通信过程中,都将Buffer的状态标记为具体的状态。这四个过程Buffer分别对应不同的四个状态:

  • DEQUEUED 状态
    Producer dequeue一个Buffer后,这个Buffer就变为DEQUEUED状态,release Fence发信号后,Producer就可以修改Buffer的内容,我们称为release Fence。此时Buffer被Producer占用。DEQUEUED状态的Buffer可以迁移到 QUEUED 状态,通过queueBuffer或attachBuffer流程。也可以迁移到FREE装,通过cancelBuffer或detachBuffer流程。

  • QUEUED 状态
    Buffer绘制完后,queue到BufferQueue中,给Consumer进行消费。此时Buffer可能还没有真正绘制完成,必现要等对应的Fence发信号出来后,才真正完成。此时Buffer是BufferQueue持有,可以迁移到ACQUIRED状态,通过acquireBuffer流程。而已可以迁移到FREE状态,如果另外一个Buffer被异步的queue进来。

  • ACQUIRED 状态
    Buffer已经被Consumer获取,但是也必须要等对应的Fence发信号才能被Consumer读写,找个Fence是从Producer那边,queueBuffer的时候传过来的。我们将其称为acquire fence。此时,Buffer被Consumer持有。状态可以迁移到FREE状态,通过releaseBuffer或detachBuffer流程。除了从acquireBuffer流程可以迁移到ACQUIRED状态,attachBuffer流程也可以迁移到ACQUIRED状态。

  • FREE 状态
    FREE状态,说明Buffer被BufferQueue持有,可以被Producer dequeue,它将迁移到DEQUEUED状态,通过dequeueBuffer流程。

  • SHARED状态
    SHARED状态是一个特殊的状态,SHARED的Buffer并不参与前面所说的状态迁移。它说明Buffer被用与共享Buffer模式。除了FREE状态,它可以是其他的任何状态。它可以被多次dequeued, queued, 或者 acquired。这中共享Buffer的模式,主要用于VR等低延迟要求的场合。

目前,Buffer的状态,都是通过各个状态的Buffer的量来表示状态,对应的关系如下:

Buffer状态 mShared mDequeueCount mQueueCount mAcquireCount
FREE false 0 0 0
DEQUEUED false 1 0 0
QUEUED false 0 1 0
ACQUIRED false 0 0 1
SHARED true any any any

Buffer的状态在代码中用BufferState描述,BufferState的定义如下:

* frameworks/native/libs/gui/include/gui/BufferSlot.h

struct BufferState {

    BufferState()
    : mDequeueCount(0),
      mQueueCount(0),
      mAcquireCount(0),
      mShared(false) {
    }

    uint32_t mDequeueCount;
    uint32_t mQueueCount;
    uint32_t mAcquireCount;
    bool mShared;

    ... ...
};

前面讲解dequeueBuffer和queueBuffer流程时,BufferQueue有很多个队列,我们再来看一下BufferQueue中,几个队列间的关系。

BufferQueueCore中的定义如下:

* frameworks/native/libs/gui/include/gui/BufferQueueCore.h

class BufferQueueCore : public virtual RefBase {

    ... ...

    typedef Vector<BufferItem> Fifo;
    
    ... ...

    // mSlots is an array of buffer slots that must be mirrored on the producer
    // side. This allows buffer ownership to be transferred between the producer
    // and consumer without sending a GraphicBuffer over Binder. The entire
    // array is initialized to NULL at construction time, and buffers are
    // allocated for a slot when requestBuffer is called with that slot's index.
    BufferQueueDefs::SlotsType mSlots;

    // mQueue is a FIFO of queued buffers used in synchronous mode.
    Fifo mQueue;

    // mFreeSlots contains all of the slots which are FREE and do not currently
    // have a buffer attached.
    std::set<int> mFreeSlots;

    // mFreeBuffers contains all of the slots which are FREE and currently have
    // a buffer attached.
    std::list<int> mFreeBuffers;

    // mUnusedSlots contains all slots that are currently unused. They should be
    // free and not have a buffer attached.
    std::list<int> mUnusedSlots;

    // mActiveBuffers contains all slots which have a non-FREE buffer attached.
    std::set<int> mActiveBuffers;
  • mSlots
    mSlots 是Buffer序号的一个数组,Producer端的mSlots也是这个mSlots,Consumer端是mSlots也是里的mSlots的引用。它可实现Buffer在Producer和Consumer之间转移,而不需要真正的在Binder间去传输一个GraphicBuffer。初始状态时为空,当requestBuffer流程执行时,将去为对应的Buffer序号,分配真正的Buffer。

  • mQueue
    mQueue是一个先进先出的Vector,是同步模式下使用。里面就是处于QUEUED状态的Buffer。

  • mFreeSlots
    mFreeSlots包含所有是FREE状态,且还没有分配Buffer的,Buffer序号集合。刚开始时,mFreeSlots被初始化为MaxBufferCount个Buffer序号集合,dequeueBuffer的时候,将先从这个集合中获取。但是消费者消费完成,释放的Buffer并不返回到这个队列中,而是返回到mFreeBuffers中。

  • mFreeBuffers
    mFreeBuffers包含的是所有FREE状态,且已经分配Buffer的,Buffer序号的结合。消费者消费完成,释放的Buffer并不返回到这个队列中,而是返回到mFreeBuffers中。

  • mUnusedSlots
    mUnusedSlots和mFreeSlots有些相似,只是mFreeSlots会被用到,而mUnusedSlots中的Buffer序号不会不用到。也就是,总的Buffer序号NUM_BUFFER_SLOTS中,除去MaxBufferCount个mFreeSlots,剩余的集合。

  • mActiveBuffers
    mActiveBuffers包含所有非FREE状态的Buffer。也就是包含了DEQUEUED,QUEUED,ACQUIRED以及SHARED这几个状态的。

我们从数学的角度来看看他们之间的关系:
mSlots的数组大小为NUM_BUFFER_SLOTS,但是其中,真正用起来的也只有MaxBufferCount个,其他的都不会被用到。所以,我们可以这么理解,mSlots是BufferQueue中实际流转起来的Buffer。

mSlots = mFreeBuffers + mActiveBuffers

对于整体而言:

NUM_BUFFER_SLOTS = mUnusedSlots + mFreeSlots + mFreeBuffers + mActiveBuffers

mSlots是BufferSlot的集合,BufferSlot定义如下:

struct BufferSlot {

    BufferSlot()
    : mGraphicBuffer(nullptr),
      mEglDisplay(EGL_NO_DISPLAY),
      mBufferState(),
      mRequestBufferCalled(false),
      mFrameNumber(0),
      mEglFence(EGL_NO_SYNC_KHR),
      mFence(Fence::NO_FENCE),
      mAcquireCalled(false),
      mNeedsReallocation(false) {
    }

    // Buffer序号对应的Buffer
    sp<GraphicBuffer> mGraphicBuffer;

    // 创建EGLSyncKHR对象用
    EGLDisplay mEglDisplay;

    // Buffer序号当前的状态
    BufferState mBufferState;

    // mRequestBufferCalled 表示Producer确实已经调用requestBuffer
    bool mRequestBufferCalled;

    // mFrameNumber 表示该Buffer序号已经被queue的次数.  主要用于dequeueBuffer时,遵从LRU,这很有用,因为buffer 变FREE时,可能release Fence还没有发信号出来。
    uint64_t mFrameNumber;

    // 现在已经被mFence替换了,基本不用
    EGLSyncKHR mEglFence;

    // mFence 是同步的一种方式,上一个owner使用完Buffer后,需要发信号出来,下一个owner才可以使用。
    sp<Fence> mFence;

    // 表示Buffer已经被Consumer取走
    bool mAcquireCalled;

    // 表示Buffer需要重新分配,需要设置BUFFER_NEEDS_REALLOCATION 通知Producer,不要用原来的缓存的Buffer
    bool mNeedsReallocation;
};

看完Buffer的状态后,再回头去看看前面介绍的dequeueBuffer和queueBuffer,是不是就很好理解了。

我们再来看看BufferQueue的工作模式,BufferQueue可以工作在几个模式:

  • 同步模式 Synchronous-like mode
    默认情况下,BufferQueue将工作在同步模式下。在该模式下,每个Buffer都从Producer进入,从Consumer退出,没有Buffer没有丢弃掉。如果Producer生产的太快,Consumer来不及消费,Producer将阻塞等待FREE的Buffer。前面的分析流程的时候在waitForFreeSlotThenRelock也说到了这点。
    这是waitForFreeSlotThenRelock函数中的逻辑:
            if (mDequeueTimeout >= 0) {
                status_t result = mCore->mDequeueCondition.waitRelative(
                        mCore->mMutex, mDequeueTimeout);
                if (result == TIMED_OUT) {
                    return result;
                }
            } else {
                mCore->mDequeueCondition.wait(mCore->mMutex);
            }
  • 非同步模式 Non-blocking mode
    和同步模式相反,BufferQueue工作在非阻塞模式下,在这种模式下,如果没有FREE Buffer,将生成一个错误,而不是阻塞等待FREE的Buffer。这种模式,也没有Buffer不丢弃。这中模式可以避免潜在的死锁,如果应用不理解Graphics框架中复杂的依赖条件。前面我们的代码分析中也看到这一点。waitForFreeSlotThenRelock什么时候不去tryAgain
        if (tryAgain) {
            if ((mCore->mDequeueBufferCannotBlock || mCore->mAsyncMode) &&
                    (acquiredCount <= mCore->mMaxAcquiredBufferCount)) {
                return WOULD_BLOCK;
            }

mAsyncMode是通过BufferQueueProducer的setAsyncMode函数设置的,从Producer调用过来,受Producer控制。

mDequeueBufferCannotBlock则是在Producer 连接到BufferQueue时,根据条件判断的,具体逻辑如下:

status_t BufferQueueProducer::connect(const sp<IProducerListener>& listener,
        int api, bool producerControlledByApp, QueueBufferOutput *output) {
    ... ...

    if (mDequeueTimeout < 0) {
        mCore->mDequeueBufferCannotBlock =
                mCore->mConsumerControlledByApp && producerControlledByApp;
    }

    mCore->mAllowAllocation = true;
    VALIDATE_CONSISTENCY();
    return status;
}
  • 舍弃模式 Discard mode
    BufferQueue可以配置为丢弃旧Buffer,而不是生成错误或进行等待。比如,如果用GL对纹理进行快速的绘制,那么旧的Buffer不要丢弃。

  • 共享Buffer模式 shared buffer mode
    共享Buffer模式,表示Buffer是Producer和Consumer共享。共享Buffer模式下,一直用的都是同一个Buffer。而Buffer的状态不能迁移为FREE状态。代码中可以留意mCore->mSharedBufferModemCore->mSharedBufferSlot。这个模式其实也包含在同步模式中,只是比较特殊,单独说一下。

现在,再回头去看看前面介绍的dequeueBuffer和queueBuffer,是不是就更好理解了。

acquireBuffer流程

Buffer queue到BufferQueue中后,将通知消费者去消费。消费时,通过acquireBuffer来获取Buffer,我们且不管acquireBuffer是什么地方调的,我们先来看BufferQueue中acquireBuffer的处理流程。

* frameworks/native/libs/gui/BufferQueueConsumer.cpp

status_t BufferQueueConsumer::acquireBuffer(BufferItem* outBuffer,
        nsecs_t expectedPresent, uint64_t maxFrameNumber) {
    ATRACE_CALL();

    int numDroppedBuffers = 0;
    sp<IProducerListener> listener;
    {
        Mutex::Autolock lock(mCore->mMutex);

        int numAcquiredBuffers = 0;
        for (int s : mCore->mActiveBuffers) {
            if (mSlots[s].mBufferState.isAcquired()) {
                ++numAcquiredBuffers;
            }
        }
        if (numAcquiredBuffers >= mCore->mMaxAcquiredBufferCount + 1) {
            BQ_LOGE("acquireBuffer: max acquired buffer count reached: %d (max %d)",
                    numAcquiredBuffers, mCore->mMaxAcquiredBufferCount);
            return INVALID_OPERATION;
        }

        bool sharedBufferAvailable = mCore->mSharedBufferMode &&
                mCore->mAutoRefresh && mCore->mSharedBufferSlot !=
                BufferQueueCore::INVALID_BUFFER_SLOT;

        // In asynchronous mode the list is guaranteed to be one buffer deep,
        // while in synchronous mode we use the oldest buffer.
        if (mCore->mQueue.empty() && !sharedBufferAvailable) {
            return NO_BUFFER_AVAILABLE;
        }
  • acquireBuffer时,也是受mCore->mMutex控制的。
  • numAcquiredBuffers,已经acquired的Buffer。mMaxAcquiredBufferCount最大可以acquire的Buffer,可以溢出一个,以便Consumer能方便替换旧的Buffer,如果旧的Buffer还没有释放时。
  • sharedBufferAvailable,共享Buffer模式下使用。在这个模式下,mAutoRefresh表示,Consumer永远可以acquire到一块Buffer,即使BufferQueue还没有处于可以acquire的状态。
  • mQueue,如没有Buffer被queue过来,mQueue为空,那么Consumer这边就acquire不到新的Buffer,Consumer这边已经acquire的会被继续使用。

如果有Buffer或是共享Buffer模式,继续~

* frameworks/native/libs/gui/BufferQueueConsumer.cpp

        BufferQueueCore::Fifo::iterator front(mCore->mQueue.begin());

        if (expectedPresent != 0 && !mCore->mQueue.empty()) {
            const int MAX_REASONABLE_NSEC = 1000000000ULL; // 1 second

            while (mCore->mQueue.size() > 1 && !mCore->mQueue[0].mIsAutoTimestamp) {
                const BufferItem& bufferItem(mCore->mQueue[1]);

                // If dropping entry[0] would leave us with a buffer that the
                // consumer is not yet ready for, don't drop it.
                if (maxFrameNumber && bufferItem.mFrameNumber > maxFrameNumber) {
                    break;
                }

                if (desiredPresent < expectedPresent - MAX_REASONABLE_NSEC ||
                        desiredPresent > expectedPresent) {
                    // This buffer is set to display in the near future, or
                    // desiredPresent is garbage. Either way we don't want to drop
                    // the previous buffer just to get this on the screen sooner.
                    BQ_LOGV("acquireBuffer: nodrop desire=%" PRId64 " expect=%"
                            PRId64 " (%" PRId64 ") now=%" PRId64,
                            desiredPresent, expectedPresent,
                            desiredPresent - expectedPresent,
                            systemTime(CLOCK_MONOTONIC));
                    break;
                }

                BQ_LOGV("acquireBuffer: drop desire=%" PRId64 " expect=%" PRId64
                        " size=%zu",
                        desiredPresent, expectedPresent, mCore->mQueue.size());

                if (!front->mIsStale) {
                    // Front buffer is still in mSlots, so mark the slot as free
                    mSlots[front->mSlot].mBufferState.freeQueued();

                    if (!mCore->mSharedBufferMode &&
                            mSlots[front->mSlot].mBufferState.isFree()) {
                        mSlots[front->mSlot].mBufferState.mShared = false;
                    }

                    // Don't put the shared buffer on the free list
                    if (!mSlots[front->mSlot].mBufferState.isShared()) {
                        mCore->mActiveBuffers.erase(front->mSlot);
                        mCore->mFreeBuffers.push_back(front->mSlot);
                    }

                    listener = mCore->mConnectedProducerListener;
                    ++numDroppedBuffers;
                }

                mCore->mQueue.erase(front);
                front = mCore->mQueue.begin();
            }

            bool bufferIsDue = desiredPresent <= expectedPresent ||
                    desiredPresent > expectedPresent + MAX_REASONABLE_NSEC;
            bool consumerIsReady = maxFrameNumber > 0 ?
                    front->mFrameNumber <= maxFrameNumber : true;
            if (!bufferIsDue || !consumerIsReady) {

                return PRESENT_LATER;
            }

        }

这里主要做了一些几件事:

  • expectedPresent 期望被显示的时间
    也就是这个Buffer希望在什么时候被显示到屏幕上。如果Buffer的DesiredPresent的时间早于这个时间,那么这个Buffer将被准时显示。或者稍晚才被显示,如果我们不想显示直到expectedPresent时间之后,我们返回PRESENT_LATER,不去acquire它。但是如果时间在一秒之内,就不会延迟了,直接acquire回去。
  • 检查是否需要丢弃一些帧
    如果是Surface自动生成的时间,就不去检查是否需要丢弃掉一些帧,这些Surface对显示时间是没有严格的要求的。如果mQueue中有多个Buffer,我们将丢掉一些queue过来比较早的Buffer。如果最近queue的Buffer,离期望显示的时间已经没有一秒了,那之前queue过来的Buffer都将被丢弃掉。这很好理解,你好比你要买一款手机,新款的广告虽然来了,但是还有一段时间才能上市,你等不了这么就久,就先买就旧款了,总得用手机吧。但是,如果新款不到一秒就上市了,我们就稍微等会儿直接买新款,不买旧款了。
    front->mIsStale,表示Buffer已经被释放了,这是在BufferQueueCore::freeAllBuffersLocked时置的位。此时,我们需要将Buffer都返回到BufferQueue FREE状态中。

该丢弃的丢弃了,余下的就可以用来去显示了。

* frameworks/native/libs/gui/BufferQueueConsumer.cpp

        int slot = BufferQueueCore::INVALID_BUFFER_SLOT;

        if (sharedBufferAvailable && mCore->mQueue.empty()) {
            // make sure the buffer has finished allocating before acquiring it
            mCore->waitWhileAllocatingLocked();

            slot = mCore->mSharedBufferSlot;

            // Recreate the BufferItem for the shared buffer from the data that
            // was cached when it was last queued.
            outBuffer->mGraphicBuffer = mSlots[slot].mGraphicBuffer;
            outBuffer->mFence = Fence::NO_FENCE;
            outBuffer->mFenceTime = FenceTime::NO_FENCE;
            outBuffer->mCrop = mCore->mSharedBufferCache.crop;
            outBuffer->mTransform = mCore->mSharedBufferCache.transform &
                    ~static_cast<uint32_t>(
                    NATIVE_WINDOW_TRANSFORM_INVERSE_DISPLAY);
            outBuffer->mScalingMode = mCore->mSharedBufferCache.scalingMode;
            outBuffer->mDataSpace = mCore->mSharedBufferCache.dataspace;
            outBuffer->mFrameNumber = mCore->mFrameCounter;
            outBuffer->mSlot = slot;
            outBuffer->mAcquireCalled = mSlots[slot].mAcquireCalled;
            outBuffer->mTransformToDisplayInverse =
                    (mCore->mSharedBufferCache.transform &
                    NATIVE_WINDOW_TRANSFORM_INVERSE_DISPLAY) != 0;
            outBuffer->mSurfaceDamage = Region::INVALID_REGION;
            outBuffer->mQueuedBuffer = false;
            outBuffer->mIsStale = false;
            outBuffer->mAutoRefresh = mCore->mSharedBufferMode &&
                    mCore->mAutoRefresh;
        } else {
            slot = front->mSlot;
            *outBuffer = *front;
        }

如果是共享Buffer模式,即使mQueue为空,也会把共享的Buffer返回去。其他情况下就返回,mQueue的第一个Buffer。

* frameworks/native/libs/gui/BufferQueueConsumer.cpp

        ATRACE_BUFFER_INDEX(slot);

        if (!outBuffer->mIsStale) {
            mSlots[slot].mAcquireCalled = true;

            if (mCore->mQueue.empty()) {
                mSlots[slot].mBufferState.acquireNotInQueue();
            } else {
                mSlots[slot].mBufferState.acquire();
            }
            mSlots[slot].mFence = Fence::NO_FENCE;
        }

        if (outBuffer->mAcquireCalled) {
            outBuffer->mGraphicBuffer = NULL;
        }

        mCore->mQueue.erase(front);

        mCore->mDequeueCondition.broadcast();

        ATRACE_INT(mCore->mConsumerName.string(),
                static_cast<int32_t>(mCore->mQueue.size()));
        mCore->mOccupancyTracker.registerOccupancyChange(mCore->mQueue.size());

        VALIDATE_CONSISTENCY();
    }

    if (listener != NULL) {
        for (int i = 0; i < numDroppedBuffers; ++i) {
            listener->onBufferReleased();
        }
    }

    return NO_ERROR;
}

acquire到Buffer后,修改mSlots中对应Buffer序号的mBufferState状态。acquire的Buffer,需要从mQueue中 删掉。留意这里的ATRACE_INT,这个在systrace分析时,非常有用。如果Buffer被丢弃了,可以通过Producer的监听者,去通知Producer Buffer已经被release掉了。

releaseBuffer流程分析

Consumer具体怎么消费的,我们暂时不管,我们先来看消费完成后,releaseBuffer的流程。

* frameworks/native/libs/gui/BufferQueueConsumer.cpp

status_t BufferQueueConsumer::releaseBuffer(int slot, uint64_t frameNumber,
        const sp<Fence>& releaseFence, EGLDisplay eglDisplay,
        EGLSyncKHR eglFence) {
    ATRACE_CALL();
    ATRACE_BUFFER_INDEX(slot);

    if (slot < 0 || slot >= BufferQueueDefs::NUM_BUFFER_SLOTS ||
            releaseFence == NULL) {
        BQ_LOGE("releaseBuffer: slot %d out of range or fence %p NULL", slot,
                releaseFence.get());
        return BAD_VALUE;
    }

    sp<IProducerListener> listener;
    { // Autolock scope
        Mutex::Autolock lock(mCore->mMutex);

        // FrameNumber已经变,buffer已经被重新分配
        if (frameNumber != mSlots[slot].mFrameNumber &&
                !mSlots[slot].mBufferState.isShared()) {
            return STALE_BUFFER_SLOT;
        }

        if (!mSlots[slot].mBufferState.isAcquired()) {
            BQ_LOGE("releaseBuffer: attempted to release buffer slot %d "
                    "but its state was %s", slot,
                    mSlots[slot].mBufferState.string());
            return BAD_VALUE;
        }

        mSlots[slot].mEglDisplay = eglDisplay;
        mSlots[slot].mEglFence = eglFence;
        mSlots[slot].mFence = releaseFence;
        mSlots[slot].mBufferState.release();

        if (!mCore->mSharedBufferMode && mSlots[slot].mBufferState.isFree()) {
            mSlots[slot].mBufferState.mShared = false;
        }
        // Don't put the shared buffer on the free list.
        if (!mSlots[slot].mBufferState.isShared()) {
            mCore->mActiveBuffers.erase(slot);
            mCore->mFreeBuffers.push_back(slot);
        }

        listener = mCore->mConnectedProducerListener;
        BQ_LOGV("releaseBuffer: releasing slot %d", slot);

        mCore->mDequeueCondition.broadcast();
        VALIDATE_CONSISTENCY();
    } // Autolock scope

    // Call back without lock held
    if (listener != NULL) {
        listener->onBufferReleased();
    }

    return NO_ERROR;
}
  • release Buffer的流程相对简单,slot就是需要释放的Buffer的序号。
  • Buffer的FrameNumber变了,可能Buffer已经重新分配,这个是不用管。
  • 只能释放acquire状态的buffer序号,释放后是Buffer放会mFreeBuffers中。
  • releaseFence,从Consumer那边传过来,Producer可以Dequeue mFreeBuffers中的Buffer,但是只有releaseFence发信号出来后,Consumer才真正用完,Producer才可以写。
  • 同样的,可以通过listener通知Producer。

就这么多~~

小结

本章主要通过测试应用,讲解ANativeWindow,Surface间的关系,Surface和Producer,Consumer间的关系;P应用怎么使用BufferQueue。讲解了BufferQueue相关的几个流程,dequeueBuffer,queueBuffer,acquireBuffer,releaseBuffer;以及Buffer的状态,DEQUEUED,QUEUED,ACQUIRED,FREE迁移。

推荐阅读更多精彩内容