webrtc源码分析之视频编码之一

新的一年又开始了,我打算先把年前看的webrtc的流程先梳理下。省的后面忙于其他事情,时间一长又忘记了。还是尽量按webrtc开篇中的顺利来梳理吧。在之前的两篇博客中基本完成了webrtc视频采集的分析,因此本篇开始就分析一下webrtc视频编码相关的内容,准备分三篇来介绍。

废话少说,如下图所示,主要是从初始化流程和编码流程来分析webrtc视频编码模块,step1~step31是初始化流程,主要是创建相关的对象,step32~step49是编码流程,本篇文章打算先分析一下初始化流程。

上面流程涉及的主要类如下所示:

VideoStreamEncoder实现了VideoSinkInterface和EncodedImageCallback接口,当做为一个VideoSinkInterface对象时,可以接收VideoBroadcaster分发的图像。当做为一个EncodedImageCallback对象时,可以接收MediaCodecVideoEncoder编码后的码流数据。MediaCodecVideoEncoder的callback_是一个VCMEncodedFrameCallback对象,VCMEncodedFrameCallback的post_encode_callback_是一个VideoStreamEncoder对象,因此编码后的数据就可以通过callback_成员回调到VideoStreamEncoder。

创建视频发送Stream时会调用WebRtcVideoSendStream的RecreateWebRtcStream,在这个过程会完成视频编码模块初始化工作,RecreateWebRtcStream主要代码如下:

void WebRtcVideoChannel::WebRtcVideoSendStream::RecreateWebRtcStream() {
  stream_ = call_->CreateVideoSendStream(std::move(config),
                                         parameters_.encoder_config.Copy());
  if (source_) {
    stream_->SetSource(this, GetDegradationPreference());
  }
}

主要包括如下两个流程:

  1. 调用Call对象的CreateVideoSendStream创建VideoSendStream对象。
  2. 调用SetSource函数设置图像数据源,最后会调用VideoTrack的AddOrUpdateSink函数注册sink对象,该sink对象是一个VideoStreamEncoder对象。

Call的CreateVideoSendStream主要代码如下:

webrtc::VideoSendStream* Call::CreateVideoSendStream(
    webrtc::VideoSendStream::Config config,
    VideoEncoderConfig encoder_config) {
  VideoSendStream* send_stream = new VideoSendStream(
      num_cpu_cores_, module_process_thread_.get(), &worker_queue_,
      call_stats_.get(), transport_send_.get(), bitrate_allocator_.get(),
      video_send_delay_stats_.get(), event_log_, std::move(config),
      std::move(encoder_config), suspended_video_send_ssrcs_,
      suspended_video_payload_states_);
  return send_stream;
}

VideoSendStream的构造函数如下所示:

VideoSendStream::VideoSendStream(
    int num_cpu_cores,
    ProcessThread* module_process_thread,
    rtc::TaskQueue* worker_queue,
    CallStats* call_stats,
    RtpTransportControllerSendInterface* transport,
    BitrateAllocator* bitrate_allocator,
    SendDelayStats* send_delay_stats,
    RtcEventLog* event_log,
    VideoSendStream::Config config,
    VideoEncoderConfig encoder_config,
    const std::map<uint32_t, RtpState>& suspended_ssrcs,
    const std::map<uint32_t, RtpPayloadState>& suspended_payload_states)
    : worker_queue_(worker_queue),
      thread_sync_event_(false /* manual_reset */, false),
      stats_proxy_(Clock::GetRealTimeClock(),
                   config,
                   encoder_config.content_type),
      config_(std::move(config)),
      content_type_(encoder_config.content_type) {
  video_stream_encoder_.reset(
      new VideoStreamEncoder(num_cpu_cores, &stats_proxy_,
                             config_.encoder_settings,
                             config_.pre_encode_callback,
                             std::unique_ptr<OveruseFrameDetector>()));
  worker_queue_->PostTask(std::unique_ptr<rtc::QueuedTask>(new ConstructionTask(
      &send_stream_, &thread_sync_event_, &stats_proxy_,
      video_stream_encoder_.get(), module_process_thread, call_stats, transport,
      bitrate_allocator, send_delay_stats, event_log, &config_,
      encoder_config.max_bitrate_bps, suspended_ssrcs, suspended_payload_states,
      encoder_config.content_type)));

  // Wait for ConstructionTask to complete so that |send_stream_| can be used.
  // |module_process_thread| must be registered and deregistered on the thread
  // it was created on.
  thread_sync_event_.Wait(rtc::Event::kForever);
  send_stream_->RegisterProcessThread(module_process_thread);
  // TODO(sprang): Enable this also for regular video calls if it works well.
  if (encoder_config.content_type == VideoEncoderConfig::ContentType::kScreen) {
    // Only signal target bitrate for screenshare streams, for now.
    video_stream_encoder_->SetBitrateObserver(send_stream_.get());
  }

  ReconfigureVideoEncoder(std::move(encoder_config));
}

主要是创建了VideoStreamEncoder对象和VideoSendStreamImpl对象,然后调用ReconfigureVideoEncoder初始化编码器,VideoSendStreamImpl对象的创建是在ConstructionTask的Run函数中完成的,如下所示:

  bool Run() override {
    send_stream_->reset(new VideoSendStreamImpl(
        stats_proxy_, rtc::TaskQueue::Current(), call_stats_, transport_,
        bitrate_allocator_, send_delay_stats_, video_stream_encoder_,
        event_log_, config_, initial_encoder_max_bitrate_,
        std::move(suspended_ssrcs_), std::move(suspended_payload_states_),
        content_type_));
    return true;
  }

VideoStreamEncoder的构造函数如下所示:

VideoStreamEncoder::VideoStreamEncoder(
    uint32_t number_of_cores,
    SendStatisticsProxy* stats_proxy,
    const VideoSendStream::Config::EncoderSettings& settings,
    rtc::VideoSinkInterface<VideoFrame>* pre_encode_callback,
    std::unique_ptr<OveruseFrameDetector> overuse_detector)
    : shutdown_event_(true /* manual_reset */, false),
      number_of_cores_(number_of_cores),
      initial_rampup_(0),
      source_proxy_(new VideoSourceProxy(this)),
      sink_(nullptr),
      settings_(settings),
      codec_type_(PayloadStringToCodecType(settings.payload_name)),
      video_sender_(Clock::GetRealTimeClock(), this),
      overuse_detector_(
          overuse_detector.get()
              ? overuse_detector.release()
              : new OveruseFrameDetector(
                    GetCpuOveruseOptions(settings.full_overuse_time),
                    this,
                    stats_proxy)),
      stats_proxy_(stats_proxy),
      pre_encode_callback_(pre_encode_callback),
      max_framerate_(-1),
      pending_encoder_reconfiguration_(false),
      encoder_start_bitrate_bps_(0),
      max_data_payload_length_(0),
      nack_enabled_(false),
      last_observed_bitrate_bps_(0),
      encoder_paused_and_dropped_frame_(false),
      clock_(Clock::GetRealTimeClock()),
      degradation_preference_(
          VideoSendStream::DegradationPreference::kDegradationDisabled),
      posted_frames_waiting_for_encode_(0),
      last_captured_timestamp_(0),
      delta_ntp_internal_ms_(clock_->CurrentNtpInMilliseconds() -
                             clock_->TimeInMilliseconds()),
      last_frame_log_ms_(clock_->TimeInMilliseconds()),
      captured_frame_count_(0),
      dropped_frame_count_(0),
      bitrate_observer_(nullptr),
      encoder_queue_("EncoderQueue") {
  RTC_DCHECK(stats_proxy);
  encoder_queue_.PostTask([this] {
    RTC_DCHECK_RUN_ON(&encoder_queue_);
    overuse_detector_->StartCheckForOveruse();
    video_sender_.RegisterExternalEncoder(
        settings_.encoder, settings_.payload_type, settings_.internal_source);
  });
}

主要是初始化source_proxy_对象和video_sender_对象,VideoSender构造函数如下所示:

VideoSender::VideoSender(Clock* clock,
                         EncodedImageCallback* post_encode_callback)
    : _encoder(nullptr),
      _mediaOpt(clock),
      _encodedFrameCallback(post_encode_callback, &_mediaOpt),
      post_encode_callback_(post_encode_callback),
      _codecDataBase(&_encodedFrameCallback),
      frame_dropper_enabled_(true),
      current_codec_(),
      encoder_params_({BitrateAllocation(), 0, 0, 0}),
      encoder_has_internal_source_(false),
      next_frame_types_(1, kVideoFrameDelta) {
  _mediaOpt.Reset();
  // Allow VideoSender to be created on one thread but used on another, post
  // construction. This is currently how this class is being used by at least
  // one external project (diffractor).
  sequenced_checker_.Detach();
}

主要是初始化_encodedFrameCallback和codecDataBase对象,VCMEncodedFrameCallback构造函数如下所示,
post_encode_callback
是一个VideoStreamEncoder对象:

VCMEncodedFrameCallback::VCMEncodedFrameCallback(
    EncodedImageCallback* post_encode_callback,
    media_optimization::MediaOptimization* media_opt)
    : internal_source_(false),
      post_encode_callback_(post_encode_callback),
      media_opt_(media_opt),
      framerate_(1),
      last_timing_frame_time_ms_(-1),
      timing_frames_thresholds_({-1, 0}),
      incorrect_capture_time_logged_messages_(0),
      reordered_frames_logged_messages_(0),
      stalled_encoder_logged_messages_(0) {
}

VCMCodecDataBase构造函数如下所示,encoded_frame_callback_是一个VCMEncodedFrameCallback对象:

VCMCodecDataBase::VCMCodecDataBase(
    VCMEncodedFrameCallback* encoded_frame_callback)
    : number_of_cores_(0),
      max_payload_size_(kDefaultPayloadSize),
      periodic_key_frames_(false),
      pending_encoder_reset_(true),
      send_codec_(),
      receive_codec_(),
      encoder_payload_type_(0),
      external_encoder_(nullptr),
      internal_source_(false),
      encoded_frame_callback_(encoded_frame_callback),
      dec_map_(),
      dec_external_map_() {}

在VideoStreamEncoder的构造函数中,调用VideoSender的RegisterExternalEncoder函数最终将编码器对象保存在VCMCodecDataBase的external_encoder_成员,如下所示:

void VCMCodecDataBase::RegisterExternalEncoder(VideoEncoder* external_encoder,
                                               uint8_t payload_type,
                                               bool internal_source) {
  // Since only one encoder can be used at a given time, only one external
  // encoder can be registered/used.
  external_encoder_ = external_encoder;
  encoder_payload_type_ = payload_type;
  internal_source_ = internal_source;
  pending_encoder_reset_ = true;
}

编码器对象是通过WebRtcVideoEncoderFactory创建的,如下所示:

void WebRtcVideoChannel::WebRtcVideoSendStream::SetCodec(
    const VideoCodecSettings& codec_settings,
    bool force_encoder_allocation) {
  std::unique_ptr<webrtc::VideoEncoder> new_encoder;
  if (force_encoder_allocation || !allocated_encoder_ ||
      allocated_codec_ != codec_settings.codec) {
    const webrtc::SdpVideoFormat format(codec_settings.codec.name,
                                        codec_settings.codec.params);
    new_encoder = encoder_factory_->CreateVideoEncoder(format);

    parameters_.config.encoder_settings.encoder = new_encoder.get();

    const webrtc::VideoEncoderFactory::CodecInfo info =
        encoder_factory_->QueryVideoEncoder(format);
    parameters_.config.encoder_settings.full_overuse_time =
        info.is_hardware_accelerated;
    parameters_.config.encoder_settings.internal_source =
        info.has_internal_source;
  } else {
    new_encoder = std::move(allocated_encoder_);
  }
  parameters_.config.encoder_settings.payload_name = codec_settings.codec.name;
  parameters_.config.encoder_settings.payload_type = codec_settings.codec.id;
}

以Android MediaCodec编码器为例,MediaCodecVideoEncoderFactory的CreateVideoEncoder定义如下:

VideoEncoder* MediaCodecVideoEncoderFactory::CreateVideoEncoder(
    const cricket::VideoCodec& codec) {
  if (supported_codecs().empty()) {
    ALOGW << "No HW video encoder for codec " << codec.name;
    return nullptr;
  }
  if (FindMatchingCodec(supported_codecs(), codec)) {
    ALOGD << "Create HW video encoder for " << codec.name;
    JNIEnv* jni = AttachCurrentThreadIfNeeded();
    ScopedLocalRefFrame local_ref_frame(jni);
    return new MediaCodecVideoEncoder(jni, codec, egl_context_);
  }
  ALOGW << "Can not find HW video encoder for type " << codec.name;
  return nullptr;
}

可见VCMCodecDataBase的external_encoder_是一个MediaCodecVideoEncoder对象。

回到VideoSendStream的构造函数中,调用的ReconfigureVideoEncoder最后会调用VCMCodecDataBase的SetSendCodec函数,如下所示,主要是创建并初始化VCMGenericEncoder对象,其中external_encoder_是一个MediaCodecVideoEncoder对象,encoded_frame_callback_是一个VCMEncodedFrameCallback对象。

bool VCMCodecDataBase::SetSendCodec(const VideoCodec* send_codec,
                                    int number_of_cores,
                                    size_t max_payload_size) {
  ptr_encoder_.reset(new VCMGenericEncoder(
      external_encoder_, encoded_frame_callback_, internal_source_));
  encoded_frame_callback_->SetInternalSource(internal_source_);
  if (ptr_encoder_->InitEncode(&send_codec_, number_of_cores_,
                               max_payload_size_) < 0) {
    RTC_LOG(LS_ERROR) << "Failed to initialize video encoder.";
    DeleteEncoder();
    return false;
  }
}

VCMGenericEncoder的构造函数如下所示,encoder_和vcm_encoded_frame_callback_保存的分别是MediaCodecVideoEncoder对象和VCMEncodedFrameCallback对象。

VCMGenericEncoder::VCMGenericEncoder(
    VideoEncoder* encoder,
    VCMEncodedFrameCallback* encoded_frame_callback,
    bool internal_source)
    : encoder_(encoder),
      vcm_encoded_frame_callback_(encoded_frame_callback),
      internal_source_(internal_source),
      encoder_params_({BitrateAllocation(), 0, 0, 0}),
      streams_or_svc_num_(0) {}

VCMGenericEncoder的InitEncode函数定义如下所示:

int32_t VCMGenericEncoder::InitEncode(const VideoCodec* settings,
                                      int32_t number_of_cores,
                                      size_t max_payload_size) {
  RTC_DCHECK_RUNS_SERIALIZED(&race_checker_);
  TRACE_EVENT0("webrtc", "VCMGenericEncoder::InitEncode");
  streams_or_svc_num_ = settings->numberOfSimulcastStreams;
  codec_type_ = settings->codecType;
  if (settings->codecType == kVideoCodecVP9) {
    streams_or_svc_num_ = settings->VP9().numberOfSpatialLayers;
  }
  if (streams_or_svc_num_ == 0)
    streams_or_svc_num_ = 1;

  vcm_encoded_frame_callback_->SetTimingFramesThresholds(
      settings->timing_frame_thresholds);
  vcm_encoded_frame_callback_->OnFrameRateChanged(settings->maxFramerate);

  if (encoder_->InitEncode(settings, number_of_cores, max_payload_size) != 0) {
    RTC_LOG(LS_ERROR) << "Failed to initialize the encoder associated with "
                         "payload name: "
                      << settings->plName;
    return -1;
  }
  vcm_encoded_frame_callback_->Reset();
  encoder_->RegisterEncodeCompleteCallback(vcm_encoded_frame_callback_);
  return 0;
}

主要是调用MediaCodecVideoEncoder的InitEncode函数创建并初始化编码器,并将VCMEncodedFrameCallback注册到MediaCodecVideoEncoder中用来接收编码后的数据。

InitEncode函数最后会调用java层MediaCodecVideoEncoder的initEncode函数,如下所示,在这里完成了创建并初始化Android MediaCodec编码器的工作。

  @CalledByNativeUnchecked
  boolean initEncode(VideoCodecType type, int profile, int width, int height, int kbps, int fps,
      EglBase14.Context sharedContext) {
    final boolean useSurface = sharedContext != null;
    Logging.d(TAG,
        "Java initEncode: " + type + ". Profile: " + profile + " : " + width + " x " + height
            + ". @ " + kbps + " kbps. Fps: " + fps + ". Encode from texture : " + useSurface);

    this.profile = profile;
    this.width = width;
    this.height = height;
    if (mediaCodecThread != null) {
      throw new RuntimeException("Forgot to release()?");
    }
    EncoderProperties properties = null;
    String mime = null;
    int keyFrameIntervalSec = 0;
    boolean configureH264HighProfile = false;
    if (type == VideoCodecType.VIDEO_CODEC_VP8) {
      mime = VP8_MIME_TYPE;
      properties = findHwEncoder(
          VP8_MIME_TYPE, vp8HwList(), useSurface ? supportedSurfaceColorList : supportedColorList);
      keyFrameIntervalSec = 100;
    } else if (type == VideoCodecType.VIDEO_CODEC_VP9) {
      mime = VP9_MIME_TYPE;
      properties = findHwEncoder(
          VP9_MIME_TYPE, vp9HwList, useSurface ? supportedSurfaceColorList : supportedColorList);
      keyFrameIntervalSec = 100;
    } else if (type == VideoCodecType.VIDEO_CODEC_H264) {
      mime = H264_MIME_TYPE;
      properties = findHwEncoder(
          H264_MIME_TYPE, h264HwList, useSurface ? supportedSurfaceColorList : supportedColorList);
      if (profile == H264Profile.CONSTRAINED_HIGH.getValue()) {
        EncoderProperties h264HighProfileProperties = findHwEncoder(H264_MIME_TYPE,
            h264HighProfileHwList, useSurface ? supportedSurfaceColorList : supportedColorList);
        if (h264HighProfileProperties != null) {
          Logging.d(TAG, "High profile H.264 encoder supported.");
          configureH264HighProfile = true;
        } else {
          Logging.d(TAG, "High profile H.264 encoder requested, but not supported. Use baseline.");
        }
      }
      keyFrameIntervalSec = 20;
    }
    if (properties == null) {
      throw new RuntimeException("Can not find HW encoder for " + type);
    }
    runningInstance = this; // Encoder is now running and can be queried for stack traces.
    colorFormat = properties.colorFormat;
    bitrateAdjustmentType = properties.bitrateAdjustmentType;
    if (bitrateAdjustmentType == BitrateAdjustmentType.FRAMERATE_ADJUSTMENT) {
      fps = BITRATE_ADJUSTMENT_FPS;
    } else {
      fps = Math.min(fps, MAXIMUM_INITIAL_FPS);
    }

    forcedKeyFrameMs = 0;
    lastKeyFrameMs = -1;
    if (type == VideoCodecType.VIDEO_CODEC_VP8
        && properties.codecName.startsWith(qcomVp8HwProperties.codecPrefix)) {
      if (Build.VERSION.SDK_INT == Build.VERSION_CODES.LOLLIPOP
          || Build.VERSION.SDK_INT == Build.VERSION_CODES.LOLLIPOP_MR1) {
        forcedKeyFrameMs = QCOM_VP8_KEY_FRAME_INTERVAL_ANDROID_L_MS;
      } else if (Build.VERSION.SDK_INT == Build.VERSION_CODES.M) {
        forcedKeyFrameMs = QCOM_VP8_KEY_FRAME_INTERVAL_ANDROID_M_MS;
      } else if (Build.VERSION.SDK_INT > Build.VERSION_CODES.M) {
        forcedKeyFrameMs = QCOM_VP8_KEY_FRAME_INTERVAL_ANDROID_N_MS;
      }
    }

    Logging.d(TAG, "Color format: " + colorFormat + ". Bitrate adjustment: " + bitrateAdjustmentType
            + ". Key frame interval: " + forcedKeyFrameMs + " . Initial fps: " + fps);
    targetBitrateBps = 1000 * kbps;
    targetFps = fps;
    bitrateAccumulatorMax = targetBitrateBps / 8.0;
    bitrateAccumulator = 0;
    bitrateObservationTimeMs = 0;
    bitrateAdjustmentScaleExp = 0;

    mediaCodecThread = Thread.currentThread();
    try {
      MediaFormat format = MediaFormat.createVideoFormat(mime, width, height);
      format.setInteger(MediaFormat.KEY_BIT_RATE, targetBitrateBps);
      format.setInteger("bitrate-mode", VIDEO_ControlRateConstant);
      format.setInteger(MediaFormat.KEY_COLOR_FORMAT, properties.colorFormat);
      format.setInteger(MediaFormat.KEY_FRAME_RATE, targetFps);
      format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, keyFrameIntervalSec);
      if (configureH264HighProfile) {
        format.setInteger("profile", VIDEO_AVCProfileHigh);
        format.setInteger("level", VIDEO_AVCLevel3);
      }
      Logging.d(TAG, "  Format: " + format);
      mediaCodec = createByCodecName(properties.codecName);
      this.type = type;
      if (mediaCodec == null) {
        Logging.e(TAG, "Can not create media encoder");
        release();
        return false;
      }
      mediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);

      if (useSurface) {
        eglBase = new EglBase14(sharedContext, EglBase.CONFIG_RECORDABLE);
        // Create an input surface and keep a reference since we must release the surface when done.
        inputSurface = mediaCodec.createInputSurface();
        eglBase.createSurface(inputSurface);
        drawer = new GlRectDrawer();
      }
      mediaCodec.start();
      outputBuffers = mediaCodec.getOutputBuffers();
      Logging.d(TAG, "Output buffers: " + outputBuffers.length);

    } catch (IllegalStateException e) {
      Logging.e(TAG, "initEncode failed", e);
      release();
      return false;
    }
    return true;
  }

MediaCodecVideoEncoder的RegisterEncodeCompleteCallback定义如下所示,可见callback_成员是一个VCMEncodedFrameCallback对象。

int32_t MediaCodecVideoEncoder::RegisterEncodeCompleteCallback(
    EncodedImageCallback* callback) {
  RTC_DCHECK_CALLED_SEQUENTIALLY(&encoder_queue_checker_);
  JNIEnv* jni = AttachCurrentThreadIfNeeded();
  ScopedLocalRefFrame local_ref_frame(jni);
  callback_ = callback;
  return WEBRTC_VIDEO_CODEC_OK;
}

回到WebRtcVideoSendStream的RecreateWebRtcStream函数,会调用VideoSendStream的SetSource设置Source,最后会在VideoSourceProxy的SetSource函数中调用WebRtcVideoSendStream的AddOrUpdateSink函数将VideoStreamEncoder这个sink对象注册到Source上,这样Source的图像数据就可以分发到VideoStreamEncoder对象进行编码了。

总结

本篇文章主要分析了webrtc视频编码模块的初始化流程,这个流程就是创建一系列相关的对象,然后编码器设置好输入输出,VideoStreamEncoder对象负责输入输出的衔接,编码器的输入是通过将VideoStreamEncoder注册到VideoTrack来完成图像数据的接收,此时VideoStreamEncoder是做为一个VideoSinkInterface对象,编码器的输出是通过将VCMEncodedFrameCallback注册到MediaCodecVideoEncoder,再经过VideoStreamEncoder来完成码流数据的打包和传输,这工作是交给VideoSendStreamImpl来完成的,此时VideoStreamEncoder是做为一个EncodedImageCallback对象。

推荐阅读更多精彩内容