视频和图片序列帧互相转换

工程地址(https://github.com/tujinqiu/KTMovieImagesTransfer)

如果你有一些连续的图片序列,那么把它转换成MP4再放到网络上传输是一个好的选择,因为size会小很多。从视频里面抽取连续的图片序列也是一个偶尔会遇到的问题。我分别尝试使用iOS原生的API,FFmpeg和OpenCV来解决这两个问题。

1、原生方法

使用原生方法主要是利用AVFoundation框架的api进行转换的。
1、将视频解成序列帧

- (NSError *)nativeTransferMovie:(NSString *)movie toImagesAtPath:(NSString *)imagesPath
{
    AVURLAsset *asset = [[AVURLAsset alloc] initWithURL:[NSURL fileURLWithPath:movie] options:nil];
    AVAssetImageGenerator *generator = [[AVAssetImageGenerator alloc] initWithAsset:asset];
    CMTime time = asset.duration;
    NSUInteger totalFrameCount = CMTimeGetSeconds(time) * kKTImagesMovieTransferFPS;
    NSMutableArray *timesArray = [NSMutableArray arrayWithCapacity:totalFrameCount];
    for (NSUInteger ii = 0; ii < totalFrameCount; ++ii) {
        CMTime timeFrame = CMTimeMake(ii, kKTImagesMovieTransferFPS);
        NSValue *timeValue = [NSValue valueWithCMTime:timeFrame];
        [timesArray addObject:timeValue];
    }
    generator.requestedTimeToleranceBefore = kCMTimeZero;
    generator.requestedTimeToleranceAfter = kCMTimeZero;
    __block NSError *returnError = nil;
    [generator generateCGImagesAsynchronouslyForTimes:timesArray completionHandler:^(CMTime requestedTime, CGImageRef  _Nullable image, CMTime actualTime, AVAssetImageGeneratorResult result, NSError * _Nullable error) {
        switch (result) {
                
            case AVAssetImageGeneratorFailed:
                returnError = error;
                [self sendToMainThreadError:returnError];
                break;
                
            case AVAssetImageGeneratorSucceeded:
            {
                NSString *imageFile = [imagesPath stringByAppendingPathComponent:[NSString stringWithFormat:@"%lld.jpg", requestedTime.value]];
                NSData *data = UIImageJPEGRepresentation([UIImage imageWithCGImage:image], 1.0);
                if ([data writeToFile:imageFile atomically:YES]) {
                    dispatch_async(dispatch_get_main_queue(), ^{
                        if ([self.delegate respondsToSelector:@selector(transfer:didTransferedAtIndex:totalFrameCount:)]) {
                            [self.delegate transfer:self didTransferedAtIndex:requestedTime.value totalFrameCount:totalFrameCount];
                        }
                    });
                    NSUInteger index = requestedTime.value;
                    if (index == totalFrameCount - 1) {
                        if ([self.delegate respondsToSelector:@selector(transfer:didFinishedWithError:)]) {
                            [self.delegate transfer:self didFinishedWithError:nil];
                        }
                    }
                } else {
                    returnError = [self errorWithErrorCode:KTTransferWriteError object:imageFile];
                    [self sendToMainThreadError:returnError];
                    [generator cancelAllCGImageGeneration];
                }
            }
                break;
                
            default:
                break;
        }
    }];
    
    return returnError;
}

主要是利用AVAssetImageGenerator抽取图片,注意在调用generateCGImagesAsynchronouslyForTimes: completionHandler方法之前,用帧率和视频时长计算帧数。

2、将序列帧压缩为视频

- (NSError *)nativeTransferImageFiles:(NSArray<NSString *> *)imageFiles toMovie:(NSString *)movie
{
    __block NSError *returnError = nil;
    AVAssetWriter *videoWriter = [[AVAssetWriter alloc] initWithURL:[NSURL fileURLWithPath:movie] fileType:AVFileTypeQuickTimeMovie error:&returnError];
    if (returnError) {
        [self sendToMainThreadError:returnError];
        return returnError;
    }
    UIImage *firstImage = [UIImage imageWithContentsOfFile:[imageFiles firstObject]];
    if (!firstImage) {
        returnError = [self errorWithErrorCode:KTTransferReadImageError object:[imageFiles firstObject]];
        [self sendToMainThreadError:returnError];
        return returnError;
    }
    CGSize size = firstImage.size;
    // h264格式
    NSDictionary *videoSettings = @{AVVideoCodecKey: AVVideoCodecH264,
                                    AVVideoWidthKey: [NSNumber numberWithInt:size.width],
                                    AVVideoHeightKey: [NSNumber numberWithInt:size.height]};
    AVAssetWriterInput* writerInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo
                                                                         outputSettings:videoSettings];
    AVAssetWriterInputPixelBufferAdaptor *adaptor = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:writerInput sourcePixelBufferAttributes:nil];
    
    dispatch_async(KTImagesMovieTransferQueue(), ^{
        [videoWriter addInput:writerInput];
        [videoWriter startWriting];
        [videoWriter startSessionAtSourceTime:kCMTimeZero];
        UIImage *tmpImage = nil;
        NSUInteger index = 0;
        while (index < imageFiles.count) {
            if(writerInput.readyForMoreMediaData) {
                CMTime presentTime = CMTimeMake(index, kKTImagesMovieTransferFPS);
                tmpImage = [UIImage imageWithContentsOfFile:[imageFiles objectAtIndex:index]];
                if (!tmpImage) {
                    returnError = [self errorWithErrorCode:KTTransferReadImageError object:[imageFiles firstObject]];
                    [self sendToMainThreadError:returnError];
                    return;
                }
                CVPixelBufferRef buffer = [self pixelBufferFromCGImage:[tmpImage CGImage] size:size];
                if (buffer) {
                    [self appendToAdapter:adaptor pixelBuffer:buffer atTime:presentTime withInput:writerInput];
                    CFRelease(buffer);
                } else {
                    // Finish the session
                    [writerInput markAsFinished];
                    [videoWriter finishWritingWithCompletionHandler:^{
                    }];
                    returnError = [self errorWithErrorCode:KTTransferGetBufferError object:nil];
                    [self sendToMainThreadError:returnError];
                    return;
                }
            }
            dispatch_async(dispatch_get_main_queue(), ^{
                if ([self.delegate respondsToSelector:@selector(transfer:didTransferedAtIndex:totalFrameCount:)]) {
                    [self.delegate transfer:self didTransferedAtIndex:index totalFrameCount:imageFiles.count];
                }
            });
            index++;
        }
        // Finish the session
        [writerInput markAsFinished];
        [videoWriter finishWritingWithCompletionHandler:^{
            if (videoWriter.status != AVAssetWriterStatusCompleted) {
                returnError = videoWriter.error;
                [self sendToMainThreadError:returnError];
            } else {
                if ([self.delegate respondsToSelector:@selector(transfer:didFinishedWithError:)]) {
                    [self.delegate transfer:self didFinishedWithError:nil];
                }
            }
        }];
    });
    
    return returnError;
}

这里主要是利用AVAssetWriter和AVAssetWriterInput来进行图片转视频的,通过字典可以给AVAssetWriterInput设置属性,从而达到设置视频属性的办法。

2、OpenCV

1、OpenCV导入工程
直接去官网下载iOS版本的framework,然后放进工程里面即可。

Snip20161016_1.png

如果出现下面的错误:

Paste_Image.png

那么是由于Objective-C默认只支持C语言,但是并不支持C++,而OpenCV用到了C++,因此将m文件的后缀改为mm,让编译支持C++即可。
如果出现下面的错误:

Undefined symbols for architecture x86_64:
  "_jpeg_free_large", referenced from:
      _free_pool in opencv2(jmemmgr.o)
  "_jpeg_free_small", referenced from:
      _free_pool in opencv2(jmemmgr.o)
      _self_destruct in opencv2(jmemmgr.o)
  "_jpeg_get_large", referenced from:
      _alloc_large in opencv2(jmemmgr.o)
      _realize_virt_arrays in opencv2(jmemmgr.o)
  "_jpeg_get_small", referenced from:
      _jinit_memory_mgr in opencv2(jmemmgr.o)
      _alloc_small in opencv2(jmemmgr.o)
  "_jpeg_mem_available", referenced from:
      _realize_virt_arrays in opencv2(jmemmgr.o)
  "_jpeg_mem_init", referenced from:
      _jinit_memory_mgr in opencv2(jmemmgr.o)
  "_jpeg_mem_term", referenced from:
      _jinit_memory_mgr in opencv2(jmemmgr.o)
      _self_destruct in opencv2(jmemmgr.o)
  "_jpeg_open_backing_store", referenced from:
      _realize_virt_arrays in opencv2(jmemmgr.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

是缺少libjpeg库造成的,首先去这里(https://sourceforge.net/projects/libjpeg-turbo/files/ )下载安装libjpeg。安装完成之后输入命令:lipo -info /opt/libjpeg-turbo/lib/libjpeg.a,可以看到支持各个处理器架构的libjpeg.a文件的路径,将这个这个文件加入工程即可。

Paste_Image.png

如果出现下面的错误:

Undefined symbols for architecture x86_64:
  "_CMSampleBufferGetImageBuffer", referenced from:
      -[CaptureDelegate captureOutput:didOutputSampleBuffer:fromConnection:] in opencv2(cap_avfoundation.o)
      CvCaptureFile::retrieveFramePixelBuffer() in opencv2(cap_avfoundation.o)
  "_CMSampleBufferInvalidate", referenced from:
      CvCaptureFile::retrieveFramePixelBuffer() in opencv2(cap_avfoundation.o)
  "_CMTimeGetSeconds", referenced from:
      -[KTImagesMovieTransfer nativeTransferMovie:toImagesAtPath:] in KTImagesMovieTransfer.o

是缺少CoreMedia.framework造成的,添加进去即可


Snip20161025_1.png

2、将视频解成序列帧
OpenCV的方法很简练,通过while循环不断取出帧,然后存盘即可:

- (NSError *)opencvTransferMovie:(NSString *)movie toImagesAtPath:(NSString *)imagesPath
{
    __block NSError *returnError = nil;
    dispatch_async(KTImagesMovieTransferQueue(), ^{
        CvCapture *pCapture = cvCaptureFromFile(movie.UTF8String);
        // 这个函数只是读取视频头文件信息来获取帧数,因此有可能有不对的情况
        // NSUInteger totalFrameCount = cvGetCaptureProperty(pCapture, CV_CAP_PROP_FRAME_COUNT);
        // 所以采取下面的遍历两遍的办法
        NSUInteger totalFrameCount = 0;
        while (cvQueryFrame(pCapture)) {
            totalFrameCount ++;
        }
        if (pCapture) {
            cvReleaseCapture(&pCapture);
        }
        pCapture = cvCaptureFromFile(movie.UTF8String);
        NSUInteger index = 0;
        IplImage *pGrabImg = NULL;
        while ((pGrabImg = cvQueryFrame(pCapture))) {
            NSString *imagePath = [imagesPath stringByAppendingPathComponent:[NSString stringWithFormat:@"%lu.jpg", index]];
            cvSaveImage(imagePath.UTF8String, pGrabImg);
            dispatch_async(dispatch_get_main_queue(), ^{
                if ([self.delegate respondsToSelector:@selector(transfer:didTransferedAtIndex:totalFrameCount:)]) {
                    [self.delegate transfer:self didTransferedAtIndex:index totalFrameCount:totalFrameCount];
                }
            });
            index++;
        }
        if (pCapture) {
            cvReleaseCapture(&pCapture);
        }
        if (index == totalFrameCount) {
            dispatch_async(dispatch_get_main_queue(), ^{
                if ([self.delegate respondsToSelector:@selector(transfer:didFinishedWithError:)]) {
                    [self.delegate transfer:self didFinishedWithError:nil];
                }
            });
        } else {
            returnError = [self errorWithErrorCode:KTTransferOpencvWrongFrameCountError object:nil];
            [self sendToMainThreadError:returnError];
        }
    });
    
    return returnError;
}

这里要注意的是OpenCV有一个获取视频属性的函数:cvGetCaptureProperty,但是这个方法获取视频帧数往往不正确,因为OpenCV只是通过这个函数读取视频的头信息,很有可能与实际帧数不相符合。因此上面是采用2次遍历的办法来获取帧数,第一次遍历获取帧数,第二次遍历执行取帧,存盘操作。另外一个需要注意的是cvQueryFrame函数返回的IplImage并不需要释放,只需要在最后释放一次cvReleaseCapture(&pCapture)即可。

3、将序列帧压缩为视频
OpenCV图片转换为视频的办法同样很简单

- (NSError *)opencvTransferImageFiles:(NSArray<NSString *> *)imageFiles toMovie:(NSString *)movie
{
    __block NSError *returnError = nil;
    UIImage *firstImage = [UIImage imageWithContentsOfFile:[imageFiles firstObject]];
    if (!firstImage) {
        returnError = [self errorWithErrorCode:KTTransferReadImageError object:[imageFiles firstObject]];
        [self sendToMainThreadError:returnError];
        return returnError;
    }
    CvSize size = cvSize(firstImage.size.width, firstImage.size.height);
    dispatch_async(KTImagesMovieTransferQueue(), ^{
        // OpenCV由于不原生支持H264(可以用其他办法做到),这里选用MP4格式
        CvVideoWriter *pWriter = cvCreateVideoWriter(movie.UTF8String, CV_FOURCC('D', 'I', 'V', 'X'), (double)kKTImagesMovieTransferFPS, size);
        for (NSUInteger ii = 0; ii < imageFiles.count; ++ii) {
            NSString *imageFile = [imageFiles objectAtIndex:ii];
            IplImage *pImage = cvLoadImage(imageFile.UTF8String);
            if (pImage) {
                cvWriteFrame(pWriter, pImage);
                cvReleaseImage(&pImage);
                dispatch_async(dispatch_get_main_queue(), ^{
                    if ([self.delegate respondsToSelector:@selector(transfer:didTransferedAtIndex:totalFrameCount:)]) {
                        [self.delegate transfer:self didTransferedAtIndex:ii totalFrameCount:imageFiles.count];
                    }
                });
            } else {
                returnError = [self errorWithErrorCode:KTTransferReadImageError object:imageFile];
                [self sendToMainThreadError:returnError];
                return;
            }
        }
        cvReleaseVideoWriter(&pWriter);
        dispatch_async(dispatch_get_main_queue(), ^{
            if ([self.delegate respondsToSelector:@selector(transfer:didFinishedWithError:)]) {
                [self.delegate transfer:self didFinishedWithError:nil];
            }
        });
    });
    
    return returnError;
}

首先初始化一个CvVideoWriter对象,然后一帧帧往里面写就行了。这里初始化CvVideoWriter

CvVideoWriter *pWriter = cvCreateVideoWriter(movie.UTF8String, CV_FOURCC('D', 'I', 'V', 'X'), (double)kKTImagesMovieTransferFPS, size);

与前面的原生方法设置writerInput属性相比较,其实非常相似,都是设置文件名,格式,大小。

NSDictionary *videoSettings = @{AVVideoCodecKey: AVVideoCodecH264, AVVideoWidthKey: [NSNumber numberWithInt:size.width], AVVideoHeightKey: [NSNumber numberWithInt:size.height]}; 
AVAssetWriterInput* writerInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:videoSettings];

3、FFmpeg

1、FFmpeg的编译和配置
FFmpeg稍显复杂一点,参照这个教程
将编译好的FFmpeg加入工程之后,除了上面这个教程所说的设置header search pathes和lib库之外,如果遇到下面的错误:

Undefined symbols for architecture arm64:
  "av_register_all()", referenced from:
      -[KTImagesMovieTransfer ffmpegTransferMovie:toImagesAtPath:] in KTImagesMovieTransfer.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

这是因为C语言的函数无法在C++中调用识别的原因,那么引用头文件的时候请这样引用(与在windows上用VS写C++遇到的问题一样的解决办法):extern "C"的作用是让括号包住的头文件中的符号以C语言的方式进行编译。这里之所以使用.mm后缀支持C++是因为之前需要支持OpenCV。如果把FFmpeg的转换方法和OpenCV的分文件写,就不存在这个问题。

extern "C"
{
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
}

未完待续。。。。

最后编辑于
©著作权归作者所有,转载或内容合作请联系作者
  • 序言:七十年代末,一起剥皮案震惊了整个滨河市,随后出现的几起案子,更是在滨河造成了极大的恐慌,老刑警刘岩,带你破解...
    沈念sama阅读 157,538评论 4 361
  • 序言:滨河连续发生了三起死亡事件,死亡现场离奇诡异,居然都是意外死亡,警方通过查阅死者的电脑和手机,发现死者居然都...
    沈念sama阅读 66,800评论 1 290
  • 文/潘晓璐 我一进店门,熙熙楼的掌柜王于贵愁眉苦脸地迎上来,“玉大人,你说我怎么就摊上这事。” “怎么了?”我有些...
    开封第一讲书人阅读 107,329评论 0 238
  • 文/不坏的土叔 我叫张陵,是天一观的道长。 经常有香客问我,道长,这世上最难降的妖魔是什么? 我笑而不...
    开封第一讲书人阅读 43,725评论 0 203
  • 正文 为了忘掉前任,我火速办了婚礼,结果婚礼上,老公的妹妹穿的比我还像新娘。我一直安慰自己,他们只是感情好,可当我...
    茶点故事阅读 52,089评论 3 286
  • 文/花漫 我一把揭开白布。 她就那样静静地躺着,像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上,一...
    开封第一讲书人阅读 40,449评论 1 212
  • 那天,我揣着相机与录音,去河边找鬼。 笑死,一个胖子当着我的面吹牛,可吹牛的内容都是我干的。 我是一名探鬼主播,决...
    沈念sama阅读 31,758评论 2 311
  • 文/苍兰香墨 我猛地睁开眼,长吁一口气:“原来是场噩梦啊……” “哼!你这毒妇竟也来了?” 一声冷哼从身侧响起,我...
    开封第一讲书人阅读 30,448评论 0 195
  • 序言:老挝万荣一对情侣失踪,失踪者是张志新(化名)和其女友刘颖,没想到半个月后,有当地人在树林里发现了一具尸体,经...
    沈念sama阅读 34,152评论 1 239
  • 正文 独居荒郊野岭守林人离奇死亡,尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 30,432评论 2 244
  • 正文 我和宋清朗相恋三年,在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 31,933评论 1 258
  • 序言:一个原本活蹦乱跳的男人离奇死亡,死状恐怖,灵堂内的尸体忽然破棺而出,到底是诈尸还是另有隐情,我是刑警宁泽,带...
    沈念sama阅读 28,289评论 2 252
  • 正文 年R本政府宣布,位于F岛的核电站,受9级特大地震影响,放射性物质发生泄漏。R本人自食恶果不足惜,却给世界环境...
    茶点故事阅读 32,921评论 3 234
  • 文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹,春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 26,023评论 0 8
  • 文/苍兰香墨 我抬头看了看天上的太阳。三九已至,却和暖如春,着一层夹袄步出监牢的瞬间,已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 26,781评论 0 192
  • 我被黑心中介骗来泰国打工, 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留,地道东北人。 一个月前我还...
    沈念sama阅读 35,477评论 2 270
  • 正文 我出身青楼,却偏偏与公主长得像,于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子,可洞房花烛夜当晚...
    茶点故事阅读 35,381评论 2 263

推荐阅读更多精彩内容