上篇文章已经简单了说了一下人脸识别的流程，那接下来的篇幅，我将详细的写出整个流程中用到的Library 和 具体的代码，让大家更直观的感受到整个人脸识别的检测过程
回顾一下人脸识别的过程
这里我将标注出用到的 Library
- 1.发现人脸 (ios-AVFoundation+MTCNN Android-MTCNN)
- 2.关键点检测 （MTCNN）
- 3.确定人脸姿态 （关键点计算，这里可以不用管，因为我用了9个方向的特征，比如说左转的姿态就用底库中左脸的特征点来比较，这里只做正脸检测）
- 4.特征抽取 （MobileFaceNet模型）
- 5.根据姿态对比相应的特征 （特征余弦相似度）
单张图片特征抽取的流程
image.png
1.发现人脸 && 关键点抽取
即 找到人脸的位置，这里iOS端多用了一个 用AVFoundation来检测人脸，为什么这么做呢？
因为MTCNN里面是3个网络，跑起来CPU很高，手机发热，耗电量也高，为了不一直去跑MTCNN，所以先用原生方法检测人脸，因为原生检测人脸效率更高，也不是特别发热，当检测到人脸后再丢给MTCNN检测，拿出关键点；
Android 因为系统版本差异太大，没有用原生，直接丢进MTCNN，检测出人脸框和关键点
- iOS代码 检测最大人脸，返回格式[关键点(array),人脸Rect(value)]
- (NSArray *)detectMaxFace:(UIImage *)image
{
int w = image.size.width;
int h = image.size.height;
unsigned char* rgba = new unsigned char[w*h*4];
{
CGColorSpaceRef colorSpace = CGImageGetColorSpace(image.CGImage);
CGContextRef contextRef = CGBitmapContextCreate(rgba, w, h, 8, w*4,
colorSpace,
kCGImageAlphaNoneSkipLast | kCGBitmapByteOrderDefault);
CGContextDrawImage(contextRef, CGRectMake(0, 0, w, h), image.CGImage);
CGContextRelease(contextRef);
}
ncnn::Mat ncnn_img;
ncnn_img = ncnn::Mat::from_pixels(rgba, ncnn::Mat::PIXEL_RGBA2RGB, w, h);
std::vector<Bbox> finalBbox;
// new_mtcnn->detect(ncnn_img, finalBbox);
cv::Mat temp;
UIImageToMat(image, temp);
// float cost;
new_mtcnn->detectMaxFace(ncnn_img, finalBbox);
int32_t num_face = static_cast<int32_t>(finalBbox.size());
int out_size = 1+num_face*14;
NSMutableArray *faceInfoArr = [NSMutableArray arrayWithCapacity:0];
//
int *faceInfo = new int[out_size];
faceInfo[0] = num_face;
for(int i=0;i<num_face;i++){
NSMutableArray *points = [NSMutableArray arrayWithCapacity:0];
CGRect rect = CGRectMake(finalBbox[i].x1, finalBbox[i].y1, finalBbox[i].x2 - finalBbox[i].x1, finalBbox[i].y2 - finalBbox[i].y1);
for (int j =0;j<5;j++){
CGPoint point = CGPointMake(finalBbox[i].ppoint[j], finalBbox[i].ppoint[j + 5]);
[points addObject:[NSValue valueWithCGPoint:point]];
}
[faceInfoArr addObject:points];
[faceInfoArr addObject:[NSValue valueWithCGRect:rect]];
}
delete [] rgba;
delete [] faceInfo;
finalBbox.clear();
return faceInfoArr;
}
2.判断人脸姿态是否为正脸
因为我们要抽取特征的脸 一定要是正脸 所抽取的特征才有意义
image.png
上图为MTCNN抽取出来的关键点，同时在数组里的位置我也有标注出来,主要比的就是眼睛到鼻子的距离
- (BOOL)faceFront:(NSArray *)shape
{
int mStartNumRightTemp = 0;
int mStartNumLeftTemp = 0;
CGPoint right_pupil;
CGPoint left_pupil;
left_pupil.x = [shape[0] CGPointValue].x ; //左眼瞳孔坐标x
left_pupil.y = [shape[0] CGPointValue].y ; //左眼瞳孔坐标y
right_pupil.x = [shape[1] CGPointValue].x ; //右眼瞳孔坐标x
right_pupil.y = [shape[1] CGPointValue].y ; //右眼瞳孔坐标y
mStartNumRightTemp = abs(right_pupil.x - [shape[2] CGPointValue].x); //到鼻子中间的距离
mStartNumLeftTemp = abs([shape[2] CGPointValue].x - left_pupil.x);
float tempp1 = mStartNumRightTemp / (float)mStartNumLeftTemp;
float tempp2 = mStartNumLeftTemp / (float)mStartNumRightTemp;
if (tempp1 > 0.7 && tempp2>0.7)
return YES;
else
return NO;
}
3.人脸对齐 （openCV）
所谓人脸对齐，看下面的图就知道什么是人脸对齐了
image.png
左边如果是原图的话，对齐后的效果就是右边这样，旋转了一下图像
下面的方法包括了 旋转与抠出人脸图像，
输入整张图片，人脸的5个关键点，返回对齐后 并截取了脸部的图
- (Mat)getAlignMat:(Mat)bgrmat landmarks:(NSArray *)landmarks
{
//left eye
float left_eye_x = [landmarks[0] CGPointValue].x;
float left_eye_y = [landmarks[0] CGPointValue].y;
//right eye
float right_eye_x = [landmarks[1] CGPointValue].x;
float right_eye_y = [landmarks[1] CGPointValue].y;
//nose
float nose_x = [landmarks[2] CGPointValue].x;
float nose_y = [landmarks[2] CGPointValue].y;
//mouth left
float mouth_left_x = [landmarks[3] CGPointValue].x;
float mouth_left_y = [landmarks[3] CGPointValue].y;
//mouth right
float mouth_right_x = [landmarks[4] CGPointValue].x;
float mouth_right_y = [landmarks[4] CGPointValue].y;
//mouth center
float mouth_center_x = (mouth_left_x + mouth_right_x) / 2;
float mouth_center_y = (mouth_left_y + mouth_right_y) / 2;
cv::Mat affineMat;
std::vector<cv::Point2f> src_pts ;
src_pts.push_back(cv::Point2f(left_eye_x, left_eye_y));
src_pts.push_back(cv::Point2f(right_eye_x, right_eye_y));
src_pts.push_back(cv::Point2f(mouth_center_x, mouth_center_y));
cv::Point2f left_eye(38, 52);
cv::Point2f right_eye(74, 52);
cv::Point2f mouth_center(56, 92);
cv::Size dsize(112, 112);
std::vector<cv::Point2f> dst_pts;
dst_pts.push_back(left_eye);
dst_pts.push_back(right_eye);
dst_pts.push_back(mouth_center);
affineMat = cv::getAffineTransform(src_pts, dst_pts);
cv::Mat alignedImg;
cv::warpAffine(bgrmat, alignedImg, affineMat, dsize, cv::INTER_CUBIC, cv::BORDER_REPLICATE);
return alignedImg;
}
4.抽取特征 （MobileFaceNet）
ncnn::Mat NCNNNet::getFaceFeatures(cv::Mat face)
{
ncnn::Mat in = ncnn::Mat::from_pixels(face.data, ncnn::Mat::PIXEL_BGR, face.cols, face.rows);
const float mean_vals[3] = {127.5f, 127.5f, 127.5f};
const float std_vals[3] = {0.0078125f, 0.0078125f, 0.0078125f};
in.substract_mean_normalize(mean_vals, std_vals);
ncnn::Extractor ex = features.create_extractor();
ex.set_light_mode(true);
ex.set_num_threads(4);
ex.input(mobilefacenet_proto_id::BLOB_data, in);
ncnn::Mat out;
ex.extract(mobilefacenet_proto_id::BLOB_fc1_fc1_scale, out);
return out;
}
抽取出来的为128个float的数据，我们需要将ncnn:Mat 转换成你要的Array，然后再进行相似度计算,这里以iOS端为🌰
- (NSArray <NSNumber *>*)turnNCNNMat:(ncnn::Mat)feature
{
NSMutableArray *features = [NSMutableArray arrayWithCapacity:0];
for (int i = 0; i < feature.w; i++) {
[features addObject:@((float)feature[i])];
}
return features;
}
同时为了提高运算精度，我们需要将抽取的特征数据归一化，这里得到的就是我们需要的人脸特征了
- (NSArray *)normalizeFeature:(NSArray <NSNumber *>*)feature
{
NSMutableArray *fixFArr = [NSMutableArray arrayWithCapacity:0];
if (!feature.count) return nil;
float norm = 0;
for (int i = 0; i < feature.count; i++) {
norm += [feature[i] floatValue] * [feature[i] floatValue];
}
norm = sqrt(norm);
if (norm == 0) return feature;
for (int i = 0; i < feature.count; i++) {
float newFeature = [feature[i] floatValue] / norm;
[fixFArr addObject:@(newFeature)];
}
return fixFArr;
}
5.人脸相似度比对（余弦相似度）
通过以上步骤，我们已经可以抽取一张人脸的特征了，如果我们抽取两张，就可以得到两张人脸的信息，这时候，我们就可以进行对比的操作了
- (CGFloat)getFeatureDistanceWithFirstFeatures:(NSArray *)firstFeature second:(NSArray *)secondFeature
{
if (!firstFeature.count || !secondFeature.count) return 0;
float distance = 0;
for (int i = 0; i < firstFeature.count && i < firstFeature.count; ++i) {
distance += ([firstFeature[i] floatValue] - [secondFeature[i] floatValue]) * ([firstFeature[i] floatValue]- [secondFeature[i] floatValue]);
}
distance = sqrt(distance);
CGFloat similarity = (1 - pow(distance / 2, 2)) * 100;
return similarity;
}
