验证码识别是3年前的一个小愿望了（当时是做一个自动回帖器，抽奖iphone），但自己这两年主要在做分布式架构，今年终于抽出了空，又战胜了对数学的恐惧，在coursera上学习了吴恩达的机器学习和深度学习，验证码识别也算是对部分课程的实践，下面就来整理一下这次识别的过程。
1. 验证码识别主流程
- 目标检测，检测出字符边距，主要是获得weight，并输出坐标
- 图片黑白处理
- cnn识别算法
2. 识别细节
2.1 目标检测，检测出验证码图片中的字符边距
这个部分使用了yolov2算法，下面介绍一下yolov2算法：
A. 理论：yolov2算法整体来说其实是把图片分成一个一个小格子，然后每个格子会有一个输出<pi, x, y, dx, dy> 分别代表这个格子是类别i的概率，x，y代表这个格子作为这个类别的中心位置，dx，dy代表以x，y为中心位置的目标长宽。
B. 实践：yolo的练习我仅仅在cousera上面进行了原生练习，但是在做验证码识别的时候我使用了一个开源版本darknet，最后租用了百度的5块钱/1小时的gpu进行训练，标记用labelImg进行了边框标记，下面主要介绍一下这两个工具的使用过程：
2.1.1 labelImg使用
http://blog.csdn.net/dcrmg/article/details/78496002
首先，采用上述博客的方法对验证码进行手工的打标签，把验证码的目标边框转换为darknet使用的格式, 接下来，如果需要使用gpu，那么按照
2.1.2 darknet使用教程
darknet的使用主要有2个部分要注意，
第一块是：gpu训练darknet
如果使用gpu来训练darknet（比如我就是去百度租了5块1个小时的gpu进行训练，效率真的提高了很多），那么需要注意一些安装和配置：
- 百度的gpu机器需要安装cudnn
a. 首先下载 https://developer.nvidia.com/rdp/cudnn-download , 请注意一定要下载cudnn-8.0-linux-x64-v5.1.tgz，别的版本可能会有问题
b. 安装cudnn
$ cd ~
$ sudo tar xvf cudnn-8.0-linux-x64-v5.1.tgz
$ cd cuda/include
$ sudo cp *.h /usr/local/include/
$ cd ../lib64
$ sudo cp lib* /usr/local/lib/
$ cd /usr/local/lib# sudo chmod +r libcudnn.so.5.1.5
$ sudo ln -sf libcudnn.so.5.1.5 libcudnn.so.5
$ sudo ln -sf libcudnn.so.5 libcudnn.so
$ sudo ldconfig
- 编译darknet：
由于要使用gpu的方式，所以我们需要修改一些配置文件后进行编译：
a. 修改makefile
GPU=1
CUDNN=1
b. 修改cuda的路径
ifeq ($(GPU), 1)
COMMON+= -DGPU -I/usr/local/cuda-8.0/include/
CFLAGS+= -DGPU
LDFLAGS+= -L/usr/local/cuda-8.0/lib64 -lcuda -lcudart -lcublas -lcurand
#########################
NVCC=/usr/local/cuda-8.0/bin/nvcc
c. make即可, 这就会产生gpu训练版本的darknet源码了，现在只需要参考：http://blog.csdn.net/dcrmg/article/details/78496002 进行运行即可。
d. 接下来就训练吧，训练产生的参数都会保存到一个文件backup下面的yolo-voc.backup, 这个文件可要好好的保存哦，可以说weight在手，一切都飞不走，我在百度的机器上跑一天，基本测试集最后能达到100%的iou，和90%的覆盖比，基本够用啦。
./darknet detector train cfg/voc.data cfg/yolo-voc.2.0.cfg cfg/yolo-voc.weight
f. 上面的训练结束之后（以测试集能达到100%的iou，90%的覆盖比为准），我们又需要改造一下代码，让darknet能够输出中心位置的坐标，而不仅仅是在图片上显示出来。打开darknet的代码: src/image.c, 在如下函数新增一句话：
void draw_detections(image im, int num, float thresh, box *boxes, float **probs, float **masks, char **names, image **alphabet, int classes)
{
int i,j;
for(i = 0; i < num; ++i){
char labelstr[4096] = {0};
int class = -1;
for(j = 0; j < classes; ++j){
if (probs[i][j] > thresh){
if (class < 0) {
strcat(labelstr, names[j]);
class = j;
} else {
strcat(labelstr, ", ");
strcat(labelstr, names[j]);
}
}
}
if(class >= 0){
int width = im.h * .006;
/*
if(0){
width = pow(prob, 1./2.)*10+1;
alphabet = 0;
}
*/
//printf("%d %s: %.0f%%\n", i, names[class], prob*100);
int offset = class*123457 % classes;
float red = get_color(2,offset,classes);
float green = get_color(1,offset,classes);
float blue = get_color(0,offset,classes);
float rgb[3];
//width = prob*20+2;
rgb[0] = red;
rgb[1] = green;
rgb[2] = blue;
box b = boxes[i];
int left = (b.x-b.w/2.)*im.w;
int right = (b.x+b.w/2.)*im.w;
int top = (b.y-b.h/2.)*im.h;
int bot = (b.y+b.h/2.)*im.h;
if(left < 0) left = 0;
if(right > im.w-1) right = im.w-1;
if(top < 0) top = 0;
if(bot > im.h-1) bot = im.h-1;
draw_box_width(im, left, top, right, bot, width, red, green, blue);
//------------------新增这句话--------------------------------
printf("rect:%d, %d, %d, %d\n", left, top, right, bot);
//----------------------------------------------------------
if (alphabet) {
image label = get_label(alphabet, labelstr, (im.h*.03)/10);
draw_label(im, top + width, left, label, rgb);
free_image(label);
}
if (masks){
image mask = float_to_image(14, 14, 1, masks[i]);
image resized_mask = resize_image(mask, b.w*im.w, b.h*im.h);
image tmask = threshold_image(resized_mask, .5);
embed_image(tmask, im, left, top);
free_image(mask);
free_image(resized_mask);
free_image(tmask);
}
}
}
}
g. 重新make，得到darknet
h. 进行预测：
./darknet detector test cfg/voc.data cfg/yolo-voc.2.0.cfg backup/yolo-voc.backup a.png|grep "rect"
# 注意rect打印出来的结果不一定是从左到右，所以我们需要进行排序，让识别的部分能从左到右识别
echo "${result//rect:/}"|sort -n -t "," -k 1 >"${destPath}"/tmp
image.png
image.png
image.png
好的，已经打印出我们的字符的坐标了。第一步顺利完成。
2.2 图片黑白处理
当图片的边框已经识别出来了，我们就需要根据给出的坐标将其切割，并二值化为黑白图片。
可以使用如下的python代码进行切割：
# -*-coding:utf-8-*-
from PIL import Image
import sys
# x1(左上角坐标x), y1(左上角坐标y), x2(右下角坐标x), y2(右下角坐标y), picName(文件名), picPath(文件路径), codeName(单字符名字), destPath(目标路径)
x1 = sys.argv[1]
y1 = sys.argv[2]
x2 = sys.argv[3]
y2 = sys.argv[4]
picName = sys.argv[5]
picPath = sys.argv[6]
codeName = sys.argv[7]
destPath = sys.argv[8]
im = Image.open(picPath)
region = im.crop((float(x1), float(y1), float(x2), float(y2)))
cropPath= destPath + "/" + codeName + "_" + picName +"_ori.png"
bwPath= destPath+ "/" + codeName + "_" + picName +".png"
region.save(destPath + "/" + codeName + "_" + picName +"_ori.png")
可以用如下代码进行二值化为黑白图片：
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Binarize (make it black and white) an image with Python."""
from PIL import Image
from scipy.misc import imsave
import numpy
def binarize_image(img_path, target_path, threshold):
"""Binarize an image."""
image_file = Image.open(img_path)
image = image_file.convert('L') # convert image to monochrome
image = numpy.array(image)
image = binarize_array(image, threshold)
imsave(target_path, image)
def binarize_array(numpy_array, threshold=254):
"""Binarize a numpy array."""
for i in range(len(numpy_array)):
for j in range(len(numpy_array[0])):
# print(numpy_array[i][j])
if numpy_array[i][j] > threshold:
numpy_array[i][j] = 255
else:
numpy_array[i][j] = 0
return numpy_array
def get_parser():
"""Get parser object for script xy.py."""
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
parser = ArgumentParser(description=__doc__,
formatter_class=ArgumentDefaultsHelpFormatter)
parser.add_argument("-i", "--input",
dest="input",
help="read this file",
metavar="FILE",
required=True)
parser.add_argument("-o", "--output",
dest="output",
help="write binarized file hre",
metavar="FILE",
required=True)
parser.add_argument("--threshold",
dest="threshold",
default=200,
type=int,
help="Threshold when to show white")
return parser
if __name__ == "__main__":
args = get_parser().parse_args()
binarize_image(args.input, args.output, args.threshold)
调用方式：
# cropPath：源文件
# bwPath: 目标文件
python ./convertblack.py -i "$cropPath" -o "$bwPath" --threshold 254
至此，我们的切割与黑白化就完成了。
2.3 cnn识别算法
这个部分我主要参考了tensorflow识别mnist的代码，对其进行了改造，我的识别图片resize到24*24，一共有62个类别(0-9, a-z, A-Z), 主要有如下2个文件：一个是huobi.py，主要是识别的主体部分：
# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Functions for downloading and reading MNIST data."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import gzip
import os
import tempfile
import numpy
from six.moves import urllib
from six.moves import xrange # pylint: disable=redefined-builtin
import tensorflow as tf
#from tensorflow.contrib.learn.python.learn.datasets.mnist import read_data_sets
import tensorflow as tf
import read_huobi
reSizePic=24
#0-9,a-z,A-Z, 62
classNum=62
originalPicSize=reSizePic*reSizePic
# read to mnist
#mnist = read_data_sets('/Users/baidu/PycharmProjects/neural/mnist/date/', one_hot=True)
mnist = read_huobi.load_huobi("/Users/baidu/PycharmProjects/neural/VerifyCodeDetection/darknet/darknet/results/")
x = tf.placeholder("float", shape=[None, originalPicSize], name='input_x')
y_ = tf.placeholder("float", shape=[None, classNum], name='input_y')
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
strides=[1, 2, 2, 1], padding='SAME')
# 1 layer: 5*5*1(input 1), 32 filters
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
x_image = tf.reshape(x, [-1,reSizePic,reSizePic,1])
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)
# pool1 output 12
#2 layer: 5*5*32(layer 1 output 32), 64 filters
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)
# pool2 output 6,
# conection 1, the 6 is related to input size
W_fc1 = weight_variable([(reSizePic/4) * (reSizePic/4) * 64, 1024])
b_fc1 = bias_variable([1024])
h_pool2_flat = tf.reshape(h_pool2, [-1, (reSizePic/4)*(reSizePic/4)*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
# drop out
keep_prob = tf.placeholder("float", name='keep_prob')
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)
# output layer
W_fc2 = weight_variable([1024, classNum])
# 0-9, a-z, a-Z=10+26+26=62
b_fc2 = bias_variable([classNum])
y_conv=tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)
# loss
cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv))
# backproporation
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
graph_location = tempfile.mkdtemp() # temp file
print('Saving graph to: %s' % graph_location)
train_writer = tf.summary.FileWriter(graph_location)
train_writer.add_graph(tf.get_default_graph())
saver=tf.train.Saver(tf.global_variables())
tf.add_to_collection('pred_network', y_conv)
tf.add_to_collection('accuracy_network', accuracy)
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
for i in range(1000):
batch = mnist.train.next_batch(1448)
if i%100 == 0:
save_path ='./tf_model/model_'+'%d'%i
print('%s' % save_path)
saver.save(sess, save_path)
train_accuracy = accuracy.eval(feed_dict={
x:batch[0], y_: batch[1], keep_prob: 1.0})
print("step %d, training accuracy %g"%(i, train_accuracy))
train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
print("test accuracy %g"%accuracy.eval(feed_dict={
x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))
另一个是read_huobi.py，主要用于图片的处理和读取：
