RFCN-tensorflow的源码

源码地址:
https://github.com/xdever/RFCN-tensorflow

简单结构:


  • k^2(C+1)的conv: ResNet101的最后的输出是WxHx1024,用K^2(C+1)个1x1的卷积核去卷积,即可得到K^2(C+1)个大小为WxH的position sensitive的score map。这步的卷积操作就是在做prediction。这里的k=3,表示把一个ROI划分成3*3,对应的9个位置分别是:上左(左上角),上中,上右,中左,中中,中右,下左,下中,下右(右下角)

  • k^2(C+1)个feature map的物理意义: 共有k x k = 9个颜色,每个颜色的立体块(WxHx(C+1))表示的是不同位置存在目标的概率值(第一块黄色表示的是左上角位置,最后一块淡蓝色表示的是右下角位置)。

  • pooling公式

z(i,j,c)是第i+k*(j-1)个立体块上的第c个map(1<= i,j <=3)。(i,j)决定了9种位置的某一种位置,假设为左上角位置(i=j=1),c决定了哪一类,假设为person类。在z(i,j,c)这个feature map上的某一个像素的位置是(x,y),像素值是value,则value表示的是原图对应的(x,y)这个位置上可能是人(c=‘person’)且是人的左上部位(i=j=1)的概率值

  • ROI pooling的输入和输出:ROI pooling操作的输入(对于C+1个类)是k^2(C+1)W' H'(W'和H'是ROI的宽度和高度)的score map上某ROI对应的那个立体块(由RPN预测的box坐标,在feature map上进行裁剪),且该立体块组成一个新的k^2(C+1)W' H'的立体块:每个颜色的立体块(C+1)都只抠出对应位置的一个bin,把这kk个bin组成新的立体块,大小为(C+1)W'H'。例如,下图中的第一块黄色只取左上角的bin,最后一块淡蓝色只取右下角的bin。所有的bin重新组合后就变成了类似右图的那个薄的立体块(图中的这个是池化后的输出,即每个面上的每个bin上已经是一个像素。池化前这个bin对应的是一个区域,是多个像素)。ROI pooling的输出为为一个(C+1)k*k的立体块

介绍结束,下面开始代码。。。

注意这是非官方版本的代码。仅仅是为了学习关于R-FCN的检测流程而去读的这份代码,代码本人并没有调试。

代码目录

Google搜索了很多博客,发现大家其实并不是特别关注R-FCN模型。就把这分代码捣鼓一下吧。

  • main.py是进行train的文件,如果你想进行训练可以修改这里的代码。
  • testCheckpoint.py是进行test的文件,就是测试ckpt文件中保存参数的shape的一个文件,这是与整个模型独立的一个文件,仅仅是对ckpt进行检查的文件。
  • test.py 这个文件是进行检测模型测试的文件。

test.py

所有的检测代码大部分都是相同的流程,所以首先看test.py模型代码。

parser = argparse.ArgumentParser(description="RFCN tester")
parser.add_argument('-gpu', type=str, default="0", help='Train on this GPU(s)')
parser.add_argument('-n', type=str, help='Network checkpoint file')
parser.add_argument('-i', type=str, help='Input file.')
parser.add_argument('-o', type=str, default="", help='Write output here.')
parser.add_argument('-p', type=int, default=1, help='Show preview')
parser.add_argument('-threshold', type=float, default=0.5, help='Detection threshold')
parser.add_argument('-delay', type=int, default=-1, help='Delay between frames in visualization. -1 for automatic, 0 for wait for keypress.')

这里的超参数设置的比较简单。

palette = Visualize.Palette(len(categories))

image = tf.placeholder(tf.float32, [None, None, None, 3])
net = BoxInceptionResnet(image, len(categories), name="boxnet")

boxes, scores, classes = net.getBoxes(scoreThreshold=opt.threshold)


input = PreviewIO.PreviewInput(opt.i)
output = PreviewIO.PreviewOutput(opt.o, input.getFps())

这里使用BoxInceptionResnet构建R-FCN模型。
使用PreviewIO进行文件的读写。

boxes, scores, classes = net.getBoxes(scoreThreshold=opt.threshold)

这里直接通过net.getBoxes获取模型的检测输出的占位符。

只看test.py 的主要结构。

with tf.Session() as sess:
        .........
        # 检查模型
    if not CheckpointLoader.loadCheckpoint(sess, None, opt.n, ignoreVarsInFileNotInSess=True):
                
        ..........
        # 对img进行预处理
        img = preprocessInput(img)  
                ...........
         # 把图片输入到检测模型进行计算
        rBoxes, rScores, rClasses = sess.run([boxes, scores, classes], feed_dict={image: np.expand_dims(img, 0)})

        ...........
        # 对输入的结果进行可视化
        res = Visualize.drawBoxes(img, rBoxes, rClasses, [categories[i] for i in rClasses.tolist()], palette, scores=rScores)

main.py

main.py就是进行训练的函数,为了简化训练过程,下面只列出重要的训练过程。下面只看loss

# 数据读取
dataset = BoxLoader()
dataset.add(CocoDataset(opt.dataset, randomZoom=opt.randZoom==1, set="train"+opt.cocoVariant))
if opt.mergeValidationSet==1:
    dataset.add(CocoDataset(opt.dataset, set="val"+opt.cocoVariant))

.....................
# 获取图片的标签
images, boxes, classes = Augment.augment(*dataset.get())

# 获取检测模型
net = BoxInceptionResnet(images, dataset.categoryCount(), name="boxnet", trainFrom=opt.trainFrom, hardMining=opt.hardMining==1, freezeBatchNorm=opt.freezeBatchNorm==1)

# 获取模型的loss
tf.losses.add_loss(net.getLoss(boxes, classes))

# 构建loss操作
def createUpdateOp(gradClip=1):
    with tf.name_scope("optimizer"):
        optimizer=tf.train.AdamOptimizer(learning_rate=opt.learningRate, epsilon=opt.adamEps)
        update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
        totalLoss = tf.losses.get_total_loss()
        grads = optimizer.compute_gradients(totalLoss, var_list=net.getVariables())
        if gradClip is not None:
            cGrads = []
            for g, v in grads:
                if g is None:
                    print("WARNING: no grad for variable "+v.op.name)
                    continue
                cGrads.append((tf.clip_by_value(g, -float(gradClip), float(gradClip)), v))
            grads = cGrads

        update_ops.append(optimizer.apply_gradients(grads))
        return control_flow_ops.with_dependencies([tf.group(*update_ops)], totalLoss, name='train_op')

# 构建优化操作
trainOp=createUpdateOp()

在main.py文件中有一个while=true的死循环,使用RunManager对训练进行管理。如下:

    runManager = RunManager(sess, options=runOptions, run_metadata=runMetadata)
    runManager.add("train", [globalStepInc,trainOp], modRun=1)

在while中进行优化训练

    ......
    # 进行训练
    while True:
        #run various parts of the network
        res = runManager.modRun(i)
        .....
        # 可是化训练结果
        visualizer.draw(res)

下面来看检测模型BoxInceptionResnet。
在test.py文件中使用net.getBoxes的的方法。

net = BoxInceptionResnet(image, len(categories), name="boxnet")

boxes, scores, classes = net.getBoxes(scoreThreshold=opt.threshold)

在train.py中使用BoxInceptionResnet的net.getLoss和net.getVariables。

net = BoxInceptionResnet(images, dataset.categoryCount(), name="boxnet", trainFrom=opt.trainFrom, hardMining=opt.hardMining==1, freezeBatchNorm=opt.freezeBatchNorm==1)
tf.losses.add_loss(net.getLoss(boxes, classes))
......
grads = optimizer.compute_gradients(totalLoss, var_list=net.getVariables())

所以在进行源码查看时就看这几个函数。
发现BoxInceptionResnet是BoxNetwork的子类。
BoxNetwork.py

# BoxNetwork.py

class BoxNetwork:
    def __init__(self, nCategories, rpnLayer, rpnDownscale, rpnOffset, featureLayer=None, featureDownsample=None, featureOffset=None, weightDecay=1e-6, hardMining=True):
        '''
        featureInput = slim.conv2d(net, 1536, 1)
        BoxNetwork.__init__(self, nCategories, rpnInput, 16, [32,32], featureInput, 16, [32,32], weightDecay=weightDecay, hardMining=hardMining)
    
        nCategories: 检测的种类
        rpnInput:输入到rpn的feature map
        rpnDownscale : rpnDownscale下采样的尺度
        rpnOffset: 图片的偏置
        featureInput : 经过编码的featureInput
        featureDownsample: featureDownsample下采样的尺度
        featureOffset:feature的偏置尺寸
        '''
        
        if featureLayer is None:
            featureLayer=rpnLayer

        if featureDownsample is None:
            featureDownsample=rpnDownscale
            
        if featureOffset is None:
            rpnOffset=featureOffset

        with tf.name_scope("BoxNetwork"):
            self.rpn = RPN(rpnLayer, immediateSize=512, weightDecay=weightDecay, inputDownscale=rpnDownscale, offset=rpnOffset)
            self.boxRefiner = BoxRefinementNetwork(featureLayer, nCategories, downsample=featureDownsample, offset=featureOffset, hardMining=hardMining)

            self.proposals, self.proposalScores = self.rpn.getPositiveOutputs(maxOutSize=300)

    # 使用阀值过滤筛选出合适的box坐标
    def getProposals(self, threshold=None):
        if threshold is not None and threshold>0:
            s = tf.cast(tf.where(self.proposalScores > threshold), tf.int32)
            return tf.gather_nd(self.proposals, s), tf.gather_nd(self.proposalScores, s)
        else:
            return self.proposals, self.proposalScores
    
    # 使用boxRefiner对feature map 进行classes 和 box坐标的预测
    def getBoxes(self, nmsThreshold=0.3, scoreThreshold=0.8):
        return self.boxRefiner.getBoxes(self.proposals, self.proposalScores, maxOutputs=50, nmsThreshold=nmsThreshold, scoreThreshold=scoreThreshold)
    
    # 输入真实的box 坐标和classes。计算RPN 和 检测的总loss
    def getLoss(self, refBoxes, refClasses):
        return self.rpn.loss(refBoxes) + self.boxRefiner.loss(self.proposals, refBoxes, refClasses)

请仔细看注释,因为这个是R-FCN的基类所以这里的程序比较简单。

大体就是使用InceptionResnetV2的最后的输出的feature map送入RPN,再对feature map进行几次conv2d运算,在BoxRefinementNetwork中进行计算。

注意这里的

def getBoxes(self, nmsThreshold=0.3, scoreThreshold=0.8):
        return self.boxRefiner.getBoxes(self.proposals, self.proposalScores, maxOutputs=50, nmsThreshold=nmsThreshold, scoreThreshold=scoreThreshold)

在test.py中的输出是

boxes, scores, classes = net.getBoxes(scoreThreshold=opt.threshold)

这里的输出就是通过RPN对所有的feature map产生box。再由self.boxRefiner.getBoxes在feature map上提取ROI,并且产生boxes, scores, classes。

BoxInceptionResnet.py

# BoxInceptionResnet.py
class BoxInceptionResnet(BoxNetwork):
    LAYER_NAMES = ['Conv2d_1a_3x3','Conv2d_2a_3x3','Conv2d_2b_3x3','MaxPool_3a_3x3','Conv2d_3b_1x1','Conv2d_4a_3x3',
              'MaxPool_5a_3x3','Mixed_5b','Repeat','Mixed_6a','Repeat_1','Mixed_7a','Repeat_2','Block8','Conv2d_7b_1x1']

    def __init__(self, inputs, nCategories, name="BoxNetwork", weightDecay=0.00004, freezeBatchNorm=False, reuse=False, isTraining=True, trainFrom=None, hardMining=True):
        self.boxThreshold = 0.5

        try:
            trainFrom = int(trainFrom)
        except:
            pass

        if isinstance(trainFrom, int):
            trainFrom = self.LAYER_NAMES[trainFrom]


        print("Training network from "+(trainFrom if trainFrom is not None else "end"))

        with tf.variable_scope(name, reuse=reuse) as scope:
            # 构建基础的特征提取模型模型
            self.googleNet = InceptionResnetV2("features", inputs, trainFrom=trainFrom, freezeBatchNorm=freezeBatchNorm)
            self.scope=scope
        
            with tf.variable_scope("Box"):
                #Pepeat_1 - last 1/16 layer, Mixed_6a - first 1/16 layer
                # 拿到Repeat_1的feature map
                scale_16 = self.googleNet.getOutput("Repeat_1")[:,1:-1,1:-1,:]
                
                # 拿到Mixed_6a的feature map 
                #scale_16 = self.googleNet.getOutput("Mixed_6a")[:,1:-1,1:-1,:]
                scale_32 = self.googleNet.getOutput("PrePool")

                with slim.arg_scope([slim.conv2d],
                        weights_regularizer=slim.l2_regularizer(weightDecay),
                        biases_regularizer=slim.l2_regularizer(weightDecay),
                        padding='SAME',
                        activation_fn = tf.nn.relu):
                    
                    # 合并Repeat_1和PrePool的feature map
                    # 从这里开始net分为两个输出,一个是输出rpnInput,一个是输出featureInput
                    net = tf.concat([ tf.image.resize_bilinear(scale_32, tf.shape(scale_16)[1:3]), scale_16], 3)
                    
                    
                    rpnInput = slim.conv2d(net, 1024, 1)
                    
                    #BoxNetwork.__init__(self, nCategories, rpnInput, 16, [32,32], scale_32, 32, [32,32], weightDecay=weightDecay, hardMining=hardMining)
                    
                    # 这里的featureInput
                    featureInput = slim.conv2d(net, 1536, 1)
                    BoxNetwork.__init__(self, nCategories, rpnInput, 16, [32,32], featureInput, 16, [32,32], weightDecay=weightDecay, hardMining=hardMining)
    
    def getVariables(self, includeFeatures=False):
        if includeFeatures:
            return tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope=self.scope.name)
        else:
            vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope=self.scope.name+"/Box/")
            vars += self.googleNet.getTrainableVars()

            print("Training variables: ", [v.op.name for v in vars])
            return vars

    def importWeights(self, sess, filename):
        self.googleNet.importWeights(sess, filename, includeTraining=True)


注意看上面的注释:

net分为两路:rpnInput和featureInput。

# 合并Repeat_1和PrePool的feature map
# 从这里开始net分为两个输出,一个是输出rpnInput,一个是输出featureInput
net = tf.concat([ tf.image.resize_bilinear(scale_32, tf.shape(scale_16)[1:3]), scale_16], 3)

rpnInput = slim.conv2d(net, 1024, 1)

# BoxNetwork.__init__(self, nCategories, rpnInput, 16, [32,32], scale_32, 32, [32,32], weightDecay=weightDecay, hardMining=hardMining)

# 这里的featureInput
featureInput = slim.conv2d(net, 1536, 1)
BoxNetwork.__init__(self, nCategories, rpnInput, 16, [32,32], featureInput, 16, [32,32], weightDecay=weightDecay, hardMining=hardMining)

如下图:

在这里有个比较重要的操作就是RPN和BoxRefinementNetwork

接下来那就看看怎么计算RPN吧

RPN.py

为了不把RPN搞得太复杂,这里就看看被调用的几个函数

在BoxNetwork.py中使用RPN.如下:

self.rpn = RPN(rpnLayer, immediateSize=512, weightDecay=weightDecay, inputDownscale=rpnDownscale, offset=rpnOffset)
......
            self.proposals, self.proposalScores = self.rpn.getPositiveOutputs(maxOutSize=300)

只用到rpn.getPositiveOutputs

# RPN.py

class RPN:
    def __init__(self, input, anchors=None, immediateSize=512, weightDecay=1e-5, inputDownscale=16, offset=[32,32]):
        self.input = input
        self.anchors = anchors
        self.inputDownscale = inputDownscale
        self.offset = offset
        self.anchors = anchors if anchors is not None else self.makeAnchors([64,128,256,512])
        print("Anchors: ", self.anchors)
        self.tfAnchors = tf.constant(self.anchors, dtype=tf.float32)

        self.hA=tf.reshape(self.tfAnchors[:,0],[-1])
        self.wA=tf.reshape(self.tfAnchors[:,1],[-1])

        self.nAnchors = len(self.anchors)

        self.positiveIouThreshold=0.7
        self.negativeIouThreshold=0.3
        self.regressionWeight=1.0
        
        self.nBoxLosses=256
        self.nPositiveLosses=128

        #dimensions
        with tf.name_scope('dimension_info'):
            s = tf.shape(self.input)
            self.hIn = s[1]
            self.wIn = s[2]

        
        self.imageH = tf.cast(self.hIn*self.inputDownscale+self.offset[0]*2, tf.float32)
        self.imageW = tf.cast(self.wIn*self.inputDownscale+self.offset[1]*2, tf.float32)

        self.define(immediateSize, weightDecay)


    def define(self, immediateSize, weightDecay):
        with tf.name_scope('RPN'):
            with slim.arg_scope([slim.conv2d], weights_regularizer=slim.l2_regularizer(weightDecay), padding='SAME'):
                #box prediction layers
                with tf.name_scope('NN'):
                    net = slim.conv2d(self.input, immediateSize, 3, activation_fn=tf.nn.relu)
                    scores = slim.conv2d(net, 2*self.nAnchors, 1, activation_fn=None)
                    boxRelativeCoordinates = slim.conv2d(net, 4*self.nAnchors, 1, activation_fn=None)

                #split coordinates
                x_raw, y_raw, w_raw, h_raw = tf.split(boxRelativeCoordinates, 4, axis=3)

                #Save raw box sizes for loss
                self.rawSizes = BoxUtils.mergeBoxData([w_raw, h_raw])
                            
                #Convert NN outputs to BBox coordinates
                self.boxes = BoxUtils.nnToImageBoxes(x_raw, y_raw, w_raw, h_raw, self.wA, self.hA, self.inputDownscale, self.offset)

                #store the size of every box
                with tf.name_scope('box_sizes'):
                    boxSizes = tf.reshape(self.tfAnchors, [1,1,1,-1,2])
                    boxSizes = tf.tile(boxSizes, tf.stack([1,self.hIn,self.wIn,1,1]))
                    self.boxSizes = tf.reshape(boxSizes, [-1,2])

                #scores
                self.scores = tf.reshape(scores, [-1,2])

这里的define输出self.boxes 和self.boxes。这里可以理解为使用Faster R-CNN一样的原理。这里输入的rpnInput输出为self.boxes 和self.boxes。这里的self.boxes是针对rpnInput的每一个点产生的box坐标,self.boxes是判断box里面是否有物体,为一个二分类判别器。

在BoxNetwork调用的是rpn.filterOutputBoxe

# RPN.py
def filterOutputBoxes(self, boxes, scores, others=[], preNmsCount=6000, maxOutSize=300, nmsThreshold=0.7): 
        with tf.name_scope("filter_output_boxes"):
            scores = tf.nn.softmax(scores)[:,1]
            scores = tf.reshape(scores,[-1])

            #Clip boxes to edge
            boxes = self.clipBoxesToEdge(boxes)

            #Remove empty boxes
            boxes, scores = BoxUtils.filterSmallBoxes(boxes, [scores])
            scores, boxes = tf.cond(tf.shape(scores)[0] > preNmsCount , lambda: tf.tuple(MultiGather.gatherTopK(scores, preNmsCount, [boxes])), lambda: tf.tuple([scores, boxes]))

            #NMS filter
            nmsIndices = tf.image.non_max_suppression(boxes, scores, iou_threshold=nmsThreshold, max_output_size=maxOutSize)
            nmsIndices = tf.expand_dims(nmsIndices, axis=-1)

            return MultiGather.gather([boxes, scores]+others, nmsIndices)
        
    def getPositiveOutputs(self, preNmsCount=6000, maxOutSize=300, nmsThreshold=0.7):
        boxes, scores = self.filterOutputBoxes(self.boxes, self.scores, preNmsCount=preNmsCount, nmsThreshold=nmsThreshold, maxOutSize=maxOutSize)
        return boxes, scores

filterOutputBoxe就是对boxes, scores进过滤对使用nms和阀值等方法过滤掉不合适的boxes。

    1. 裁剪超出边界的box
  • 2.一出不包含物体的box
  • 3.使用NMS进行过滤

BoxRefinementNetwork.py

在BoxNetwork中是使用BoxRefinementNetwork

self.boxRefiner = BoxRefinementNetwork(featureLayer, nCategories, downsample=featureDownsample, offset=featureOffset, hardMining=hardMining)

................

self.boxRefiner.getBoxes(self.proposals, self.proposalScores, maxOutputs=50, nmsThreshold=nmsThreshold, scoreThreshold=scoreThreshold)

这里通过BoxRefinementNetwork使用到了self.boxRefiner.getBoxes

直接来看BoxRefinementNetwork

# BoxRefinementNetwork.py

class BoxRefinementNetwork:
    POOL_SIZE=3

    def __init__(self, input, nCategories, downsample=16, offset=[32,32], hardMining=True):
        self.downsample = downsample
        self.offset = offset
        
        # 设置分类个数
        self.nCategories = nCategories
        
        # 输出(self.POOL_SIZE**2)*(1+nCategories)个feature map的classes scores。
        self.classMaps = slim.conv2d(input, (self.POOL_SIZE**2)*(1+nCategories), 3, activation_fn=None, scope='classMaps')
        
        # (self.POOL_SIZE**2)*4 个feature map的box坐标
        self.regressionMap = slim.conv2d(input, (self.POOL_SIZE**2)*4, 3, activation_fn=None, scope='regressionMaps')

        self.hardMining=hardMining

        #Magic parameters.
        self.posIouTheshold = 0.5
        self.negIouThesholdHi = 0.5
        self.negIouThesholdLo = 0.1
        self.nTrainBoxes = 128
        self.nTrainPositives = 32
        self.falseValue = 0.0002

这里比较绕的就是这里的三个函数,从下往上调用。

    def roiPooling(self, layer, boxes):
        return positionSensitiveRoiPooling(layer, boxes, offset=self.offset, downsample=self.downsample, roiSize=self.POOL_SIZE)

    def roiMean(self, layer, boxes):
        with tf.name_scope("roiMean"):
            return tf.reduce_mean(self.roiPooling(layer, boxes), axis=[1,2])

    def getBoxScores(self, boxes):
        with tf.name_scope("getBoxScores"):
            return self.roiMean(self.classMaps, boxes)

下图就是实现过程:

上面的ROI就是这里输入的box坐标。box的坐标在self.classMaps上进行裁剪,拿到对应的ROI的feature map再进行对应的roi_pooling操作。选择出合适的类别。这里的具体过程还没仔细分析。。。。。。

详细的原理细节看这里

还记得self.classMaps和self.regressionMap吗??

这里的self.classMaps是用来计算分数的,使用getBoxScores已经对self.classMaps操作完了,筛选出合适的box了。现在需要使用refineBoxes对self.regressionMap进行裁剪操作,得到最后的positives 的box


def getBoxes(self, proposals, proposal_scores, maxOutputs=30, nmsThreshold=0.3, scoreThreshold=0.8):
        if scoreThreshold is None:
            scoreThreshold = 0

        with tf.name_scope("getBoxes"):
            scores = tf.nn.softmax(self.getBoxScores(proposals))
            
            classes = tf.argmax(scores, 1)
            scores = tf.reduce_max(scores, axis=1)
            posIndices = tf.cast(tf.where(tf.logical_and(classes > 0, scores>scoreThreshold)), tf.int32)

            positives, scores, classes = MultiGather.gather([proposals, scores, classes], posIndices)
            positives = self.refineBoxes(positives, False)

            #Final NMS
            posIndices = tf.image.non_max_suppression(positives, scores, iou_threshold=nmsThreshold, max_output_size=maxOutputs)
            posIndices = tf.expand_dims(posIndices, axis=-1)
            positives, scores, classes = MultiGather.gather([positives, scores, classes], posIndices)   
            
            classes = tf.cast(tf.cast(classes,tf.int32) - 1, tf.uint8)

            return positives, scores, classes

可以这么说这里检测流程已经结束了

loss

def loss(self, proposals, refBoxes, refClasses):
        with tf.name_scope("BoxRefinementNetworkLoss"):
            proposals = tf.stop_gradient(proposals)
            
            # 位置loss 
            def getPosLoss(positiveBoxes, positiveRefIndices, nPositive):
                with tf.name_scope("getPosLoss"):
                    positiveRefIndices =  tf.reshape(positiveRefIndices,[-1,1])

                    positiveClasses, positiveRefBoxes = MultiGather.gather([refClasses, refBoxes], positiveRefIndices)
                    positiveClasses = tf.cast(tf.cast(positiveClasses,tf.int8) + 1, tf.uint8)

                    if not self.hardMining:
                        selected = Utils.RandomSelect.randomSelectIndex(tf.shape(positiveBoxes)[0], nPositive)
                        positiveBoxes, positiveClasses, positiveRefBoxes = MultiGather.gather([positiveBoxes, positiveClasses, positiveRefBoxes], selected)

                    return tf.tuple([self.classRefinementLoss(positiveBoxes, positiveClasses) + self.boxRefinementLoss(positiveBoxes, positiveRefBoxes), tf.shape(positiveBoxes)[0]])
            # 
            def getNegLoss(negativeBoxes, nNegative):
                with tf.name_scope("getNetLoss"):
                    if not self.hardMining:
                        negativeIndices = Utils.RandomSelect.randomSelectIndex(tf.shape(negativeBoxes)[0], nNegative)
                        negativeBoxes = tf.gather_nd(negativeBoxes, negativeIndices)

                    return self.classRefinementLoss(negativeBoxes, tf.zeros(tf.stack([tf.shape(negativeBoxes)[0],1]), dtype=tf.uint8))
            
            def getRefinementLoss():
                with tf.name_scope("getRefinementLoss"):
                    iou = BoxUtils.iou(proposals, refBoxes)
                    
                    maxIou = tf.reduce_max(iou, axis=1)
                    bestIou = tf.expand_dims(tf.cast(tf.argmax(iou, axis=1), tf.int32), axis=-1)

                    #Find positive and negative indices based on their IOU
                    posBoxIndices = tf.cast(tf.where(maxIou > self.posIouTheshold), tf.int32)
                    negBoxIndices = tf.cast(tf.where(tf.logical_and(maxIou < self.negIouThesholdHi, maxIou > self.negIouThesholdLo)), tf.int32)

                    #Split the boxes and references
                    posBoxes, posRefIndices = MultiGather.gather([proposals, bestIou], posBoxIndices)
                    negBoxes = tf.gather_nd(proposals, negBoxIndices)

                    #Add GT boxes
                    posBoxes = tf.concat([posBoxes,refBoxes], 0)
                    posRefIndices = tf.concat([posRefIndices, tf.reshape(tf.range(tf.shape(refClasses)[0]), [-1,1])], 0)

                    #Call the loss if the box collection is not empty
                    nPositive = tf.shape(posBoxes)[0]
                    nNegative = tf.shape(negBoxes)[0]

                    if self.hardMining:
                        posLoss = tf.cond(nPositive > 0, lambda: getPosLoss(posBoxes, posRefIndices, 0)[0], lambda: tf.zeros((0,), tf.float32))
                        negLoss = tf.cond(nNegative > 0, lambda: getNegLoss(negBoxes, 0), lambda: tf.zeros((0,), tf.float32))

                        allLoss = tf.concat([posLoss, negLoss], 0)
                        return tf.cond(tf.shape(allLoss)[0]>0, lambda: tf.reduce_mean(Utils.MultiGather.gatherTopK(allLoss, self.nTrainBoxes)), lambda: tf.constant(0.0))
                    else:
                        posLoss, posCount = tf.cond(nPositive > 0, lambda: getPosLoss(posBoxes, posRefIndices, self.nTrainPositives), lambda: tf.tuple([tf.constant(0.0), tf.constant(0,tf.int32)]))
                        negLoss = tf.cond(nNegative > 0, lambda: getNegLoss(negBoxes, self.nTrainBoxes-posCount), lambda: tf.constant(0.0))

                        nPositive = tf.cast(tf.shape(posLoss)[0], tf.float32)
                        nNegative = tf.cond(nNegative > 0, lambda: tf.cast(tf.shape(negLoss)[0], tf.float32), lambda: tf.constant(0.0))
                        
                        return (tf.reduce_mean(posLoss)*nPositive + tf.reduce_mean(negLoss)*nNegative)/(nNegative+nPositive)
    

        return tf.cond(tf.logical_and(tf.shape(proposals)[0] > 0, tf.shape(refBoxes)[0] > 0), lambda: getRefinementLoss(), lambda:tf.constant(0.0))

这里只是R-FCN的loss

总的loss

联合训练需要RPN和R-FCN总的loss

def getLoss(self, refBoxes, refClasses):
        return self.rpn.loss(refBoxes) + self.boxRefiner.loss(self.proposals, refBoxes, refClasses)

参考:

RFCN论文笔记
R-FCN论文阅读

推荐阅读更多精彩内容