In this post I’ll show you how to debug and set break-point in caffe on Ubuntu with GDB , the GNU Project debugger.

First of all you should install and build caffe from source, see the documentation for detailed installation instructions, don’t forget to turn ‘DEBUG’ option on in ‘Makefile.config’.

Prelimilaries

To run up the caffe mnist example, there are two pre-processing steps:

1. cd to the caffe source directory and build caffe with ‘DEBUG := 1’ uncommented in ‘Makefile.config’;

2. download mnist data: bash data/mnist/get_mnist.sh;

3. convert mnist data set to lmdb: bash examples/mnist/create_mnist.sh;

4. edit examples/mnist/lenet_solver.prototxt, swith ‘solver_mode’ from GPU to CPU.

Starting

After above prelimilary steps, now we can set berak points and probe into the caffe source code.

First make sure your ‘$(pwd)’ is in caffe source directory;

Activate GDB

Then start debugging with GDB: gdb --args build/tools/caffe train --solver examples/mnist/lenet_solver.prototxt, the ‘–args’ option indicates we will pass arguments into the program we are debugging. In our circumstance, we are debugging build/tools/caffe with argument train --solver examples/mnist/lenet_solver.prototxt.

1

Set break point

Set a break point in ‘src/caffe/layers/base_conv_layer.cpp’ line 117: b src/caffe/layers/base_conv_layer.cpp:117

2

The related code is:

channels_ = bottom[0]->shape(channel_axis_);
num_output_ = this->layer_param_.convolution_param().num_output();
CHECK_GT(num_output_, 0);

GDB command b means ‘break’, use b path/to/code.cpp:#line to set break point in a specifical line of a specified source file.

Run the program

Then we run up the program: r

3

Now it’s running and you can get the standard log output from the terminal, just as above image shows.

The program will stop at the break point If nothing unexpected happens.

4

Print intermidiate variables

Now you can use p var to print value of variable, and n to step to the next line, and c to cotinue(stop until the next break point).

For example, now we are at line 117 in ‘src/caffe/layers/base_conv_layer.cpp’: channels_ = bottom[0]->shape(channel_axis_),NOTE: that this line hasn’t been executed yet, use n to step to line 118:

5

Print the value of variable ‘channels_’ with command p channels_:

6

We got ‘channels_ = 1’, in fact channels_ means thechannels of input blob of convolution layer, since we break at the first convolutional layer, the input blob is mnist image data, images in mnist dataset are single channel gray images.

And you can print whatever internal variables you want to help you to analysis, for example, print the first element of first input(bottom) blob: p this->bottom[0]->cpu_data()[0], or print the first element of layer parametric weights: p blobs_[0]->cpu_data()[0](generally blobs_[0] stores weight and blobs_[1]stores bias).

Navigate code with ctags & sublime

1. install “ctags” on ubuntu: sudo apt-get install exuberant-ctags;

2. install “package control” for sublime following this ;

3. press ‘ctr+shift+p’ to invoke package control, type “install” and select “Package Control: Install Package”, then search “ctags” and install CTags extension for sublime;

4. generate tags index file at caffe source root via ctags -Rf .tags

Now restart sublime and press “ctr + shift” then click function/class name to navigate defination and reference e.t.c.

This post is aimed at introducing how to debug caffe on ubuntu, detailed GDB usage is out of the scope of this post. You can refer to this tutorial or some more GDB skills and gdb_refcard for more GDB commands, there are also lots of posts/tutorials on that topic. If you wanna debug cuda GPU code, you should use cuda-gdb, it’s similar to GDB, official documentation for cuda-gdb