ubuntu16.04 python3.6 caffe(CPU) 配置记录

从头开始配置编译python3.6版本的caffe整整花了10天时间,期间经历了很多事,所以状态一直很差,真正的配置时间应该在五天左右.

目前搜索到的caffe配置版本多数是2.7和3.5,关于python3.6的配置基本没有.但是目前anaconda3用的python版本为3.6,tensorflow最新使用的都是3.6,所以我电脑里存在的就是3.6版本.所以我决定配置python3.6版本的caffe.

step by step!

1,配置opencv 3.4.1

我们需要达到的目标是在python3中可以直接使用import cv2,其中关于编译这方面的资料很多.主要分为编译本体库,以及编译一个附加下载的库.但是在编译完成之后发现,依然无法成功import cv2.程序中的Import cv2提示no modules named cv2错误

解决方法:sudo pip3 install opencv-python.注意这里一定要用pip3,之前一直有资料说用pip,这样是无法成功的.使用pip3后,直接成功.

2,编译caffe

这里的大前提是已经编译配置好anaconda3,python的版本为3.6.python3.6的bin和lib文件全部在anaconda3中.

这里主要参考的是这篇博文.https://blog.csdn.net/isuccess88/article/details/70165726,但也有很多不同,他那篇是针对python3.5的.

2.1 安装依赖

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler

sudo apt-get install --no-install-recommends libboost-all-dev

sudo apt-get install libatlas-base-dev

sudo apt-get install libhdf5-serial-dev

sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev

然后把CAFFE的源代码下载下来: git clone https://github.com/BVLC/caffe.git

(当然没有安装GIT的得先安装一下)

下载完成之后,进入CAFFE文件夹, 进入里面的PYTHON文件夹,然后输入

for req in $(cat requirements.txt); do pip3 install $req; done

(PIP如果没有安装得先安装一下:sudo apt install python-pip)

注意的地方是:pip3安装的,第二点是sudo apt-get install --no-install-recommends libboost-all-dev这一条安装编译的并没有关于python3.6版本的libboost,我们需要重新编译.否则caffe无法调用成功.(怎么编译后面说)

2.2 安装caffe

到CAFFE文件夹, 使用模板写个Makefile.config. 具体就是先复制一下模板, 再改一些内容(我喜欢用EMACS).

cp Makefile.config.example Makefile.config

-因为CPU MODE, 所以在CPU_ONLY := 1前面的#要去掉.

-两个路径要改成这样:(添加后面的两个hdf5的路径, 否则编译时报hdf5错误)

# Whatever else you find you need goes here.

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include/usr/include/hdf5/serial

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib/usr/lib/x86_64-linux-gnu/hdf5/serial

一份完整的python3.6配置文件.（这里黑色加粗字体的位置就是我们要做修改注意的地方）

## Refer to http://caffe.berkeleyvision.org/installation.html

# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).

# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).

CPU_ONLY := 1#（因为我们要配置cpu版本的，所以这里把引号去掉）

# uncomment to disable IO dependencies and corresponding data layers

# USE_OPENCV := 0

# USE_LEVELDB := 0

# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)

# You should not set this flag if you will be reading LMDBs with any

# possibility of simultaneous read and write

# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3

OPENCV_VERSION := 3#（我们的opencv是3版本的，所以这里我们把引号去掉，之前opencv3已经安装并编译成功）

# To customize your choice of compiler, uncomment and set the following.

# N.B. the default for Linux is g++ and the default for OSX is clang++

# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.

CUDA_DIR := /usr/local/cuda

# On Ubuntu 14.04, if cuda tools are installed via

# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:

# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.

# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.

# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.

# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \

-gencode arch=compute_20,code=sm_21 \

-gencode arch=compute_30,code=sm_30 \

-gencode arch=compute_35,code=sm_35 \

-gencode arch=compute_50,code=sm_50 \

-gencode arch=compute_52,code=sm_52 \

-gencode arch=compute_60,code=sm_60 \

-gencode arch=compute_61,code=sm_61 \

-gencode arch=compute_61,code=compute_61

# BLAS choice:

# atlas for ATLAS (default)

# mkl for MKL

# open for OpenBlas

BLAS := atlas

# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.

# Leave commented to accept the defaults for your choice of BLAS

# (which should work)!

# BLAS_INCLUDE := /path/to/your/blas

# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path

# BLAS_INCLUDE := $(shell brew --prefix openblas)/include

# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.

# MATLAB directory should contain the mex binary in /bin.

# MATLAB_DIR := /usr/local

# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.

# We need to be able to find Python.h and numpy/arrayobject.h.

#PYTHON_INCLUDE := /usr/include/python2.7 \

# /usr/lib/python2.7/dist-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:

# Verify anaconda location, sometimes it's in root.

ANACONDA_HOME := $(HOME)/anaconda3#（我们使用的python3.6版本在anaconda3中，所以这里要取消注释，修改为你的根目录位置）

PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

$(ANACONDA_HOME)/include/python3.6m \#（这两个目录要看具体anaconda3中目录的名字，我做了对应修改）

$(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)

PYTHON_LIBRARIES := boost_python3 python3.6m#（这里我取消了注释，并且修改了内容，重新编译boost python3，后面讲）

# PYTHON_INCLUDE := /usr/include/python3.5m \

# /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.

#PYTHON_LIB := /usr/lib#（把这里注释掉）

PYTHON_LIB := $(ANACONDA_HOME)/lib#（修改为我这里的位置）

# Homebrew installs numpy in a non standard path (keg only)

# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include

# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)

# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies

# INCLUDE_DIRS += $(shell brew --prefix)/include

# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)

# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)

# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.

# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)

# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`

BUILD_DIR := build

DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171

# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.

TEST_GPUID := 0

# enable pretty build (comment to see full commands)

Q ?= @

LINKFLAGS := -Wl,-rpath,$(HOME)/anaconda3/lib#（这里是由于会出现一个关于png的错误，我后面讲）

按照流程接下来的配置应该是：

准备好了.

make pycaffe -j 4

另外, 这个make默认是用CPU单核运算,如果想要快一点, 比如我想使用四核, 在make后面加上-j4标签.

如果上面4行某一行报错之后想要重试,建议先make clean再重新开始.

如果这里出现问题，马上 make clean.

配置好了之后，make all

全部成功后才算是成功。

2.3 配置环境变量

去到CAFFE文件夹里面的python文件夹, 把当前路径记录下来(pwd). 然后输入以下命令(把记下的路径放在相应地方)

export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH

2.4 大功告成

命令行中输入 python，然后 import caffe ,基本是错的。接下来讲在编译caffe以及import caffe中出现的各种问题。

3.问题以及解决

3.1 编译时候，cannot find -lboost_python3, python3.6m

原因是这一行中，PYTHON_LIBRARIES := boost_python3 python3.6m，首先我们boost并没有编译3.6版本，其次python3.6m这个link不在 /usr/lib 。

1)，sudo ln -s /home/horsetif/anaconda3/bin/python3 /usr/lib/python3.6m(这里是建立一个软链接，把我本地目录下的python3.6加到usr/lib/中去。

2)，编译3.6版本的libboost_python3.so

首先去/usr/lib/x86_64-linux-gnu目录下查看是否有python3版本的libboost，如果有类似libboost_python35.so但是没有libboost_python3.so则需要手动建立连接。

方法为：

sudo ln-slibboost_python-py35.so libboost_python3.so

这个方法只是针对与3.5版本，现在我们重新编译，然后把编译好的结果的软链接加到/usr/lib/x86_64-linux-gnu这个目录里面。

1.首先下载安装包，然后解压，切换目录

wget -O boost_1_67_0.tar.gz http://sourceforge.net/projects/boost/files/boost/1.67.0/boost_1_67_0.tar.gz/download

tar xzvf boost_1_67_0.tar.gz

cd boost_1_67_0/

安装包也可以在http://www.boost.org/users/history/version_1_67_0.html下载

2.安装附加依赖库

sudo apt-get update

sudo apt-get install build-essential g++ python-dev autotools-dev libicu-dev build-essential libbz2-dev libboost-all-dev

linux系统自带python2.7，boost编译后会生成libboost_python27.so，但有些应用需要libboost_python3，下面我们就来编译。

进入boost源文件目录boost_1_67_0，配置仅仅编译python，python路径指向anaconda3中的python3.6。

./bootstrap --with-libraries=python --with-toolset=gcc

./b2 --with-python include="/home/horsetif/anaconda3/include/python3.6m/"

sudo ./b2 install

编译安装成功后，/usr/local/lib下会有libboost_python36.so和libboost_python36.a，有些应用link时需要的是libboost_python3.so或者libboost_python3.a，我们建个软链：

cd /usr/local/lib

sudo ln -s libboost_python-py36.so libboost_python3.so

sudo ln -s libboost_python-py36.a libboost_python3.a

这时，我们要把/usr/local/lib 中，相关文件，建立相对于名称的软链接到 /usr/lib/x86_64-linux-gnu中。

sudo cp /usr/local/lib/libboost_python36.a /usr/lib/x86_64-linux-gnu/libboost_python_python36.a

sudo cp /usr/local/lib/libboost_python36.so.1.67.0 /usr/lib/x86_64-linux-gnu/libboost_python3.so

3.2 多个opencv问题

错误：make pycaffe可以通过，但是make all时出现错误，具体错误我没有记录，但是好像是因为系统装了多个opencv的原因，不管怎么说，至少可以在python环境下import caffe了

make all时出现的错误：

.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'

.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector >&, std::vector > const&)'

.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'

collect2: error: ld returned 1 exit status

Makefile:625: recipe for target '.build_release/tools/upgrade_net_proto_text.bin' failed

make: *** [.build_release/tools/upgrade_net_proto_text.bin] Error 1

make: *** Waiting for unfinished jobs....

解决办法：也是，的确在这之前我分别在python 3.5和python 2.7下面装了opencv，而且该教程前面又装了libopencv-dev，所以我就怀疑是这个，但是我尝试着卸载了opencv还是import出现同样的错误，最后不断找解决方案，也尝试了他们的各种解决方法，都不知道make clean然后又make all了多少次，还是不管用。最后在网站https://github.com/BVLC/caffe/issues/2348#issuecomment-95156848看到有人说”Uncomment

if you're using OpenCV 3”于是我在配置文件中去掉 ”OPENCV_ VERSION :=3“的注释，我又make clean，接着make all,奇迹终于出现了，上面的错误消失了,居然make all成功了。

3.3 matplotlib 问题

3、错误：现在可以import caffe了，但是又出现了以下错误：

>>> import caffe

Traceback (most recent call last):

File "", line 1, in

File "/home/.../Downloads/caffe-master/python/caffe/__init__.py", line 1, in

from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver, NCCL, Timer

File "/home/.../Downloads/caffe-master/python/caffe/pycaffe.py", line 15, in

import caffe.io

File "/home/..../Downloads/caffe-master/python/caffe/io.py", line 2, in

import skimage.io

File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/__init__.py", line 15, in

reset_plugins()

File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 93, in reset_plugins

_load_preferred_plugins()

File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 73, in _load_preferred_plugins

_set_plugin(p_type, preferred_plugins['all'])

File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 85, in _set_plugin

use_plugin(plugin, kind=plugin_type)

File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 255, in use_plugin

_load(name)

File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/manage_plugins.py", line 299, in _load

fromlist=[modname])

File "/home/.../Downloads/yes/lib/python3.5/site-packages/skimage/io/_plugins/matplotlib_plugin.py", line 3, in

import matplotlib.pyplot as plt

File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/pyplot.py", line 36, in

from matplotlib.figure import Figure, figaspect

File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/figure.py", line 40, in

from matplotlib.axes import Axes, SubplotBase, subplot_class_factory

File "/home/..../Downloads/yes/lib/python3.5/site-packages/matplotlib/axes/__init__.py", line 4, in

from ._subplots import *

File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/axes/_subplots.py", line 10, in

from matplotlib.axes._axes import Axes

File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/axes/_axes.py", line 23, in

import matplotlib.dates as _ # <-registers a date unit converter

File "/home/.../Downloads/yes/lib/python3.5/site-packages/matplotlib/dates.py", line 126, in

from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY, MONTHLY, WEEKLY, DAILY, HOURLY, MINUTELY, SECONDLY)

File "/home/.../Downloads/yes/lib/python3.5/site-packages/dateutil/rrule.py", line 55

raise ValueError, "Can't create weekday with n == 0"

SyntaxError: invalid syntax

错误解决办法：从上面可以看出也就是在 import matplotlib.pyplot as plt 时出现错误，百度了以下，尝试了几种解决方案，原来是matplotlib版本太低了，需要升级，于是我pip3 install matplotlib --upgrade 了以下，果然解决了，但是又出现了下面的问题。

3.4 numpy 问题

错误：Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06)

[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import caffe

Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

解决方案：继续百度了下，哈哈，原来是因为numpy版本过低导致，于是我 conda update numpy ，哈哈又成功了。。所以，我的anaconda里边的包必须得升级啊，里边的包都还是安装anaconda时系统自带的，太老了。

3.5 错误信息

//home/idc/anaconda3/lib/libpng16.so.16: undefined reference to `inflateValidate@ZLIB_1.2.9'

后面还有几行，主要是这个。

我的解决方法是在Makefile.config加入下列信息：

LINKFLAGS := -Wl,-rpath,$(HOME)/anaconda3/lib

3.6 错误信息

可能会出现下面的错误：

Failed to include caffe_pb2, things might go wrong!

Traceback (most recent call last):

File "", line 1, in

File "/home/david/download/caffe-master/python/caffe/__init__.py", line 4, in

from .proto.caffe_pb2 import TRAIN, TEST

File "/home/david/download/caffe-master/python/caffe/proto/caffe_pb2.py", line 7, in

from google.protobuf import reflection as _reflection

File "/home/david/opt/anaconda3/lib/python3.5/site-packages/google/protobuf/reflection.py", line 68, in

from google.protobuf.internal import python_message

File "/home/david/opt/anaconda3/lib/python3.5/site-packages/google/protobuf/internal/python_message.py", line 848

except struct.error, e:

SyntaxError: invalid syntax

这是因为 protobuf 不支持 python3，解决方案是安装pip install protobuf-py3, 一个 python3.x 版本的替代包。‘’

4.错误代码总汇：

opencv 配置问题

make all

.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector>&, std::vector > const&)'

.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'

collect2: error: ld returned 1 exit status

Makefile:625: recipe for target '.build_release/tools/compute_image_mean.bin' failed

make: *** [.build_release/tools/compute_image_mean.bin] Error 1

python版本与编译caffe版本问题

from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \

ImportError: /home/horsetif/Software/caffe/python/caffe/../../build/lib/libcaffe.so.1.0.0: undefined symbol: _ZN2cv6imreadERKNS_6StringEi

File "/home/horsetif/Software/caffe/python/caffe/pycaffe.py", line 13, in from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \

ImportError: dynamic module does not define module export function (PyInit__caffe)

boost 配置问题

CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp

/usr/bin/ld: cannot find -lboost_python3

collect2: error: ld returned 1 exit status

Makefile:507: recipe for target 'python/caffe/_caffe.so' failed

make: *** [python/caffe/_caffe.so] Error 1

boost修改好了之后没有放对位置

File "/home/horsetif/Software/caffe/python/caffe/pycaffe.py", line 13, in from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \

ImportError: libboost_python36.so.1.67.0: cannot open shared object file: No such file or directory

matplotlib 问题

from matplotlib.axes._axes import Axes File "/home/horsetif/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 23, inimport matplotlib.dates as _ # <-registers a date unit converter File "/home/horsetif/anaconda3/lib/python3.6/site-packages/matplotlib/dates.py", line 148, in from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY,

File "/home/horsetif/anaconda3/lib/python3.6/site-packages/dateutil/rrule.py", line 55

raise ValueError, "Can't create weekday with n == 0"

^

SyntaxError: invalid syntax