Setting up OpenCL 2.0 Caffe @ lubuntu 1

If you have an AMD GPU and you want to do some Deep Learning, there is not much choice around. You have to pick something based on OpenCL, until AMD starts to support CUDA. The Boltzman Initiative announced recently  will bring CUDA and C++ support for AMD’s GPUs, but we have to wait for that a bit longer. As for now lets have a look how to set up a OpenCL-Caffe working with AMD’s GPU.

As I am not big fan of Unity I picked lubuntu, which is much lighter than the original Ubuntu. Lets assume you managed to install the 15.10, and you are ready to go. After the installation first thing I did was update and upgrade all the defaults and restart.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

After that the system should report to have a kernel 4.2.0-22

uname -a
Linux thesun 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Now there are a couple of files to download. Some of them are not needed straightaway, but might be useful in the future. I assume all of the files will be saved in $HOME/Downloads.

  • AMD Linux Download Center – a version of the drivers for your system. At the moment of writing the latest available version was Crimson Edition 15.12.
  • AMD APP SDK – the SDK for OpenCL provided by AMD, at the moment of writing the latest available version was v3.0.130-135.
  • CodeXL – debugging, profiling and analysis. Not needed at the moment. I have downloaded the rmp file.
    Now we need to download a couple of packages. I agree there are a lot of them. A couple are just the software I think is useful in general or the ones we will need in the later stages (e.g. checkinstall, aptitude, htop, alien, vim, git, cmake-curses-gui), others are prerequisites for AMD GPU driver (e.g., mesa-common-dev, execstack, dkms,lib32gcc1), and really many are required to build OpenCV, which is needed to build OpenCL-Caffe.

    sudo apt-get install apt-show-versions cmake cmake-curses-gui \
     checkinstall gfortran aptitude htop gfortran-5 alien vim ant \
     default-jdk dh-modaliases execstack dkms lib32gcc1 unzip     \
     libfftw3-3 libfftw3-dev libX11-dev mesa-common-dev zlib1g-dev\
     build-essential doxygen protobuf-compiler liblmdb0           \
     libleveldb-dev liblmdb-dev libsnappy1v5 libsnappy-dev        \
     libgl1-mesa-dev freeglut3 freeglut3-dev libglm-dev mesa-utils\
     libatlas-dev libatlas-base-dev gcc g++ git libunwind-dev     \
     libopenblas-base libopenblas-dbg libopenblas-dev             \
     python-pip ipython-scipy libleveldb1v5 libblas-dev valgrind

    Prerequisites for OpenCV.

    sudo apt-get install qt5-default libvtk6-dev zlib1g-dev       \
     libjpeg-dev libwebp-dev libpng-dev libtiff5-dev libjasper-dev\
     libopenexr-dev libgdal-dev libdc1394-22-dev libavcodec-dev   \
     libavformat-dev libswscale-dev libtheora-dev libvorbis-dev   \
     libxvidcore-dev libx264-dev yasm libopencore-amrnb-dev       \
     libopencore-amrwb-dev libv4l-dev libxine2-dev libtbb-dev     \
     libeigen3-dev  build-essential cmake git libgtk2.0-dev       \
     pkg-config python-dev python-numpy libdc1394-22 libqt4-dev   \
     libdc1394-22-dev libjpeg-dev libpng12-dev  libjasper-dev     \
     libavcodec-dev libavformat-dev libswscale-dev libtbb-dev     \
     libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev       \
     libv4l-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev  \
     libopencore-amrwb-dev libtheora-dev libvorbis-dev x264       \
     libxvidcore-dev  v4l-utils libgstreamer1.0-dev libopencv-dev \
     libgstreamer-plugins-base1.0-dev libtiff5-dev gphoto2        \
     libtiff-tools libbz2-dev libgphoto2-dev libavresample-dev    \
     gtk+-3.0-dev libv4l-dev libudev-dev libglu1-mesa-dev         \
     g++-multilib  libtifiles2-9 tcl-vtk  libpng++-dev libpng3    \
     zlib1g-dbg swig swig2.0 libjasper-runtime libpq-dev          \
     vtk6.2 libgdal-dev libx264-dev

    Before doing anything lets set up some environment variables. Some of the folders they refer to do not exist yet, but they will in the future as we keep on going. Open ~/.bashrc for editing and paste at the end all the exports listed below.

    export PATH="/opt/boost_1_60_0/bin:$PATH"
    export LD_LIBRARY_PATH="/usr/local/lib64:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/AMDAPPSDK-3.0/lib/x86_64:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/GLES_SDK_v31/x86-64/lib:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/netlib_blas:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/acml5.3.1/gfortran64_mp/lib:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/boost_1_60_0/lib:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/googletest/lib:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/clBLAS/lib64:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/clRNG/lib64:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/opt/clFFT/lib64:$LD_LIBRARY_PATH"
    export TMP="/tmp"
    export TMPDIR="/tmp"
    export ACML_ROOT="/opt/aclm5.3.1"
    export ACML_INCLUDE_DIRS="/opt/acml5.3.1/gfortran64_mp/include"
    export ACML_LIBRARIES="/opt/acml5.3.1/gfortran64_mp/lib"
    export BOOST_ROOT="/opt/boost_1_60_0"
    export CLBLAS_STORAGE_PATH="/opt/clBLAS"
    export CLBLAS_ROOT="/opt/clBLAS"
    export CLBLAS_INCLUDE_DIR="/opt/clBLAS/include"
    export CLRNG_ROOT="/opt/clRNG"
    export CLSPARSE_ROOT="/opt/clSPARSE"
    export CLFFT_ROOT="/opt/clFFT"
    export CAFFE_ROOT="/opt/opencl-caffe"
    export CLFFT_INCLUDE_DIR="/opt/clFFT/include"
    export CLRNG_INCLUDE_DIR="/opt/clRNG/include"
    export CLSPARSE_ROOT="/opt/clSPARSE/include"
    export FFTW_LIBRARIES="/usr/lib/x86_64-linux-gnu"
    export FFTW_INCLUDE_DIRS="/usr/include"
    export GTEST_LIBRARY="/opt/googletest/lib"
    export GTEST_INCLUDE_DIR="/opt/googletest/include"
    export GTEST_ROOT="/opt/googletest"
    export GFLAGS_INCLUDE_DIRS="/opt/gflags/include"
    export GFLAGS_LIBRARY="/opt/gflags/lib"
    export GLOG_INCLUDE_DIR="/opt/glog/include"
    export GLOG_LIBRARY="/opt/glog/lib"
    export AMDAPPSDK_ROOT="/opt/AMDAPPSDK-3.0"
    export AMDAPPSDKROOT="/opt/AMDAPPSDK-3.0"
    export AMDAPP="/opt/AMDAPPSDK-3.0"
    export CXX=g++
    export CC=gcc
    export FC=gfortran

    Having all the essentials we may continue. The first thing will be to install the driver.

    cd $HOME/Downloads
    unzip radeon-crimson-*.zip
    cd fglrx-*/
    sudo ./amd-driver-installer-*.run
    sudo aticonfig --initial

    Now you may check if the driver was installed successfully

    dpkg -l fglrx fglrx-core fglrx-dev fglrx-amdcccle

    The output will probably be like the one below, with a msg about missing fglrx-dev.

    #| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
    #|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
    #||/ Name                                  Version                 Architecture            Description
    #ii  fglrx                                 2:15.302-0ubuntu1       amd64                   Video driver for the AMD graphics accelerators
    #ii  fglrx-amdcccle                        2:15.302-0ubuntu1       amd64                   Catalyst Control Center for the AMD graphics accelerators
    #ii  fglrx-core                            2:15.302-0ubuntu1       amd64                   Minimal video driver for the AMD graphics accelerators
    #dpkg-query: no packages found matching fglrx-dev

    The fglrx-dev*.deb file is in the folder with all other drivers so we may just install it from there.

    sudo dpkg -i fglrx-dev*_amd64.deb

    Now there are a couple of libraries that we need to download and build from the source (Boost, GFlags, GLog, GTest, clRNG, clFFT, VTK, libtiff 4.0, opencv) or just install (AMD SDK)

    AMD SDK OpenCl

    cd $HOME/Downloads
    tar -xvf AMD-APP-SDKInstaller-v3.0.130.135-GA-linux64.tar.bz2
    sudo ./ 

    There is a problem with one shared library provided with the SDK. Apparently it is older than the one provided with the driver. To make our lives easier lets just rename it so it does not get in the way (you may try running


    after installing the SDK, if the library is in the way it should crash with segmentation fault).

    sudo mv /opt/AMDAPPSDK-3.0/lib/x86_64/ /opt/AMDAPPSDK-3.0/lib/x86_64/libamdocl12cl64.so_old
    sudo ldconfig

    Boost 1.60

    cd $HOME/Downloads
    wget --output-document=boost_1_60_0.tar.gz
    tar -zxvf boost_1_60_0.tar.gz
    cd boost_1_60_0/
    sudo ./ --prefix=/opt/boost_1_60_0
    sudo ./b2 install -j9 --with-program_options address-model=64  link=static,shared threading=multi --with-python --with-system --with-thread --with-mpi --with-signals --with-regex --with-timer --with-iostreams --with-graph_parallel --with-serialization

    Google Flags

    cd $HOME/Downloads
    git clone
    cd gflags
    mkdir build
    cd build
    make -j4
    sudo checkinstall --pkgname=gflags

    Google Test

    cd $HOME/Downloads
    git clone
    cd googletest
    mkdir build
    cd build
    cmake .. -DCMAKE_INSTALL_PREFIX=/opt/googletest -DBUILD_SHARED_LIBS=ON
    make -j4
    sudo checkinstall --pkgname=gtest
    sudo updatedb
    sudo ldconfig

    Google Log

    cd $HOME/Downloads
    git clone
    cd glog
    mkdir build
    cd build
    sudo make -j4
    sudo checkinstall  --pkgname=glog


    cd $HOME/Downloads
    git clone
    cd clRNG
    mkdir build
    cd build
    make -j9
    sudo checkinstall --pkgname=clrng

    You may run a small clRNG test



    cd $HOME/Downloads
    git clone
    cd clFFT
    mkdir build
    cd build
    make -j9
    sudo checkinstall --pkgname=clfft

    You may run a small clFFT tests


    clSPARSE – at the moment of writing (January 2016) it does not build against OpenCL 2.0, see

    cd $HOME/Downloads
    git clone
    cd clSPARSE
    mkdir build
    cd build
    make -j9
    sudo checkinstall --pkgname=clsparse

    VTK – Although we have installed vtk6.2 and libvtk6-dev it may happen that while building OpenCV CMake will ask about the folder with vtk build files. I thought it will be easier just to download it and build it locally.

    cd $HOME/Downloads
    git clone
    cd VTK
    git checkout --track -b release origin/release
    git checkout v6.2.0
    git tag
    mkdir build 
    cd build
    make -j9

    libTiFF 4.0 – When I was trying to build OpenCV it was complaining that there is no with symbols for version 4.0. I found it online and build it, here is the code.

    cd $HOME/Downloads
    sudo dpkg -i libtiff4_3.9.7-2ubuntu1_amd64.deb 
    sudo ldconfig

    libLZMA – Similarilly to libtiff, I got complaints that does not hold version details. I found the proper one in new version of XZ.

    cd $HOME/Downloads
    tar -zxvf xz-5.2.2.tar.gz
    cd xz-5.2.2/
    sudo checkinstall --pkgname=xz-5.2.2


    sudo apt-get install  
    cd $HOME/Downloads
    git clone
    cd FFmpeg
    ./configure --enable-gpl --enable-libfaac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-nonfree --enable-postproc --enable-version3 --enable-x11grab --enable-shared
    sudo make  -j9
    sudo checkinstall --pkgname=FFmpeg

    OpenCL 2.0/2.1 header files from Khronos

    sudo mkdir -p $TGT_DIR && cd $TGT_DIR
    sudo wget{opencl,cl_platform,cl,cl_ext,cl_gl,cl_gl_ext}.h


    cd $HOME/Downloads
    git clone
    cd clBLAS
    git checkout v2.8
    mkdir build
    cd build
    cmake ../src \
    -DOPENCL_VERSION:STRING=2.0                                 \
    -DCMAKE_INSTALL_PREFIX=/opt/clBLAS                          \
    -DBLAS_DEBUG_TOOLS=ON                                       \
    -DBUILD_PERFORMANCE=ON                                      \
    -DBUILD_SHARED_LIBS=ON                                      \
    -DUSE_SYSTEM_GTEST=ON                                       \
    -DOPENCL_LIBRARIES=/usr/lib/                  \
    -DOPENCL_INCLUDE_DIRS=/opt/AMDAPPSDK-3.0/include            \
    make -j9
    sudo checkinstall --pkgname=clblas

    If make does not work because some references could not be found you have to make sure that links to the library coming from Netlib’s BLAS. You may do that by running

    sudo update-alternatives --display
    sudo update-alternatives --config

    If you see that /usr/lib/libblas/ is not default then run the following two lines

    sudo update-alternatives --install /usr/lib/ /usr/lib/libblas/ 60
    sudo update-alternatives --install /usr/lib/ /usr/lib/libblas/ 60

    There are a couple of test to check. They are all located in /opt/clBLAS/bin/.

    EDIT: As for 2015-03-04, the issues below are no longer issues, as they have been fixed and merged to a development branch of the clBLAS project on GitHub.

    The test-functional will fail on the first two tests because one of the methods instead of throwing an error just quits execution. You may skip those tests by editing


    and commenting out those tests

    TEST(ERROR, InvalidCommandQueue) {
        ErrorClass<GemmMetod<float> > ec;
    TEST(ERROR, InvalidEventWaitList) {
        ErrorClass<GemmMetod<float> > ec;

    After doing so the result of the run of test-functional was

    [----------] Global test environment tear-down
    [==========] 713 tests from 5 test cases ran. (230487 ms total)
    [  PASSED  ] 706 tests.
    [  FAILED  ] 7 tests, listed below:
    [  FAILED  ] ERROR.InvalidMemObject
    [  FAILED  ] ERROR.InvalidValue
    [  FAILED  ] ERROR.InvalidValuesymm
    [  FAILED  ] ERROR.InvalidValuehemm
    [  FAILED  ] THREAD.cgemm
    [  FAILED  ] THREAD.dgemm
    [  FAILED  ] THREAD.zgemm

    Another test to run would be test-short, which is a short version of test-correctness but the long version may run for a couple of hours. At the moment on my instance all tests in test-short were PASSED, but test-correctness crashed with segmentation fault on test ColumnMajor_SmallRange/SYR2K.ssyr2k/261.

    IMPORTANT: if you want to build OpenCV with OpenCL 2.0 you have to comment out one line in opencv/cmake/OpenCVDetectOpenCL.cmake. In the current version it line number 10

    set(OPENCL_INCLUDE_DIR "${OpenCV_SOURCE_DIR}/3rdparty/include/opencl/1.2")

    If you do not do that cmake will use the OpenCL 1.2 provided with the source code.

    cd $HOME/Downloads
    git clone
    cd opencv
    git checkout --track -b release
    git checkout tags/3.1.0
    git tag
    mkdir build
    cd build
    cmake .. 
    -DWITH_OPENCL=ON                \
    -DHAVE_OPENCL=ON                \
    -DHAVE_CLAMDBLAS=ON             \
    -DHAVE_CLAMDFFT=ON              \
    -DWITH_QT=ON                    \
    -DWITH_OPENGL=ON                \
    -DWITH_TBB=ON                   \
    -DWITH_GDAL=ON                  \
    -DWITH_XINE=ON                  \
    -DBUILD_EXAMPLES=ON             \
    -DBUILD_TESTS=ON                \
    -DWITH_CUFFT=OFF                \
    -DWITH_CUDA=OFF                 \
    -DWITH_TIFF=ON                  \
    -DBUILD_TIFF=OFF                \
    -DOPENCL_INCLUDE_DIR=/opt/AMDAPPSDK-3.0/include    \
    -DCMAKE_INSTALL_PREFIX=/opt/opencv                 \
    -DCLAMDFFT_ROOT_DIR=/opt/clFFT                     \
    -DCLAMDFFT_INCLUDE_DIR=/opt/clFFT/include          \
    -DCLAMDBLAS_ROOT_DIR=/opt/clBLAS                   \
    -DCLAMDBLAS_INCLUDE_DIR=/opt/clBLAS/include        \
    -DVTK_DIR=$HOME/Downloads/VTK/build                \
    sudo make -j9
    sudo checkinstall  --pkgname=opencv    


    cd $HOME/Downloads
    git clone
    cd OpenCL-caffe
    mkdir build
    cd build
    cmake ../ 
    -DCMAKE_BUILD_TYPE=Debug                           \
    -DCMAKE_INSTALL_PREFIX:PATH=/opt/opencl-caffe      \
    -DGFLAGS_INCLUDE_DIR=/opt/gflags/include           \
    -DGFLAGS_LIBRARY=/opt/gflags/lib/      \
    -DGLOG_INCLUDE_DIR=/opt/glog/include               \
    -DGLOG_LIBRARY=/opt/glog/lib/            \
    -DCMAKE_CXX_FLAGS=-Wdeprecated-declarations        \
    -DCLBLAS_LIBRARIES=/opt/clBLAS/lib64/  \
    -DCLBLAS_INCLUDE_DIRS=/opt/clBLAS/include          \
    -DOPENCL_INCLUDE_DIRS=/opt/AMDAPPSDK-3.0/include   \
    -DBLAS=open                                        \
    -DOpenCV_CONFIG_PATH=/opt/opencv/share/OpenCV      \
    -DOpenCV_DIR=/opt/opencv/share/OpenCV              \
    -DOpenCV_LIB_DIR_OPT=/opt/opencv/lib               \
    -DOpenCV_LIB_DIR_DBG=/opt/opencv/share/OpenCV      \
    -DOpenBLAS_INCLUDE_DIR=/usr/include/openblas       \
    -DOpenBLAS_LIB=/usr/lib/               \
    -DBoost_DIR=/opt/boost_1_60_0                      \
    -DBOOST_ROOT=/opt/boost_1_60_0                     \
    -DHDF5_z_LIBRARY_DEBUG=/usr/lib/x86_64-linux-gnu/  \
    sudo make -j9
    sudo checkinstall --pkgname=opencl-caffe
    cd $HOME/Downloads/OpenCL-caffe
    mkdir log

    You can view the progress of the model build by sniffing one of the log files

    tail log/caffe.INFO -f


  • If you try to build clBLAS with ACML test they may fail. Also if you try do not use the latest version 6.1, but at most 5.3.1.
  • I tried to build everything having Anaconda installed, but I face many problems with system libraries being covered by the ones from anaconda and some version discrepancies. With Python2.7 there were no problems.
  • You can install OpenCV with apt-get but it will not be build against OpenCL 2.0
  • At the moment (version 2.8, January 2016), the test-functional for clBLAS will fail on the first two tests, for more details see
  • Make sure the folder in which you run Caffe contains folder named log that is where all logging using GLog will go, and that is where you can track the model build process.
  • There are two commends that may be used to monitor your AMD GPU
    amdconfig --odgt
    amdconfig --odgc

    the first one shows GPU’s temperature, and the second one the current GPU load. If you want to feed them to some other script those two lines of code will get you the temperature and GPU load respectively.

    amdconfig --adapter=$1 --odgt | grep 'Temperature' | cut -d'-' -f2 | cut -d'.' -f1 | tr -d ' '
    amdconfig --adapter=$1 --odgc | grep 'GPU load' | cut -f1 -d'%' | cut -f2 -d':'| tr -d ' '
  • If your OpenCL/Caffe script crashes you may find some logs in /var/log/apport.log.
  • In many places through the script I used parameter -j9 after make, it makes make execute in parallel, and 9 stands for the number of threads. You can use different number.
  • If running

    ends up with segmentation fault try changing the name of /opt/AMDAPPSDK-3.0/lib/x86_64/

  • If something OpenCL related does not compile properly it is worth checking which library is being used. For example this is how it looks at my compute
    $ locate
    $ ldconfig -p | grep -i opencl (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/ (libc6,x86-64) => /usr/lib/ (libc6) => /usr/lib32/
    $ ldconfig -p | grep -i 'opencl'     | grep "=> /" | awk '{print $4}'  | xargs ls -la | grep " -> " | awk '{print $11}' | xargs locate | xargs ls -la
    	-rw-r--r-- 1 root root 43408 jun 10  2015 /usr/lib/x86_64-linux-gnu/

    the first line finds all present in the system, second one all libraries available for dynamic linking that have OpenCL in the name (case insensitive), and the last one gets the location of the that is not a link and is available for linking (other files listed after second command might actually be soft links).

  • Too see dependencies of a library on other libraries use ldd command, e.g.
    $ ldd  -v -d /usr/lib/x86_64-linux-gnu/ =>  (0x00007ffc3c3b2000) => /lib/x86_64-linux-gnu/ (0x00007fabf8e63000) => /lib/x86_64-linux-gnu/ (0x00007fabf8a98000)
    	/lib64/ (0x000056019ddfc000)
    	Version information:
    	/usr/lib/x86_64-linux-gnu/ (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/ (GLIBC_2.3) => /lib64/ (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/ (GLIBC_2.4) => /lib/x86_64-linux-gnu/ (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/
    	/lib/x86_64-linux-gnu/ (GLIBC_PRIVATE) => /lib64/ (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/ (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/
    	/lib/x86_64-linux-gnu/ (GLIBC_2.3) => /lib64/ (GLIBC_PRIVATE) => /lib64/
  • If something goes wrong for example with running tests you may try using valgrind on them to see what libraries are actually being dynamically linked e.g. after running
    valgrind /opt/clBLAS/bin/test-functional

    in the output I will see something like

    ==28773==    by 0x89F22F6: ??? (in /usr/lib/
    ==28773==    by 0x89C2E52: clIcdGetPlatformIDsKHR (in /usr/lib/
    ==28773==    by 0x6E7E786: ??? (in /usr/lib/x86_64-linux-gnu/
  • When building OpenCL-Caffe do not try to set varialbe -DBLAS to clblas. It will not work, as the only allowed options by cmake are atlas, open, and mkl.
  • If you wonder what are the other parameters that you may pass to cmake use ccmake instead. It is a visual editor for cmake.
  • To monitor temperature and use of AMD’s gpu use
    watch -n 1 "amdconfig --odgc" "amdconfig --odgt"
  • One comment on “Setting up OpenCL 2.0 Caffe @ lubuntu

    1. Reply Anonymous Feb 5,2017 11:59

      CMake is now 3.2.2

      CMake Warning (dev) at tests/CMakeLists.txt:388 (get_target_property):
      Policy CMP0026 is not set: Disallow use of the LOCATION target property.
      Run “cmake –help-policy CMP0026” for policy details. Use the cmake_policy
      command to set the policy and suppress this warning.

      The LOCATION property should not be read from target “test-correctness”.
      Use the target name directly with add_custom_command, or use the generator
      expression $, as appropriate.

    Leave a Reply