fix CUDA trouble with TF2 (GPU) on Ubuntu

I couldn't find similar case on the web, so I introduce my trouble and method to fix it.

At first, my environment is follows.

Ubuntu 18.04 (amd64) + TensorFlow 2.3.1 (GPU)

• CUDA : 10.1.243
• cuDNN : 7.6.5.32

Error occurred : Cannot dlopen some GPU libraries

_

At some point, I cannot use GPU for TF2 task.

(tf2) $ python3
>>> import tensorflow as tf
>>> tf.test.is_gpu_available()
...
False

And important error logs are follows.

...
Could not load dynamic library 'libcublas.so.10'; \
dlerror: libcublas.so.10: cannot open shared object file: \
No such file or directory;
...
Cannot dlopen some GPU libraries. \
Please make sure the missing libraries mentioned above \
are installed properly if you would like to use GPU.

Why is libcublas.so missing ?

_

I searched libcublas.so, but it didn't exist at expected path.

$ find /usr/lib -name "libcublas.so*"
$ find /usr/local/cuda/ -name "libcublas.so*"

then, I checked dirs on /usr/local/.

$ ls -1F /usr/local/ | grep cuda
cuda@
cuda-10.1/
cuda-10.2/

Why is 10.2 exist ??

$ find /usr/local/cuda-10.2 -name "libcublas.so*"

libcublas.so.10 exists in /usr/local/cuda-10.2/ !
But, it should exist in /usr/local/cuda-10.1/.

I checked dependency of deb package.

$ apt show cuda-toolkit-10-1
$ apt show cuda-libraries-dev-10-1

cuda-toolkit-10-1
-> cuda-libraries-dev-10-1 (>= 10.1.243)
--> libcublas-dev (>= 10.2.1.243)

cause : wrong version of 'libcublas10' and 'libcublas-dev'.

_

Please refer cuda repos.

versions of libcublas10 and libcublas-dev

• 10.1.0.105-1
• 10.2.0.163-1
• 10.2.0.168-1
• 10.2.1.243-1
• 10.2.2.89-1
• 10.2.2.214-1

versions of cuda

• 10.1.105-1
• 10.1.168-1
• 10.1.243-1
• 10.2.89-1

these minor versions are same.
so maybe libcublas 10.2.2.89-1 and 10.2.2.214-1 are for CUDA 10.2.

$ dpkg -l | grep libcublas
ii  libcublas-dev                          10.2.2.214-1
ii  libcublas10                            10.2.2.214-1

• actual installed : 10.2.2.214-1
• expected : 10.2.1.243-1

Fix it

_

$ sudo apt install --reinstall libcublas10=10.2.1.243-1 libcublas-dev=10.2.1.243-1

But, they are going to appear upgradable candidate. So we need to prevent it.

$ sudo apt-mark hold libcublas10
$ sudo apt-mark hold libcublas-dev

resolved.

(tf2) $ python3
>>> import tensorflow as tf
>>> tf.test.is_gpu_available()
...
True

Thanks for your reading ! 👍