Pages

Tuesday, August 15, 2017

TensorFlow: could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

When running the TensorFlow's object-detection model inference with my own dataset, I got the following error (with Nvidia GTX 970, CUDA 8, TensorFlow 1.2.1 through pip, and Ubuntu 16.04):
2017-08-15 21:18:06.254989: E tensorflow/stream_executor/cuda/cuda_dnn.cc:359] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2017-08-15 21:18:06.255027: E tensorflow/stream_executor/cuda/cuda_dnn.cc:326] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM 2017-08-15 21:18:06.255036: F tensorflow/core/kernels/conv_ops.cc:671] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)
Solution from https://github.com/tensorflow/tensorflow/issues/6698, by enabling the GPU's allow_growth flag, solved my problem:
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)
That's all :)

3 comments:

  1. OMG MATE THANK YOU SO MUCH, I HATE A 970 TOO, I THINK IT'S A GENERAL PROBLEM !!!!!!!!!!!!!! THANK YOU VERY MUCH !!!!!!!!!!!!!
    GREETINGS from Portugal!!!!

    ReplyDelete
  2. This solution works ! Thanks,

    ReplyDelete