TensorFlow) Fixing the "Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence" error
This post was migrated from Tistory. You can find the original here.
Environment
tensorflow/tensorflow:2.16.1-gpu (docker)
RTX 3060
TF) OUT_OF_RANGE: End of sequence error
I was training a classification model and wanted to do transfer learning on a ResNet model using TF Keras.
1
tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
Training kept failing every other epoch, along with the log message above.
The pattern was: epoch 1 trains fine, epoch 2 immediately ends with the error log and accuracy:0, loss:0, epoch 3 trains fine, and so on.
Suspicion 1.
Is the problem with how the last epoch of steps_per_epoch pads the data by repeating it?
I set steps_per_epoch to math.ceil(number of data / batchSize).
For example, if the number of data = 1001 and batchSize = 32, then steps_per_epoch = 32.
32 x 31 = 992, and 1001 - 992 = 9, so on the last step of the epoch, 23 data points would be padded by repetition. I suspected this might be the issue, but even after adjusting the data count to 1024, the problem persisted.
Suspicion 2.
Is the problem in def load_data(), on the data loading side?
I was using flow_from_directory for data loading and ImageDataGenerator for data augmentation, so I suspected the issue might be there.
I tried switching to image_dataset_from_directory for loading and tf.keras.Sequential for augmentation, but that didn’t fix it either.
Suspicion 3. (Resolved)
https://github.com/tensorflow/tensorflow/issues/68593
(The reproduction steps are different, but the error log is identical.)
The last comment says: “I am running into the same issue. This issue was not there with tf 2.13.”
After seeing that comment about there being no issue with 2.13, I switched to the tensorflow/tensorflow:2.13.0-gpu container.
Everything else — code, data, the rest of the environment — stayed the same, and just switching the TF container from 2.16 to 2.13 made it work properly.
No more OUT_OF_RANGE error, and no more epochs getting skipped.