OutOfRangeError (see above for traceback): End of sequence

[xuan20050301]

(sound_separation) PS C:\Users\28231\Desktop\Wave-U-Net-master> python training.py
WARNING - Waveunet Training - No observers have been added to this run
INFO - Waveunet Training - Running command 'run'
INFO - Waveunet Training - Started
SCRIPT START
EPOCH: 0
Dataset ready!
Training...
Sep_Vars: 22
Num of variables15
2025-02-22 14:16:42.580519: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2025-02-22 14:16:43.085179: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties:
name: NVIDIA GeForce RTX 4060 Laptop GPU major: 8 minor: 9 memoryClockRate(GHz): 2.37
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.93GiB
2025-02-22 14:16:43.085321: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2025-02-22 14:16:43.621528: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2025-02-22 14:16:43.621605: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0
2025-02-22 14:16:43.621713: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N
2025-02-22 14:16:43.621933: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6690 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 4060 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.9)
ERROR - Waveunet Training - Failed after 0:00:02!
Traceback (most recent calls WITHOUT Sacred internals):
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call
return fn(*args)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[4,16,1], [4,16,1], [4,16,1], [4,16,1], [4,16,1]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: IteratorGetNext/_9 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_117_IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent calls WITHOUT Sacred internals):
File "training.py", line 162, in run
sup_model_path, sup_loss = optimise()
File "training.py", line 137, in optimise
model_path = train(load_model=model_path)
File "training.py", line 105, in train
_, _sup_summaries = sess.run([separator_solver, sup_summaries])
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\client\session.py", line 900, in run
run_metadata_ptr)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run
run_metadata)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[4,16,1], [4,16,1], [4,16,1], [4,16,1], [4,16,1]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: IteratorGetNext/_9 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_117_IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Caused by op 'IteratorGetNext', defined at:
File "training.py", line 152, in
@ex.automain
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\sacred\experiment.py", line 137, in automain
self.run_commandline()
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\sacred\experiment.py", line 260, in run_commandline
return self.run(cmd_name, config_updates, named_configs, {}, args)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\sacred\experiment.py", line 209, in run
run()
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\sacred\run.py", line 221, in call
self.result = self.main_function(*args)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\sacred\config\captured_function.py", line 46, in captured_function
result = wrapped(*args, **kwargs)
File "training.py", line 162, in run
sup_model_path, sup_loss = optimise()
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\sacred\config\captured_function.py", line 46, in captured_function
result = wrapped(*args, **kwargs)
File "training.py", line 137, in optimise
model_path = train(load_model=model_path)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\sacred\config\captured_function.py", line 46, in captured_function
result = wrapped(*args, **kwargs)
File "training.py", line 41, in train
batch = iterator.get_next()
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 370, in get_next
name=name)), self._output_types,
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 1495, in iterator_get_next
output_shapes=output_shapes, name=name)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\framework\ops.py", line 3392, in create_op
op_def=op_def)
File "C:\Users\28231\miniconda3\envs\sound_separation\lib\site-packages\tensorflow\python\framework\ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): End of sequence
[[Node: IteratorGetNext = IteratorGetNextoutput_shapes=[[4,16,1], [4,16,1], [4,16,1], [4,16,1], [4,16,1]], output_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: IteratorGetNext/_9 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_117_IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]