I am running with a batch size of two, here is the output during training. Any help would be appreciated.
/home/allen/.pyenv/versions/metavision/lib/python3.10/site-packages/lightning_fabric/connector.py:571: `precision=16` is supported for historical reasons but its usage is discouraged. Please set your precision to 16-mixed instead!
Using 16bit Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
`Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used..
`Trainer(limit_test_batches=1.0)` was configured so 100% of the batches will be used..
You are using a CUDA device ('NVIDIA RTX 4000 Ada Generation Laptop GPU') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read
https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision2025-11-11 15:07:18.461175: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-11-11 15:07:18.499993: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-11-11 15:07:19.500305: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
No Learning Rate Scheduler
/home/allen/.pyenv/versions/metavision/lib/python3.10/site-packages/pytorch_lightning/utilities/model_summary/model_summary.py:231: Precision 16-mixed is not supported by the model summary. Estimated model size in MB will not be accurate. Using 32 bits instead.
| Name | Type | Params | Mode
---------------------------------------------------------
0 | detector | SingleStageDetector | 24.1 M | train
---------------------------------------------------------
24.1 M Trainable params
0 Non-trainable params
24.1 M Total params
96.297 Total estimated model params size (MB)
108 Modules in train mode
0 Modules in eval mode
Sanity Checking: | | 0/? [00:00<?, ?it/s]/home/allen/.pyenv/versions/metavision/lib/python3.10/site-packages/metavision_ml/data/scheduler.py:193: RuntimeWarning: overflow encountered in scalar negative
excess_duration[mask] = duration[mask] % - total_tbins_delta_t
Sanity Checking DataLoader 0: 0%| | 0/2 [00:00<?, ?it/s]/home/allen/.pyenv/versions/metavision/lib/python3.10/site-packages/torch/functional.py:554: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:4322.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Sanity Checking DataLoader 0: 100%|██████████████████████████████| 2/2 [00:00<00:00, 2.31it/s]==> Start evaluation
Warning : No boxes were added to the evaluation !
mean_ap75 : -1.0
mean_ar_medium : -1.0
mean_ap50 : -1.0
mean_ap_big : -1.0
mean_ap : -1.0
mean_ap_small : -1.0
mean_ap_medium : -1.0
mean_ar_big : -1.0
mean_ar_small : -1.0
mean_ar : -1.0
Epoch 0: 100%|██████████████████████████████████████| 581/581 [01:58<00:00, 4.88it/s, v_num=1==> Start evaluationer 0: 100%|███████████████████████████████| 583/583 [02:15<00:00, 4.29it/s]
Warning : No boxes were added to the evaluation !
mean_ap75 : -1.0
mean_ar_medium : -1.0
mean_ap50 : -1.0
mean_ap_big : -1.0
mean_ap : -1.0
mean_ap_small : -1.0
mean_ap_medium : -1.0
mean_ar_big : -1.0
mean_ar_small : -1.0
mean_ar : -1.0
Epoch 1: 100%|██████████████████████████████████████| 581/581 [02:03<00:00, 4.69it/s, v_num=1==> Start evaluationer 0: 100%|███████████████████████████████| 583/583 [02:19<00:00, 4.17it/s]
Warning : No boxes were added to the evaluation !
mean_ap75 : -1.0
mean_ar_medium : -1.0
mean_ap50 : -1.0
mean_ap_big : -1.0
mean_ap : -1.0
mean_ap_small : -1.0
mean_ap_medium : -1.0
mean_ar_big : -1.0
mean_ar_small : -1.0
mean_ar : -1.0
Epoch 2: 100%|██████████████████████████████████████| 581/581 [02:05<00:00, 4.63it/s, v_num=1==> Start evaluationer 0: 100%|███████████████████████████████| 583/583 [02:22<00:00, 4.09it/s]
Warning : No boxes were added to the evaluation !
mean_ap75 : -1.0
mean_ar_medium : -1.0
mean_ap50 : -1.0
mean_ap_big : -1.0
mean_ap : -1.0
mean_ap_small : -1.0
mean_ap_medium : -1.0
mean_ar_big : -1.0
mean_ar_small : -1.0
mean_ar : -1.0