Checkpoint https://dl.fbaipublicfiles.com/detectron2/COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x/139173657/model_final_68b088.pkl not found!

[ozgurkan]

I've been getting an error like this since last night. I trained 5 models, there was no problem. Then there was such a problem. I would be glad if you could help.

WARNING [12/16 08:11:53 d2.data.datasets.coco]:
Category ids in annotations are not in [1, #categories]! We'll apply a mapping for you.

[12/16 08:11:53 d2.data.datasets.coco]: Loaded 36 images in COCO format from /content/train/_annotations.coco.json
[12/16 08:11:53 d2.data.build]: Removed 0 images with no usable annotations. 36 images left.
[12/16 08:11:53 d2.data.common]: Serializing 36 elements to byte tensors and concatenating them all ...
[12/16 08:11:53 d2.data.common]: Serialized dataset takes 0.11 MiB
[12/16 08:11:53 d2.data.detection_utils]: TransformGens used in training: [ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice'), RandomFlip()]
[12/16 08:11:53 d2.data.build]: Using training sampler TrainingSampler

AssertionError Traceback (most recent call last)
in ()
34 os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
35 trainer = CocoTrainer(cfg)
---> 36 trainer.resume_or_load(resume=False)
37 trainer.train()

2 frames
/usr/local/lib/python3.6/dist-packages/fvcore/common/checkpoint.py in load(self, path, checkpointables)
118 if not os.path.isfile(path):
119 path = self.path_manager.get_local_path(path)
--> 120 assert os.path.isfile(path), "Checkpoint {} not found!".format(path)
121
122 checkpoint = self._load_file(path)

AssertionError: Checkpoint https://dl.fbaipublicfiles.com/detectron2/COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x/139173657/model_final_68b088.pkl not found!

[amjindal]

i'm getting the same error but when i run the code on a server GPU.
AssertionError: Checkpoint https://dl.fbaipublicfiles.com/detectron2/COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x/139173657/model_final_68b088.pkl not found

The code is running perfectly fine on google colab but now after copying the code into .py file i'm facing this issue.below code is what i'm trying in my python code

cfg = get_cfg() # obtain detectron2's default config
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml"))# load values from a file
cfg.DATASETS.TRAIN = ("my_dataset_train",)
cfg.DATASETS.TEST = ("my_dataset_val",)

cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml") # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 4 #images per batch i.e if 4 GPU than each GPU will see 1 image only.
cfg.SOLVER.BASE_LR = 0.00025 # pick a good LR

cfg.SOLVER.WARMUP_ITERS = 1000
cfg.SOLVER.MAX_ITER = 50000 #adjust up if val mAP is still rising, adjust down if overfit
cfg.SOLVER.STEPS = (1000, 1500)
cfg.SOLVER.GAMMA = 0.05

#epoch = (batch_size*no of iterations)/Total images
#1 epoch means one time data is fed completly.

cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 64
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 # only has one class textline. (see https://detectron2.readthedocs.io/tutorials/datasets.html#update-the-config-for-new-datasets)
#your number of classes + 1(this comment is from older one)

cfg.TEST.EVAL_PERIOD = 500 #After this no of iteration, the validation data will evaluate the model accuracy.

#print(cfg.dump()) # print formatted configs
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = CocoTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

[ozgurkan]

I trained 5 models last night, there was no problem on google colab. I started a new tutorial and encountered this error. I guess there is a problem with the link of the model. I downloaded the model, how can I follow a way for training.

[ozgurkan]

i'm getting the same error but when i run the code on a server GPU.
AssertionError: Checkpoint https://dl.fbaipublicfiles.com/detectron2/COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x/139173657/model_final_68b088.pkl not found

The code is running perfectly fine on google colab but now after copying the code into .py file i'm facing this issue.below code is what i'm trying in my python code

cfg = get_cfg() # obtain detectron2's default config
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml"))# load values from a file
cfg.DATASETS.TRAIN = ("my_dataset_train",)
cfg.DATASETS.TEST = ("my_dataset_val",)

cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml") # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 4 #images per batch i.e if 4 GPU than each GPU will see 1 image only.
cfg.SOLVER.BASE_LR = 0.00025 # pick a good LR

cfg.SOLVER.WARMUP_ITERS = 1000
cfg.SOLVER.MAX_ITER = 50000 #adjust up if val mAP is still rising, adjust down if overfit
cfg.SOLVER.STEPS = (1000, 1500)
cfg.SOLVER.GAMMA = 0.05

#epoch = (batch_size*no of iterations)/Total images
#1 epoch means one time data is fed completly.

cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 64
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 # only has one class textline. (see https://detectron2.readthedocs.io/tutorials/datasets.html#update-the-config-for-new-datasets)
#your number of classes + 1(this comment is from older one)

cfg.TEST.EVAL_PERIOD = 500 #After this no of iteration, the validation data will evaluate the model accuracy.

#print(cfg.dump()) # print formatted configs
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = CocoTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

Do you have a chance to try the same code on colab now? I get the same error with this code.

[metempasa]

Hello Guys!

@ozgurkan @amjindal @stepancheg @likethesky @yfeldblum @ppwwyyxx Still same error at colab. You have any process how to figure it out ?

[ozgurkan]

Hello Guys!

@ozgurkan @amjindal @stepancheg @likethesky @yfeldblum @ppwwyyxx Still same error at colab. You have any process how to figure it out ?

For now, I found a manual solution. I downloaded the model from the link and saved it to my drive profile. Then I copied the model from the drive profile to the output file and made changes to the code below.

cfg.MODEL.WEIGHTS = "/content/output/model_final_68b088.pkl"

Maybe there is a better solution but I don't know. I think there is a problem in accessing the model file in the link.

[metempasa]

Thanks for your response @ozgurkan !

[ppwwyyxx]

Likely some bad combination of library versions.

The official colab tutorial works fine at the moment. Please provide reproducible instructions for the problem.

[ozgurkan]

@amjindal @metempasa @ppwwyyxx First of all, I would like to thank you all for your help.

I tried the official colab tutorial now and yes it works.I used the following code to install dependencies and detectron2.

install dependencies: (use cu101 because colab has CUDA 10.1)

!pip install -U torch==1.5 torchvision==0.6 -f https://download.pytorch.org/whl/cu101/torch_stable.html
!pip install cython pyyaml==5.1
!pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
import torch, torchvision
print(torch.version, torch.cuda.is_available())
!gcc --version

install detectron2:

!pip install detectron2==0.1.3 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.5/index.html

The official colab tutorial used the following:

install dependencies:

!pip install pyyaml==5.1
import torch, torchvision
print(torch.version, torch.cuda.is_available())
!gcc --version

install detectron2: (Colab has CUDA 10.1 + torch 1.7)

import torch
assert torch.version.startswith("1.7")
!pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.7/index.html

Installing like the official colab tutorial, the problem disappeared. But there is an interesting situation that I trained 5 models without any problem. When I switched to the sixth model, I encountered such an error, which is really weird. The code that worked 2 hours ago did not work after 2 hours.

Thank you all again. I wish you all a good work.

[amjindal]

Hello Guys!
@ozgurkan @amjindal @stepancheg @likethesky @yfeldblum @ppwwyyxx Still same error at colab. You have any process how to figure it out ?

For now, I found a manual solution. I downloaded the model from the link and saved it to my drive profile. Then I copied the model from the drive profile to the output file and made changes to the code below.

cfg.MODEL.WEIGHTS = "/content/output/model_final_68b088.pkl"

Maybe there is a better solution but I don't know. I think there is a problem in accessing the model file in the link.

i don't think the model final weights file is missing. from the error message it doesn't seems lile.
As i said, the code is working fine on google colab from the starting.