How to extract feature from any layer in a pre-trained detector?

[yeliudev]

Thanks for your great work! The performances of the detectors are amazing!

I have some questions on detectron2.

1. Now I have a list of bbox coordinates with respect to some images, how can I run a pre-trained detector (i.e. Faster-RCNN-R50-FPN) on these images, and crop the feature map (i.e. extract the 7x7 rois in the detector) according to the bbox coordinates? Is there an API in detectron2 to do the work?
2. How can I generate the proposals for Fast-RCNN, should I run the RPN directly to get the .pkl file?

Thank you again for your attention!

[ppwwyyxx]

UPDATE: see https://detectron2.readthedocs.io/tutorials/models.html#partially-execute-a-model

from detectron2.modeling import build_backbone
from detectron2.modeling.poolers import ROIPooler
backbone = build_backbone(cfg)
pooler = ROIPooler(...)
rois = pooler(backbone(NCHW_images), [kx4 tensors])

2. Running inference on a dataset with a RPN-only model will produce the proposals for the dataset in output/inference.
   An example command is

python tools/train_net.py --config-file configs/COCO-Detection/rpn_R_50_FPN_1x.yaml --eval-only

[yeliudev]

Thank you for your kind reply!

As for Q1, regarding that the coordinates of the bboxes (either abs or rel ones) might change due to the pre-processing of images (i.e. padding or scaling), should I modify the [kx4 tensors] in your code manually to meet these transforms?

[ppwwyyxx]

The boxes should be absolute coordinates in XYXY, on the scale of [0, H or W], where H or W matches the size of tensor you give to the backbone.

[yeliudev]

Thanks

[kachiO]

Hello,
I have a related question. After running inference with Detectron2 and I get the class predictions and bounding boxes, how do I get the features associated with each detected object/region? It seems redundant to have to run re-run the images via the backbone and ROIPool.

[ppwwyyxx]

@kachiO the densepose example does what you described:

detectron2/projects/DensePose/densepose/roi_head.py

Lines 84 to 85 in 8170509

	pred_boxes = [x.pred_boxes for x in instances]
	features_dp = self.densepose_pooler(features, pred_boxes)

@ppwwyyxx I am confused. I think this line

rois = pooler(backbone(NCHW_images), [kx4 tensors])

has already generated a features for the regions. Is that right?