Allow hard negatives in keypoint ROI flow for generalized RCNN

Summary:
When training ROI heads with hard negative samples (images with no associated annotations), their Instances have no field "gt_keypoints". This breaks the filtering by visible keypoints in `select_proposals_with_visible_keypoints`, and if the entire minibatch consists *only* of hard negatives, this breaks ROI pooling (`self.keypoint_pooler`) as well.

Desired behavior: when hard negatives are fed into the keypoint detection pipeline, we want error signals for the RCNN bbox classification branch (and potentially RPN branch, TODO) to learn there are no object ROIs present. There should be no signal from the bbox loc/regression and keypoint ROI head branches, as loss is undefined w/o gt boxes/keypoint annotations.

Fortunately it seems this functionality only requires minimal changes:

1) in `roi_heads.py:select_proposals_with_visible_keypoints:95`, skip over empty proposals/Instances (i.e. the hard negatives in a minibatch); this does not affect existing behavior because the keypoint rcnn loss computation filters out empty Instances using the same pattern in `keypoint_head.py:50`, excluding them from the loss downstream, regardless.

2) An additional safeguard: If a minibatch ever consists entirely of hard negatives, then the change in 1) will leave us with an empty list of proposals after the filtering step in `_forward_keypoint()`, which breaks ROI pooling. To address this, perform filtering only if ROI proposals w/ nonempty gt >= 1.

Reviewed By: ppwwyyxx

Differential Revision: D18068957

fbshipit-source-id: 09e1e989350a3a4b46d2333923a208f378abffbb

Author: Owen Wang

detectron2/modeling/poolers.py

@@ -171,6 +171,15 @@ def forward(self, x, box_lists):
                 boxes aggregated over all N batch images and C is the number of channels in `x`.
         """
         num_level_assignments = len(self.level_poolers)
+
+        assert len(x) == num_level_assignments, \
+            "unequal value, num_level_assignments={}, but x is list of {} Tensors".format(
+                num_level_assignments, len(x))
+
+        assert len(box_lists) == x[0].size(0), \
+            "unequal value, x[0] batch dim 0 is {}, but box_list has length {}".format(
+                x[0].size(0), len(box_lists))
+
         pooler_fmt_boxes = convert_boxes_to_pooler_format(box_lists)
 
         if num_level_assignments == 1:

detectron2/modeling/roi_heads/roi_heads.py

@@ -92,6 +92,10 @@ def select_proposals_with_visible_keypoints(proposals):
     ret = []
     all_num_fg = []
     for proposals_per_image in proposals:
+        # If empty/unannotated image (hard negatives), skip filtering for train
+        if len(proposals_per_image) == 0:
+            ret.append(proposals_per_image)
+            continue
         gt_keypoints = proposals_per_image.gt_keypoints.tensor
         # #fg x K x 3
         vis_mask = gt_keypoints[:, :, 2] >= 1