Skip to content
This repository was archived by the owner on Feb 22, 2023. It is now read-only.

[camera] Android: Improve image streaming by creating a request suita… #2239

Merged
mklim merged 3 commits into from
Oct 29, 2019
Merged

[camera] Android: Improve image streaming by creating a request suita… #2239

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1f4bd87
[camera] Android: Improve image streaming by creating a request suita…
PerLycke Oct 27, 2019
a5a5552
[camera] Add Android image streaming to e2e tests
PerLycke Oct 29, 2019
ece9ec0
Merge branch 'master' into streaming_fix
Oct 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions packages/camera/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
## 0.5.6+4

* Android: Use CameraDevice.TEMPLATE_RECORD to improve image streaming.

## 0.5.6+3

* Remove AndroidX warning.
Expand Down
2 changes: 1 addition & 1 deletion packages/camera/android/src/main/java/io/flutter/plugins/camera/Camera.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,7 @@ public void startPreview() throws CameraAccessException {

public void startPreviewWithImageStream(EventChannel imageStreamChannel)
throws CameraAccessException {
createCaptureSession(CameraDevice.TEMPLATE_STILL_CAPTURE, imageStreamReader.getSurface());
createCaptureSession(CameraDevice.TEMPLATE_RECORD, imageStreamReader.getSurface());
Copy link
Contributor

@mklim mklim Oct 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think maybe it would make even more sense to use TEMPLATE_PREVIEW? That setting is optimized for a high FPS without much post processing, which I think makes sense here. RECORD is very similar but it has a "stable" FPS with more post processing.

Copy link
Contributor Author

@PerLycke PerLycke Oct 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested with both but using TEMPLATE_PREVIEW breaks trying to detect objects in a CameraImage using the ml kit on my test devices. With TEMPLATE_RECORD it both fixes the streaming issues and keeps the functionality intact. @mehmetf might have more insight in this as I know he looked in to improving streaming.

Copy link
Contributor

@mehmetf mehmetf Oct 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pinging me here. We forked this plugin for Google and have made a number of improvements. Glad to see similar efforts upstream:

  • TEMPLATE_RECORD is indeed what we use now for preview.
  • We also start a different thread for image stream handler (that we dispose when image stream is stopped).
  • We drain the image queue while waiting for processing to finish. This allows preview to be smooth even if the stream does not deliver high fps (which is often unneeded).

Our code looks a bit like this:

public void startPreviewWithImageStream(
      ImageReader imageStreamReader, StreamProcessor<Image> processor)
      throws CameraAccessException {
    imageStreamHandler = CameraUtils.startThread("imageStreamThread");
    createCaptureSession(CameraDevice.TEMPLATE_RECORD, imageStreamReader.getSurface());
    imageStreamReader.setOnImageAvailableListener(getImageStreamListener(processor), imageStreamHandler);
  }

  public void stopPreviewWithImageStream(ImageReader imageStreamReader)
      throws CameraAccessException {
    CameraUtils.stopThread(imageStreamHandler);
    imageStreamReader.setOnImageAvailableListener(null, null);
    startPreview();
  }

  private ImageReader.OnImageAvailableListener getImageStreamListener(
      StreamProcessor<Image> processor) {
    return reader -> {
      Image img = reader.acquireLatestImage();
      if (img == null) {
        return;
      }
      if (openImage.getAndSet(true)) {
        img.close();
        return;
      }
      imageStreamHandler.post(
          (Runnable)
              () -> {
                try {
                  Uninterruptibles.getUninterruptibly(processor.apply(img));
                } catch (ExecutionException e) {
                  Log.e(TAG, "Exception from image processing", e);
                } finally {
                  openImage.set(false);
                  img.close();
                }
              });
    };
  }

Copy link
Contributor Author

@PerLycke PerLycke Oct 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Mehmet, I hope some of those improvements can find its way to this plugin later. As of now, and regarding this PR, changing TEMPLATE_STILL_CAPTURE to TEMPLATE_RECORD singlehandedly fixes streaming on several devices. There's not much documentation around this in particular, but seems needed


imageStreamChannel.setStreamHandler(
new EventChannel.StreamHandler() {
Expand Down
35 changes: 35 additions & 0 deletions packages/camera/example/test_driver/camera_e2e.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,4 +200,39 @@ void main() {

expect(duration, lessThan(recordingTime - timePaused));
}, skip: !Platform.isAndroid);

testWidgets(
'Android image streaming',
(WidgetTester tester) async {
final List<CameraDescription> cameras = await availableCameras();
if (cameras.isEmpty) {
return;
}

final CameraController controller = CameraController(
cameras[0],
ResolutionPreset.low,
enableAudio: false,
);

await controller.initialize();
bool _isDetecting = false;

await controller.startImageStream((CameraImage image) {
if (_isDetecting) return;

_isDetecting = true;

expectLater(image, isNotNull).whenComplete(() => _isDetecting = false);
});

expect(controller.value.isStreamingImages, true);

sleep(const Duration(milliseconds: 500));

await controller.stopImageStream();
controller.dispose();
},
skip: !Platform.isAndroid,
);
}