Mobile Streaming from Java

A Java app is not an obvious choice for a streaming media server, but for us it was the logical one. With our distributed architecture, we expected roughly constant scaling above the app level and each individual app would only need to support a small number of users.

We were designing a backend for mobile apps that already supported playback over HTTP byte range requests. The complicated parsing logic in a typical streaming server wasn't needed. We had already implemented access control in Java through a labeling mechanism. If a user and a data object are assigned the same label, that contact can retrieve the data object. We now only needed to support byte range requests within the same access control mechanism.

Setup

For our testing, we used the Nativescript Video Player (v4.2.1), which invokes native code in both iOS and Android. The code was run on an iPhone 8 Plus and a Samsung Galaxy S9 to playback of an MP4 video from start to finish. We observed the following request sequences. The Requested Range column is what Java app receives and the Data Sent column is the corresponding value extracted from the localhost_access log file.

iOS: user-agent: AppleCoreMedia/1.0.0.18D70 (iPhone; U; CPU OS 14_4_2 like Mac OS X; en_us)
Requested Range Data Sent
1 bytes=0-1 2B
2 bytes=0-1756815 245760B
3 bytes=1736704-1756815 20112B
4 bytes=1753088-1756815 3728B
5 bytes=1752886-1753087 202B
6 bytes=16186-1736703 1720518B
Android: user-agent: stagefright/1.2 (Linux;Android 10)
Requested Range Data Sent
1 entire file 720896B
2 bytes=1481503- 275313B
3 entire file 491520B
4 bytes=262144- 606208B
5 entire file 360448B
6 bytes=327680- 1429136B

Because the video file is processed in non-linear order, the media player can't efficiently playback the video with a single entire file request. Interestingly, both media players terminate their connection before all of the data is received. Further, they then request overlapping regions.

Implementation

The requested by range can be identified through the HttpServletRequest object. This object is added as an instance variable to our controller receiving the request.

private final HttpServletRequest request;

We need to support both range requests and entire file requests. Extracting the range when set is accomplished with the following code block.

Long min = (long)-1;
Long max = (long)-1;
Boolean rangeSet = false;
Enumeration<String> nameIt = request.getHeaderNames();
while (nameIt.hasMoreElements()) {
  String name = nameIt.nextElement();
  if(name.equals("range")) {
    Enumeration<String> valueIt = request.getHeaders(name);
    while (valueIt.hasMoreElements()) {
      String value = valueIt.nextElement();
      String[] range = value.split("=");
      String[] values = range[1].split("-");
      if(values.length > 0) {
        min = Long.parseLong(values[0]);
      }
      if(values.length > 1) {
        max = Long.parseLong(values[1]);
      }
      rangeSet = true;
    }
  }
}

After retrieving the requested range, we then need to populate the header response. A full file request expects an HTTP 200 response, while the byte range request expects an HTTP 206 response. Additionally, we needed to set the Content Length and Content Type headers according to the data retrieved. The code for this is given in the following code block.

if(rangeSet) {
  AssetData data = showService.getSubjectAsset(app.getAccount(), 
      subjectId, assetId, min, max);
  HttpHeaders responseHeaders = new HttpHeaders();
  responseHeaders.set("Accept-Ranges", "bytes");
  responseHeaders.set("Content-Range", "bytes " + 
      data.getBegin().toString() + "-" + 
      data.getEnd().toString() + "/" + 
      data.getSize().toString());
  Long len = (data.getEnd() + 1) - data.getBegin();
  return ResponseEntity.status(HttpStatus.PARTIAL_CONTENT).
      headers(responseHeaders)
      .contentType(MediaType.parseMediaType(data.getContentType()))
      .contentLength(len).body(data.getResource());
}
else {
  AssetData data = showService.getSubjectAsset(app.getAccount(), 
      subjectId, assetId);
  return ResponseEntity.status(HttpStatus.OK)
      .contentType(MediaType.parseMediaType(data.getContentType()))
      .contentLength(data.getSize()).body(data.getResource());
}

Results

Both media players could be optimized in terms of the data they request. They both have overlapping data requests and close the connection before all of the data is received. They however both performed well; we were able to playback content from both a Raspberry Pi server and an AWS server.

The content was generated using ffmpeg and the following switches:

-vf scale=1920:-2 -vcodec libx265 -crf 23 -preset veryfast -tag:v hvc1

Although this is dependent on our connection speed and the content, playback delay was generally less than one second, and we didn't observe any stuttering

References

We would love for anyone to test out our project; CoreDB is the media server, and Jupstream is the video streaming mobile app.

project webpage:
https://diatum.org

github repository:
https://github.com/diatum-org

coredb installer for ubuntu 18.04:
https://github.com/diatum-org/coredb/releases

installation instructions:
raspberry pi -  https://diatum.org/getinvolved/node-installation/coredb-setup-on-raspberry-pi-4/
aws  - https://diatum.org/getinvolved/node-installation/coredb-setup-on-aws/

dikota mobile app:
ios - https://apps.apple.com/us/app/dikota/id1526510086
android - https://play.google.com/store/apps/details?id=org.diatum.dikota

jupstream mobile app:
ios - https://apps.apple.com/us/app/jupstream/id1559285751
android - https://play.google.com/store/apps/details?id=org.diatum.jupstream

Roland Osborne

Roland Osborne

San Francisco
views