How To Build The Hololens Spectator View Rig That Let’s You Film In Mixed Reality

Virtual reality currently has many variations in capturing footage happening in the virtual world. But there aren’t too many options for the augmented reality technology such as the Hololens. To fix this solution, Microsoft released a series of videos in helping you capture the mixed reality footage for people to see what it’s like to experience mixed reality.

In this article we’ll go over how you setup the Microsoft Spectator View Rig so that you can record the mixed reality experience for your audience.

Rig Assembly:

You’ll be requiring multiple components to setup this particular rig. And of course you’ll need the basics such as the Hololens, a PC, and a DSLR camera.

• Use a T7 screwdriver to remove the headband from the HoloLens. Once the screws are loose, poke them out with a paperclip from the other side.
• Remove the screw cap on the inside front of the HoloLens visor with a small flat head screwdriver.
• Use a T15 screwdriver to remove the small torx bolts from the HoloLens bracket to remove the U and Hook-shaped attachments.
• Place the HoloLens on the bracket, lining up the exposed hole on the inside of the visor with the extrusion on the front of the bracket. The HoloLens’ arms should be kept in place by the pins on the bottom of the bracket.
• Reattach the U and Hook-shaped attachments to secure the HoloLens to the bracket.
• Attach the hotshoe fastener to the hotshoe of your camera.
• Attach the mount adapter to the hotshoe fastener.
• Rotate the adapter so the narrow side is facing forward and parallel to the camera’s lens.
• Secure the adapter in place with a 1/4″ nut using the 7/16 nut driver.
• Position the bracket against the adapter so the front of the HoloLens’ visor is as close as possible to the front of the camera’s lens.
• Attach the bracket with 4 1/4″ nuts and bolts using the 7/16 nut driver.

PC Setup

• Install the software from the software components section.
• Add the capture card to an open PCIe slot on your motherboard.
• Plug an HDMI cable from your camera to the outer HDMI slot (HDMI-In) on the capture card.
• Plug an HDMI cable from the center HDMI slot (HDMI-Out) on the capture card to an optional preview monitor.

Camera Setup

• Change your camera to Video Mode so it outputs at the full 1920×1080 resolution rather than a cropped 3:4 photo resolution.
• Find your camera’s HDMI settings and enable Mirroringor Dual Monitor.
• Set the output resolution to 1080P.
• Turn off Live View On Screen Displayso any screen overlays do not appear in the composite feed.
• Turn on your camera’s Live View.
• If using the Canon SDKand would like to use a flash unit, disable Silent LV Shoot.
• Plug an HDMI cable from the camera to the outer HDMI slot (HDMI-In) on the capture card.

You’ll also be requiring some software in order to get this rig working.

Software components

1. Software downloaded from the GitHub project for spectator view.
2. Blackmagic Capture Card SDK.
   Search for Desktop Video Developer SDK in “Latest Downloads”.
3. Blackmagic Desktop Video Runtime.
   Search for Desktop Video Software Update in “Latest Downloads”.
   Ensure the version number matches the SDK version.
4. OpenCV 3.1 For calibration or video capture without the Blackmagic capture card.
5. Canon SDK (Optional).
   If you are using a Canon camera and have access to the Canon SDK, you can tether your camera to your PC to take higher resolution images.
6. Unity for your holographic app development.
   Supported version can be found in the OSS project.
7. Visual Studio 2015 with latest updates.

Calibration

After setting up your spectator view rig, you must calibrate in order to get the position and rotation offset of your camera to your HoloLens.

• Open the Calibration Visual Studio solution under Calibration\Calibration.sln.
• In this solution, you will find the file dependencies.props which creates macros for the inc locations of the 3rd party sources.
• Update this file with the location you installed OpenCV 3.1, the Blackmagic SDK, and the Canon SDK (if applicable)

• Print out the calibration pattern Calibration\CalibrationPatterns\2_66_grid_FULL.png on a flat, rigid surface.
• Plug your HoloLens into your PC over USB.
• Update the preprocessor definitions HOLOLENS_USER and HOLOLENS_PW in stdafx.h with your HoloLens’ device portal credentials.
• Attach your camera to your capture card over HDMI and turn it on.
• Run the Calibration solution.
• Move the checkerboard pattern around the view like this:

  

  Calibrating the spectator view rig
• A picture will automatically be taken when a checkerboard is in view. Look for the white light on the HoloLens’ visor before advancing to the next pose.
• When finished, press Enter with the Calibration app in focus to create a CalibrationData.txt file.
• This file will be saved to Documents\CalibrationFiles\CalibrationData.txt
• Inspect this file to see if your calibration data is accurate:
  • DSLR RMS should be close to 0.
  • HoloLens RMS should be close to 0.
  • Stereo RMS may be 20-50, this is acceptable since the field of view between the two cameras may be different.
  • Translation is the distance from the HoloLens’ camera to the attached camera’s lens. This is in meters.
  • Rotation should be close to identity.
  • DSLR_fov y value should be close to the vertical field of view expected from your lens’ focal length and any camera body crop factor.
• If any of the above values do not appear to make sense, recalibrate.
• Copy this file to the Assets directory in your Unity project.

Compositor

The compositor is a Unity extension that runs as a window in the Unity editor. To enable this, the Compositor Visual Studio solution first needs to be built.

• Open the Compositor Visual Studio solution under Compositor\Compositor.sln
• Update dependencies.props with the same criteria from the Calibration solution above. If you followed the calibration steps, this file will already have been updated.
• Build the entire solution as Release and the architecture that matches your Unity version’s architecture. If in doubt, build both x86 and x64.
• If you built the solution for x64, also build the SpatialPerceptionHelper project as x86 since it will run on the HoloLens.
• Close Unity if it is running your application. Unity needs to be relaunched if DLLs are changed at runtime.
• Run Compositor\CopyDLL.cmd to copy the DLLs built from this solution to your Unity project. This script will copy the DLLs to the included sample project. Once you have your own project set up, you can run CopyDLL with a command line argument pointing to your project’s Assets directory to copy there as well.
• Launch the sample Unity app.

Unity App

The compositor runs as a window in the Unity Editor. The included sample project has everything set up to work with spectator view once the compositor DLLs are copied.

Spectator view requires the application to be run as a shared experience. This means that any application state changes that happen on the HoloLens need to be networked to update the app running in Unity too.

If starting from a new Unity project, you will need to do some setup first:

• Copy Assets/Addons/HolographicCameraRig from the sample project to your project.
• Add the latest HoloToolkit to your project, including Sharing, csc.rsp, gmcs.rsp, and smcs.rsp.
• Add your CalibrationData.txt file to your Assets directory.

• Add the HolographicCameraRig\Prefabs\SpectatorViewManager prefab to your scene and fill in the fields:
  • HolographicCameraManager should be populated with the HolographicCameraManager prefab from the HolographicCameraRig prefab directory.
  • Anchor should be populated with the Anchor prefab from the HolographicCameraRig prefab directory.
  • Sharing should be populated with the Sharing prefab from the HoloToolkit.
  • Note: If any of these prefabs already exist in your project hierarchy, the existing prefabs will be used instead of these ones.
  • Spectator View IP should be the IP of your HoloLens attached to your spectator view rig.
  • Sharing Service IP should be the IP of the PC running the HoloToolkit SharingService.
  • Optional: If you have multiple spectator view rigs attached to multiple PC’s, Local Computer IP should be set with the PC the spectator view rig will communicate with.

• Start the HoloToolkit Sharing Service
• Build and deploy the app as a D3D UWP to the HoloLens attached to the spectator view rig.
• Deploy the app to any other HoloLens devices in the experience.
• Check the Run In Background checkbox under edit/project settings/player.

• Launch the compositor window under Spectator View/Compositor

• This window allows you to:
  • Start recording video
  • Take a picture
  • Change hologram opacity
  • Change the frame offset (which adjusts the color timestamp to account for capture card latency)
  • Open the directory the captures are saved to
  • Request spatial mapping data from the spectator view camera (if a SpatialMappingManager exists in your project)
  • Visualize the scene’s composite view as well as color, holograms, and alpha channel individually.

• Turn your camera on.
• Press play in Unity.
• When the camera is moved, holograms in Unity should be where they are in the real world relative to your camera color feed.

Microsoft also provided many use cases for this setup and rig which helps demonstrates the strengths of mixed reality.

This isn’t a cheap setup. The Hololens is currently meant for developers and creatives working in the mixed reality industry. Many companies like Microsoft are currently fascinated with AR technology for its components of immersive interaction between the digital and physical world. Tim Cook recently stated that there is a big future for augmented reality and hinted that the company is working on a solution of their own. These are the beginning stages to what the general population will adopt in the near future (meaning 5-10 years).