Prerequisites
We developed the application for the HoloLens and OptiTrack platforms. As a first step, you should meet the minimum requirements for Hololens applications. The system should have at least the following capabilities to build and deploy the Hololens application either in Hololens Emulator or in the Hololens real device:
- 64-bit Windows 10 Pro, Enterprise, or Education (The Home edition does not support Hyper-V)
- 64-bit CPU
- 8 GB of RAM or more
- In the BIOS, the following features must be supported and enabled:
- Hardware-assisted virtualization
- Second Level Address Translation (SLAT)
- Hardware-based Data Execution Prevention (DEP)
- GPU (The emulator might work with an unsupported GPU, but will be significantly slower)
- DirectX 11.0 or later
- WDDM 1.2 driver or later
OptiTrack Motive
We used the OptiTrack cameras to apture the motion of multiple rigidbodies placed in different locations in the classroom. The positions and movements of these rigidbodies are tracked to drive the augmentation of virtual objects.
We calibrated 4 rigidbodies in the classroom as shown in the figures below. Figure 4 shows the physical rigidbodies in the classroom. Figure 5 shows their corresponding calibrated virtual rigidbodies. We load the Motive project with all of the rigid bodies and began broadcasting tracking data using Data Streaming mode in Motive application.
We calibrated 4 rigidbodies in the classroom as shown in the figures below. Figure 4 shows the physical rigidbodies in the classroom. Figure 5 shows their corresponding calibrated virtual rigidbodies. We load the Motive project with all of the rigid bodies and began broadcasting tracking data using Data Streaming mode in Motive application.
Limitations of the current OptiTrack setup
In the current setup of the OptiTrack system, we only have three installed cameras at one side of the classroom. The three cameras track a small area in the classroom from one direction. During the rigid body calibration process, all the three cameras capture the rigid body markers and try to register that rigid body with a unique ID. During the running of the application, all markers of a specific rigid body must be visible so the tracking of that rigid body won't be lost.
In ideal situation, cameras should track the area from all angles so the markers of a rigid body still visible from all directions when the rigid body moves rapidly. That is, when a rigid body moves, each camera observes a set of markers from its angle and all cameras accumulate the visible markers to identify a rigid body and keep a tracking of it. I our case, that was a limitation to keep the rigidbody's markers visible to the cameras during the movement. In addition, we had to keep the rigidbody within an area of 6 by 3 ft2 which represents the intersection of the views of the three cameras.
Universal Windows Platform (UWP)
1- Unity Technical Preview
We built a Universal Windows Platform (UWP) application for the Hololens. We used Unity3D to develop our application contents. We used the Unity Technical Preview version because we are targeting Hololens headset. Mixed Reality Application. to export as UWP application, different settings are needed to be adjust in the Unity project to add mixed reality capabilities and inputs. They are broken down into two categories: per-project and per-scene settings. Here are the details on setting up these capabilities.
You can alos read here and here about Hololens system requirements and installation for more details.
2- Deploying to UWP environment
Unity supports a variety of deployment targets. However, Hololens is only compatible with the Windows Store deployment platform.
We built a Universal Windows Platform (UWP) application for the Hololens. We used Unity3D to develop our application contents. We used the Unity Technical Preview version because we are targeting Hololens headset. Mixed Reality Application. to export as UWP application, different settings are needed to be adjust in the Unity project to add mixed reality capabilities and inputs. They are broken down into two categories: per-project and per-scene settings. Here are the details on setting up these capabilities.
You can alos read here and here about Hololens system requirements and installation for more details.
2- Deploying to UWP environment
Unity supports a variety of deployment targets. However, Hololens is only compatible with the Windows Store deployment platform.
3- Run on Visual studio
When deploying the project to a UWP app in unity, it packages the project binaries and creates a project bundle to run in visual studio as UWP application. The project can be run virtually using Hololens emulator or the real device.
TCP/UDP Exchange Protocol
1- System.Net Client
We wrote a TCP/UDP exchange protocol to stream data from OptiTrack to Hololens. We wrote a client to ping the Omicron server and start listening for incoming packages of data. Omicron server, done by Arthur, exchanges data with OptiTrack and preprocesses" the tracking frames: receive packages, removes redundant data, converts coordinates from right-handed to left-handed, etc.
2- System.Net Server
In the server scenario, we wrote a server that receives packages from Omicron and forward the received packages to the Windows.Networking Client on Hololens. We followed this scenario to enforce Hololens to receive packages since data listener on Hololens always drops incoming packages.
3- Windows.Networking Client
UWP environment has its own networking system to send and receive data over TCP or UDP protocols. We wrote a UWP compatible client to receive the forwarded packages. Read more about the incompatibilities in our Hololens challenges page.
Data streaming executive summary
Our system logic was as follows:
- We load the Motive project with all of the rigid bodies and begin broadcasting tracking data using Data Streaming mode in Motive.
- We launch the Omicron server to exchange with Optitrack, preprocess data and send data as a list of bytes.
- The connection to the Hololens app, works as follows:
- On Unity Editor (System.Net Client/Server), we wrote a TCP client/server to work as a middleware between Omicron and Hololens:
- TCP socket is used to connect to Omicron server and send shake message
- Connection is made
- UDP socket is used to receive data on as bytes, then we read few bytes to get to the data that we need like position, orientation, and rigidbody ID.
- On UWP Editor (windows.Networking Client), we wrote a TCP server to read the data:
- StreamSocket is used to connect to System.Net server over TCP
- Connection is made
- StreamSocket socket is used to receive data on.