About nDisplay

The Unreal Engine supports advanced Igloo structures through a system called nDisplay. This system addresses some of the most important challenges in rendering 3D content simultaneously to multiple displays:

• It eases the process of deploying and launching multiple instances of your Project across different computers in the network, each rendering to one or more display devices.

• It manages all the calculations involved in computing the viewing frustum for each screen at every frame, based on the spatial layout of your display hardware.

• It ensures that the content being shown on the various screens remains exactly in sync, with deterministic content across all instances of the Engine.

• It offers passive and active stereoscopic rendering.

• It can be driven by input from VR tracking systems, so that the viewpoint in the displays accurately follows the point of view of a moving viewer in real life.

• It is flexible enough to support any number of screens in any relative orientation, and can be easily reused across any number of Projects.

Every nDisplay setup has a single master computer, and any number of additional computers.

• Each computer in the network runs one or more instances of your Project's packaged executable file.

• Each Unreal Engine instance handles rendering to one or more display devices, such as screens or projectors.

• For each of the devices an instance of Unreal Engine handles, it renders a single viewpoint on the same 3D scene. By setting up these viewpoints so that their location in the 3D world matches the physical locations of the screens or projected surfaces in the real world, you give viewers the illusion of being present in the virtual world.

• The master node is also responsible for accepting input from spatial trackers and controllers through connections to Virtual-Reality Peripheral Networks (VRPNs), and replicating that input to all other connected computers.

The image above shows a possible nDisplay network. Like all nDisplay networks, one of its PCs acts as the master node. This master node accepts input into the system from a VRPN server, which relays signals that come from spatial tracking devices and other controller devices. The network also contains several other PCs that run other instances of the Unreal Engine Project. Each of these cluster nodes drives one or more display projectors.

Setting up nDisplay within Unreal

Prerequisites

• Unreal Version 22 or newer.

• A computer with more than 1 screen, in an extended configuration.

• The computer should be capable of running Unreal Engine, based on these recommended specifications.

A New Project

Epic games have kindly given us a template that sets everything up for you. If this is your first time using nDisplay, Igloo highly recommends creating a new project using this template, and experimenting with the config file.

To create a new project, on the Epic Launcher, click Unreal Engine on the left toolbar, then Library on the Top toolbar. Ensure that you have a version of Unreal installed. I would recommend the most recent version if you’re starting from scratch.

Click Next, and on the Select Template menu, click nDisplay and once again click Next.

Finally, select a location for your new project, and wether you would like to include Starter Content, and enable Raytracing. I would recommend disabling Raytracing for your first project unless you’re familiar with it.

Upon selection of Create Project the editor will begin to unpack and create the project at the location you have specified. This might take a long time depending on your computers specifications.

Existing Project

1. Open your project, and navigate to the Plugin’s window. Search for nDisplay and Enable the nDisplay plugin. This plugin is included in Unreal 2.46 and newer.
   

   

2. Once the editor has re-opened, remove all players + cameras from the scene. The nDisplay system creates it’s own player and camera. This can be edited later using the blueprints and config files.

3. Add nDisplayClusterRootActor, from the Place Actors menu, into the scene where you would like the player to start. Place it on the floor, as it will be generated from the floor based on height values.

Building a project

[How to build the project, and where to place it once built. Adding the listener.exe, and where to find that. Placing the built version within a network shared location./]

When you are ready to build your project, it is done in the usual way by clicking File, Package Project, and Windows (64 bit).

You will then be asked to specify a folder for your completed build to be placed. At this point we recommend a folder that is easily shared. For instance I’m using:

D:/Unreal/nDisplayTest

Once the build is complete it is also a good idea to include, within the build folder, the nDisplay listener. This can be found in your Unreal Engine Binaries folder at the following path:

Program Files\Epic Games\UE_4.XX\Engine\Binaries\DotNET\

Copy the DisplayListener.exe from this location, and place it in the same location as your built application’s executable file (.exe)

Creating the config file

A config file is required to inform each of the machines within the nDisplay cluster, on how to create the various nDisplay components. Luckily Epic has given us a lot of examples of how this configuration file should look. They can be found at this location within your engine folder:

\Program Files\Epic Games\UE_4.XX\Templates\TP_nDisplayBP\Content\ExampleConfigs

Please take some time to view how each of them is constructed, and experiment with modifying them.

To preview how it will look, within the unreal editor, you can load a configuration within the properties of the DisplayClusterRootActor and it will load the asset structure created by the config file. This is a great tool for finding errors.

Config file with Scalable

Within Igloo structures, instead of creating the setup using the config file, we can create a barebones file, and instead use Scalable to create the screens, blends, and systems.

Luckily, Epic have done the work for us and created an Easyblend base config file, which can be found within the EasyBlend folder of the ExampleConfigs.

This configuration just handles the player, and the base locations for the camera system. Then using the scalable editor we can generate the required files for the warping and blending.

For instructions of editing scalable configuration file, it can be found further into this guide.

Using Scalable to generate warp and blend files

Setting up Scalable for the first time

Once you’ve registered for scalable, purchased a license, or signed up for the free trial license, you will be given a download link for Scalable’s software; Display Manager.

You will also require a suitable camera, to create the blends, the list of supported cameras is HERE

You will need to install the Display manager on every machine, and have it launch on startup. It can then be configured from any machine as a single entity. However, it’s best to use the machine you have the cameras plugged into, unless they are gigE cameras (i.e. Basler)

Once you load up the client for the first time, it will ask you for your license file. This may have come with your download, or it may have been emailed to you.

You will be greeted with a wizard type system, with 4 pages. The first being:

Display Clients

The Display Clients panel presents an interface that allows you to setup the connections to the remote computers.  The right side of the panel shows the displays connected to the current system, while the left side shows all available Display Clients.

Use the Display Client on the Local Computer

If the computer running  Scalable Display Manager is the only computer used in the system, select the radio button Use Local Display Clients Only.  No additional setup is required in the Display Clients panel.

Display Clients on Remote Computers

If your Scalable Display Manager configuration will require connecting to one or more remote computers, select the radio button for Use Remote Display Clients.

Set Up Remote Computers

Add a Display Client to the System

Scalable Display Manager automatically detects all Display Clients on the network subnet.  The status of each Display Client is displayed in a colored square to the left of its network identifier.

Display Clients shown in the list on the right will be part of the system.

1. Click to highlight one of the clients in your system.

2. Click Assign > to move the clients into the system.

3. Repeat for all remaining clients.

Scalable Display Manager will by default output a scalable mesh file, if instead you would like to directly apply the warp and blend in the graphics driver check the option Apply Warp & Blend in Graphics Driver.

Warping and Blending in the graphics card requires Nvidia Mosaic Mode.  Please check the website for the latest recommended version of the Nvidia Quadro driver.

Projectors Panel

The Projectors panel should accurately reflect the number and resolution of the displays connected to your computer, however Scalable Display Manager can only auto-detect certain common resolutions. If it is unable to detect the current resolution of your display(s), the most common response is that it will simply show a single display at the combined resolution of all your projectors.

To help with making sure that the projectors reach about an overlap value of 15-20% you can click Show Overlap Pattern. There are 3 sections Yellow,Green, and White. Having the edges of the bands touch will result in the level of overlap it represents.

• Yellow is 15%

• Green is 20%

• White is 25%

Here is an example of a 20% overlap setup:

Enter the Projector Arrangement

1. Select the physical arrangement of the projectors: Tiled, if the projectors are next to each other, or Stacked, if the projectors are on top of each other.

2. Select the number of projectors in a row.

3. Select the number of projectors in a column.

4. Click Redetect Displays to save the settings.

Order the Display Clients

The order of the display clients will have a direct effect on the projector numbering. To properly order projectors, set the displays such that the left-most projectors on the blended display appear first in the list. To change the order use the arrow key to move the computer IP up or down.

Cameras Panel

Basic Camera Configuration

Verify that the number and type of cameras detected by Scalable Display Manager correspond to your system. If the Automatic type doesn't detect the camera configuration properly, please choose your camera type manually.

Advanced Camera Configuration

The Advanced Camera Configuration allows the user to grab a subset of the total cameras and arrange the order based on serial numbers.

Data Collection

Adjust the Camera Settings

For a proper calibration, it is necessary to focus the camera and adjust its exposure settings.  If the camera captures an image that is too bright it will have difficulty detecting the calibration spot patterns.  Similarly, not focusing the camera will reduce the accuracy of the camera detection and may cause artifacts in the resulting warp and blend.  It is thus important to make sure the camera(s) can see the entire screen and are capturing well-focused and properly exposed images.

Camera Brightness

Scalable Display Manager requires the image to appear with normal saturation in order to properly detect the patterns displayed during calibration. If the image presented to Scalable Display Manager is over or undersaturated, it will result in an error or produce an incorrect geometry calibration. The camera's brightness needs to be set so that there is enough contrast between the light and dark areas of the screen to see the calibration patterns. The image preview window should look similar to how you see the image in real life.

If after manual adjustment the image brightness still has saturation problems, click the Auto Tune button. A series of pictures will be taken to auto-adjust the camera.  If the image is not normally saturated after the Auto Tune, you may need to manually adjust the camera's brightness. Follow the instructions below for your particular camera.

Begin Data Collection

Click Begin Data Collection.

The data collection process will begin by showing a solid, white image on each projector, starting with the first projector and going in order to the last. The white image is used to find the location of the projector in the camera image.  Make sure that the area of the screen that is intended to be the target area of the screen is completely covered by one or all of the projectors.

Next, two images will be displayed on each projector, starting again with the first projector and going in order to the last.  The first image will be a 5-dot pattern which is used to determine the center of the projected image. The second image displayed is the grid pattern which is used to map where the pixels are falling on the screen.

White Pattern	5 Dot Pattern	Grid Pattern

What to do if an Error Occurs

Most errors encountered during calibration are caused by poor camera positioning or improper saturation of the camera image, resulting in the inability of the software to detect the calibration patterns being displayed. When an error occurs, click the link labeled Click here to troubleshoot this error in the error message window. This will open a browser with the suggested solutions for this particular issue.

Data Collection Error Actions

Some of the most common errors encountered during the data collection process have been included in the Error Actions. Error actions allow you to ignore issues with the data collection images which do not inhibit the data collection.

The error action below will be displayed when a camera cannot fully see a projected image. You are given the choice to "End Calibration" or "Ignore and Continue". If you are expecting that the entire projected image will not be seen, Ignore and Continue. However, if you are not seeing the entire projected image for other reasons, you should End Calibration and correct the error at its source.

The software can not discern the difference between a projected image that does not fill the screen and one that fills the screen but spills off. 

End Calibration: In this case, the calibration should be stopped so that the camera can be re-positioned to see the entire projected image.
Ignore and Continue: The error action is expected and triggered because the projector is overshooting the screen. Continuing will not cause any issues.

Projector Visibility

Be careful when editing

Removing a projector from the visibility list for a camera can result in unwanted warp and blends and often cause the system stop calibrating

Exporting Data to Unreal

Once your screen has been calibrated using the Scalable system, the master machine will have saved the DataSet to it’s C:/ drive, in the following folder:

C:/ScalableDisplay/DataSets/DatasetXXXXXXXXXX/LastCalibration

In this folder, you will find a myriad of files, most of which are required for the nDisplay system to work.
Unfortunately, due to how these files are created, it’s difficult to setup this location as a shared space, so you will need to copy these files *(or rather the LastCalibration Folder as a whole) to the same location as your nDisplay config file.

Not all the files are required, but it’s easier to copy the entire thing.

Creating Scalable nDisplay config file

This can be created from scratch, or by copying an nDisplay config file, our our sample file below.

#####################################################################
# Igloo nDisplay Example Configuarion
#
# Note:
# Before use, make sure all settings correspond to your system.
#####################################################################

# Config file header.
[info] version="23"

# Cluster nodes
[cluster_node] id="IMP1" addr="192.168.0.101" window="window1" master="true"
[cluster_node] id="IMP2" addr="192.168.0.102" window="window2"
[cluster_node] id="IMP3" addr="192.168.0.103" window="window3"

# Application windows
[window] id=window1 fullscreen=false WinX=0 WinY=0 ResX=1920 ResY=1080 viewports="viewport1"
[window] id=window2 fullscreen=false WinX=0 WinY=0 ResX=1920 ResY=1080 viewports="viewport2"
[window] id=window3 fullscreen=false WinX=0 WinY=0 ResX=1920 ResY=1080 viewports="viewport3"

# Projection policies
[projection] id=projector1 type="easyblend" file="LastCalibration\ScalableDataOrthographic.pol"   origin=easyblend_origin scale=1
[projection] id=projector2 type="easyblend" file="LastCalibration\ScalableDataOrthographic.pol_1" origin=easyblend_origin scale=1
[projection] id=projector3 type="easyblend" file="LastCalibration\ScalableDataOrthographic.pol_2" origin=easyblend_origin scale=1

# Viewports
[viewport] id="viewport1" x=0 y=0 width=1920 height=1080 projection=projector1
[viewport] id="viewport2" x=0 y=0 width=1920 height=1080 projection=projector2
[viewport] id="viewport3" x=0 y=0 width=1920 height=1080 projection=projector3

# Cameras
[camera] id=camera_static loc="X=0,Y=0,Z=0"

# Scene nodes
[scene_node] id=cave_origin       loc="X=0,Y=0,Z=0" rot="P=0,Y=0,R=0"
[scene_node] id=wand              loc="X=0,Y=0,Z=1" rot="P=0,Y=0,R=0"
[scene_node] id=easyblend_origin  loc="X=0,Y=0,Z=0" rot="P=0,Y=0,R=0"

# General settings
[general] swap_sync_policy=1

# Network settings
[network] cln_conn_tries_amount=10 cln_conn_retry_delay=1000 game_start_timeout=30000 barrier_wait_timeout=5000

# Custom arguments
[custom] SampleArg1=SampleVal1 SampleArg2=SampleVal2

Scalable Config Explained

Header Info

[info] version=”23”

This is the config file header, it specifies the version of Unreal that it supports. It’s best to leave this at default “23” otherwise it causes issues.

Cluster Nodes

[cluster_node] id="node_main" addr="10.1.5.110" window="wnd_all" master="true"

A cluster node is a physical machine, capable of running the Unreal project. Each cluster will need a unique ID (you can make this up), a local IP address, and then the window it should display.

Additionally you can also specify:
- which node is the master (there can only be one)
- which node has the sound output (master by default)
- The various ports used for synchronization between the machines (ports 41001 → 41003 are used by default)

Windows

[window] id=wnd_all fullscreen=false WinX=0 WinY=0 ResX=1920 ResY=1080 viewports="vp_1"

This defines the application window for the game. This must cover the entire projection space you are going to use. If, for instance, you have 3 1080p projectors connected to a single machine, the ResX value should be 3x1920 (5760) to span all 3 outputs.

If the machine has a console monitor (a screen where no projection will take place), ensure that it is the first in the screen ordering, and then use the WinX value to adjust the start position of the game canvas.

Viewport, specifies which viewport (explained below) should be rendered to this window. If multiple viewports are required a comma can be used to separate the viewport IDs e.g. “vp_1, vp_2”

The ID should be both Unique, and match the ‘window’ field on the cluster node intended to display this.

Additionally, you can also specify ‘Fullscreen’ which is useful if you have a single display output (or a mozaic/eyefinity output) which will increase game rendering capabilities.

Viewport

[viewport] id=vp_1 x=0 y=0 width=3840 height=1080 projection=proj_easyblend_1

The viewport is an area of the game window where a frame is being rendered. Usually it’s the same size as the entire window, however multiple viewports is supported for multi-window setups like LED walls.

It’s X and Y values donate the its position on the Window, it is not related to the position on the desktop. So if you move the window over 1 1080p screen to allow for a console screen, you do not need to add 1920 to the X value of the viewport.

The projection value, explained below, is which projection policy will be used

Projection

[projection] id=proj_easyblend_1 type="easyblend" file="LocalCalibrationWhiteboard\ScalableDataOrthographic.pol" origin=easyblend_origin_1 scale=0.1

This is where the scalable setup differs from a standard nDisplay system. Instead of a ‘simple’ projection system, which specifies the cameras created by unreal, we use Scalable & Unreal’s easyblend system. This tells unreal how to create cameras based on the .pol data created by Scalable during setup.

Like the other values, it requires a unique ID, but additionally you have a file field, an origin, and a scale value.

The field value should be the location of your ScalabaleData.pol file relative to the location of this file. It’s also possible to put a complete file address, but this may cause issues as it will likely be stored on a shared drive.

Camera

[camera] id=camera_static loc="X=0,Y=0,Z=0"

This is the camera used by the master node to create the player, it has many optional properties that relate to 3D, tracking, and hierarchy.

parent - ID of the parent component, default is VR root
tracker_id - the ID of the tracking device, default there is no tracking.
tracker_ch - the ID of the tracking device’s channel
eye_swap - swap eyes if in stereo mode; default is false.
eye_dist - distance in meters between the eyes, default is 0.064.
force_offset - force’s a mono camera to behave like a stereo camera, eye_offset works for this behavior too.

Scene Nodes

[scene_node] id=cave_origin  loc="X=0,Y=0,Z=0"   rot="P=0,Y=0,R=0"
[scene_node] id=easyblend_origin_1 loc="X=0,Y=0,Z=0" rot="P=0,Y=0,R=0"

These are the objects created in the game world, that make the framework for the player. The first object should always be labeled ‘cave_origin’ as this matches the object placed within the Unreal Build. Everything will parent to this object by default. There is no need to specify a parent unless you require a different structure.

With simple camera systems, a projector will be paired to a scene object, which would require you to position, or offset the scene object to place the projector correctly. With scalable displays this is not the case, you only need to create as many scene objects for your projectors, the rest is handled by scalable.

Other settings

The rest of the settings are explained in more detail within the example files. 99% of the time the defaults are perfect.

Creating a cluster

[Multiple machine setup - what needs to be included on each machine]

[Using the nDisplay launcher]

[Launching the project for the first time (usual errors the pop up)]