Mitchelville XR Tour

FAU Media, Technology, and Entertainment faculty and students have designed an immersive and interactive extended reality (XR) tour experience of Reconstruction-era (1861-1900) narratives for Historic Mitchelville Freedom Park on Hilton Head Island, South Carolina. We used historical photography and period maps to visualize the Mitchelville structures and Gullah Geechee residents in the Unity and Unreal game engines. This project is funded by a Walter and Lalita Janke Emerging Technologies Fund seed grant, an NEH Digital Projects for the Public Discovery grant, and an Epic Games Megagrant.

We finished the preproduction design phase over the 2019-2020 academic year. The main deliverable for the “Discovery” phase of the project was a design document for the NEH, which can be viewed here as a PDF… >Mitchelville XR Tour Design Doc 2020<

2019-2020 Video Demo of Virtual Preproduction and Headset Tests:

Spring 2020 we used historical photography taken by Henry P Moore in 1862 of Mitchelville inhabitants and houses to create a 3D simulation of the town in the Unity game engine:

Mitchelville 1862 3D Game Engine Simulation
Mitchelville game engine simulation of 1862 historical photograph (Library of Congress). 3D models by Ledis Molina and James Jean-Pierre, Spring 2020.

We have designed a multimodal production pipeline to distribute 3D scenes developed in a game engine (Unity and/or Unreal) to multiple XR platforms, including augmented reality (AR- below left) mobile devices and mixed reality (MR – below middle) headsets in the park, a downloadable app for virtual reality (VR below right) headsets, and a web site portal to a 360 interactive web experience (below next image).

Mitchelville AR-MR-VR Multi-modal Apps
Mitchelville AR-MR-VR Multi-modal Apps

The prototype will also include a web site that features portals to 360 web pages of our AR tour experiences, as well as additional educational content and mini-documentaries:

Mitchelville AR Tour Web Design 2020
Mitchelville AR Tour Web Design by Topher Maraffi (2020), 3D models by Ledis Molina and James Jean-Pierre, historical images courtesy of the Library of Congress.

Here is a 360 web browser test of the Harriet Tubman animation set at the Prayer House in Mitchelville (left mouse click and drag to look around the 360 rendered scene):

  • 360 Demo by FAU MFA student Ledis Molina. Music by Anita Singleton-Prather and the Gullah Kinfolk, recorded live at the 2018 Original Gullah Festival in Beaufort, SC.

Fall 2020 we started developing an AI 3D character design pipeline that uses Generative Adversarial Networks (GANs) and Neuro-evolutionary AI breeding tools to design realistic versions of historical characters, by inputting source photographs from the National Archives to learn features. Here are some examples of Harriet Tubman and Frederick Douglass that were done at Artbreeder.com:

Harriet Tubman Breeding Tree, Maraffi 2020
AI breeding tree that shows how target features in the image were created by analyzing historical photos, and then refined by manipulating genes, cross breeding lines, and selecting children that fit the desired Harriet aesthetic.
Harriet Tubman AI Examples, Maraffi 2020
We were able to create young, middle aged, and old versions of Harriet by manipulating age genes and cross breeding from different photos.
Frederick Douglass Breeding Tree, Maraffi 2020
We wanted to develop full color characters that would resemble the originals, and be suitable for using with other AI tools in our pipeline to generate rigged 3D character models.
Frederick Douglass AI Examples, Maraffi 2020
Because Frederick Douglass and Harriet Tubman are not in the BigGAN database the images generated from the process are more general than the original photos, but they can be rendered from multiple angles and still resemble the source.

Fall 2019 we started Mitchelville pre-production work in my Preproduction, Prototyping, and Previs (DIG 6546) MTEn grad course that is cross-listed with my undergrad Visual Design for Film, Animation, and Games (FIL 4703/DIG 4122C) course, with both grads and undergrads working together on visualizing several stories related to Mitchelville and Reconstruction in Beaufort County. Students created 3D models of historic personalities like Harriet Tubman, Robert Smalls, and Frederick Douglass, and contemporary Gullah-Geechee storytellers like Aunt Pearlie Sue as performed by Anita Singleton-Prather:

Harriet Tubman Combahee River Raids research poster by undergraduate Dawn Davidson:

Mitchelville Project Harriet River Raids Undergrad Poster by Dawn Davidson, Fall 2019

We started virtual pre-production to create 3D cinematics for the Harriet Tubman Combahee River Raids storyboard using our Rokoko motion capture suit, HTC Vive base stations and trackers, and Unreal game engine. Graduate student Ledis Molina performed Harriet, assisted by graduate students Alberto Alvarez, Brandon Martinez, Camilo Morales, and undergraduate students Dawn Davidson, Mike Gonzalez, Juan Aragon, Tifffany Fedak, Alexandra Lusaka, Brandon Cela, and Robert Barclay:

Here is a render test in Unreal game engine of some shots edited by undergraduate student Tiffany Fedak, 3D model by graduate student Ledis Molina, virtual production by MFA students Alberto Alvarez, Brandon Martinez, and Camilo Morales:

See more FAU Media, Technology, and Entertainment research projects at: FAU MTEn-XR Lab

Categories: Research, Uncategorized

tophermaraffi

I am an Assistant Professor of Realtime VFX and Virtual Production at NCSU, where I teach game design, 3D animation, and extended reality courses in the Art + Design Dept.