Media, Technology, and Entertainment Extended Reality (MTEn-XR) MFA Lab Courses & Projects at Florida Atlantic University
Extended Reality (XR) courses I teach at FAU utilize motion capture animation, game engines, and the latest VR-AR headsets, and include Preproduction, Prototyping and Previsualization (DIG6546), 3D Production for Interactivity (DIG6547), and Interactive Interface Design (DIG6605). My grad courses are often cross-listed with undergrad Film, Video, and New Media BA courses, such as Visual Design for Film, Animation, and Games (FIL4703), Advanced 3D Computer Animation (DIG3306C), and Special Topics: 3D Video Game Design (DIG4930). Our labs are equipped with Noitom Perception Neuron Pro mocap suits, Rokoko Virtual Production Studio, HTC Vive, Oculus, and Magic Leap headsets, and Unity, Unreal, Autodesk Maya, and Adobe Creative Suite software.
Projects, Courses, and Events
NEH Funded Historical Site AR Project: Mitchelville Stories of Emancipation & Freedom in Beaufort County SC
We received a National Endowment for the Humanities (NEH) Digital Projects for the Public “Discovery” grant, an Epic Games Megagrant, and a Walter & Lalita Janke Emerging Technologies Fund seed grant to begin design work on an augmented reality (AR) tour of Mitchelville, a historic site on Hilton Head Island SC which was the first Freedman’s town in the US during the Civil War, and a Gullah-Geechee heritage site today. We will be creating a 360 experience of the story of America’s first efforts at Civil Rights for African Americans during the Reconstruction period as it relates to the Port Royal Experiment, with life-sized historical figures like Harriet Tubman, who will be experienced on site with Magic Leap headsets and mobile phones.
We have also applied to an NEH Digital Projects for the Public grant to fund development, and are collaborating with The Mitchelville Preservation Project, Reconstruction Beaufort, Penn Center, Fort Lauderdale Museum of Discovery & Science, Daruma Tech, and researchers from University of South Carolina Beaufort, North Carolina State University, Coastal Carolina University, and Columbus State University. This project is supported by educational partnerships with Magic Leap and the FAU Center for Body, Mind, and Culture. Read the full proposal: NEH Mitchelville Grant Submission 2019 Maraffi.
Fall 2019 we started Mitchelville pre-production work in my Preproduction, Prototyping, and Previs (DIG 6546) MTEn grad course that is cross-listed with my undergrad Visual Design for Film, Animation, and Games (FIL 4703/DIG 4122C) course, with both grads and undergrads working together on visualizing several stories related to Mitchelville and Reconstruction in Beaufort County. Students created 3D models of historic personalities like Harriet Tubman, Robert Smalls, and Frederick Douglass, and contemporary Gullah-Geechee storytellers like Aunt Pearlie Sue as performed by Anita Singleton-Prather:
Harriet Tubman Combahee River Raids research poster by undergraduate Dawn Davidson:
We started virtual pre-production to create 3D cinematics for the Harriet Tubman Combahee River Raids storyboard using our Rokoko motion capture suit, HTC Vive base stations and trackers, and Unreal game engine. Graduate student Ledis Molina performed Harriet, assisted by graduate students Alberto Alvarez, Brandon Martinez, Camilo Morales, and undergraduate students Dawn Davidson, Mike Gonzalez, Juan Aragon, Tifffany Fedak, Alexandra Lusaka, Brandon Cela, and Robert Barclay:
Here is a render test in Unreal game engine of some shots edited by undergraduate student Tiffany Fedak, 3D model by graduate student Ledis Molina, virtual production by MFA students Alberto Alvarez, Brandon Martinez, and Camilo Morales:
Museum Research & Pedagogy to Enhance Exhibits With XR
We have submitted an IMLS grant to expand our summer App-titude collaboration with Fort Lauderdale Museum of Discovery and Science (MODS) to do year-round research and pedagogy on using XR to enhance museum exhibits. Over three weeks in the summer we taught App-titude high-school interns how to create AR enhanced research posters on STEM topics featured in MODS exhibits.
Fall 2019 graduate student Richie Christian started pre-production on a Prehistoric Florida AR project in my 3D Production for Interactivity (DIG 6547) MFA course. Here is a video of the initial design:
VR MFA Project: Homebound
MTen MFA students Alberto Alvarez and Brandon Martinez created the initial designs and prototype for their VR thesis project on an undocumented immigrant’s journey across the Southern US border in my spring 2019 Interactive Interface Design (DIG6605) course. Their research poster (click to see details):
Some image samples from their design document:
Spring 2019 “The experience will convey the hardship of a life changing journey for those that may never have this experience and promote a spirit of empathy and understanding towards our neighbors while exploring how these stories impact our lives in south Florida and create an open dialogue about the current state of immigration.”
Funded Interdisciplinary Driving Game Study
Spring 2019 We received a Dorothy F. Schmidt College of Arts and Letters seed grant to begin working on an interdisciplinary project to develop a driving game simulation using the Magic Leap One headset that will be used to inform autonomous vehicle design (in collaboration with Computer Engineering faculty Dani Raviv, Hari Kalva, and Aleks Stevanovic). Our game simulation will track what highly ranked drivers do with their eyes and bodies to successfully maneuver a vehicle in an urban environment. MTEn MFA students Alberto Alvarez and Brandon Martinez will be developing this project throughout the 2019-2020 academic year. Here is a video of a smaller VR car project they created for my spring 2019 Interactive Interface Design course:
Performatology Research Project: MRder Mystery Theatre
We are designing a spatial computing app for the Magic Leap One headset that is a Mixed Reality (MR) version of the classic murder mystery live improv-theatre game. A virtual body is found in the environment, with clues to the killer that can only be seen when wearing the headset. Animated avatar faces and costumes are superimposed on the players to hide the headsets and create a masked-theatre aesthetic of classic Clue-style characters reminiscent of commedia dell’arte archetypes. Suggested lines are generated in the HUD to drive the narrative of the play in both the game and theatrical sense. This project is part of our lab’s Performatology research, which applies performing arts and animation theory to interactive media.
Somaesthetics VR Locomotion Study
This project seeks to simulate mindfulness or meditative walking practices where practitioners gain heightened awareness of their body as they move through a natural landscape, by focusing their gaze on the area where their feet hit the ground in front of their body while walking. I am collaborating with FAU Philosophy Professor Dr. Richard Shusterman to apply his Somaesthetic concepts to human computer interaction (HCI) design of VR avatar feedback, so that players perceive more agency and presence as an avatar, and to relieve the problem of VR motion sickness.
Art Gallery Research Project: Climate Reality in South Florida
I have applied to the Knight Foundation Art Challenge 2019 for Miami area to create a virtual gallery experience that uses Magic Leap headsets to better understand the impacts of Climate Change in South Florida:
Description: “The Miami art gallery appears empty, with spotlights on blank walls and a sign reading “Climate Reality in SoFlo”. Three pedestals contain Magic Leap headsets. When wearing a headset, the walls reveal spherical images, virtual “tiny world” photographs of South Florida sites. You hear water sounds as you approach an image of downtown Miami, when it suddenly surrounds you as a 360-degree video. Water rises in the 3D environment until it reaches your waist, and as the vision fades, text informs you of projected sea level rise at this location. Along with other people wearing headsets, you notice virtual figures in the gallery. One of the shadowy figures reaches out towards you, and a climate related question appears. This art experience connects people to place and people to people through the issue of Climate Change using augmented reality headsets that communicate between a gallery in Miami and on FAU campus.”
FAU MTEn-XR Lab Grad Research Day Posters
MFA students in my Interactive Interface Design (DIG6605) course presented research posters at the 10th annual FAU Graduate and Professional Student Association (GPSA) Research Day. Pictured are Camilo Morales (above), clockwise: Brandon Martinez and Alberto Alvarez; Raven Doyle; Monisha Selvaraj; Alex Henry; Raven, Monisha, and John Horvath:
DIG3306C: Mocap Animation Reel
Spring 2019 My Advanced 3D Computer Animation (DIG3306C) students created animation sequences from motion capture data in online libraries and custom movements we captured with the Noitom Perception Neuron suit. Here is a selection of their work:
MTEn-XR Lab Open House
Spring 2019 We had an open house in our MTEn-XR lab for potential students where we demo’d our Magic Leap, Oculus Rift, HTC Vive, and Noitom Perception Neuron mocap technology:
DIG4930/DIG6547: 3D Game Design Projects
Fall 2018 Students in my Special Topics: 3D Video Game Design course used Unity game engine and Autodesk Maya to create seek-and-find 3D island adventure games. They applied the Mechanics, Dynamics, Aesthetics (MDA) game design framework to design the look and interaction, and then coded in C# to develop the operational logics. Here are some screen shots from some of their games:
Students also created research posters, and Marshall Fornataro-Minnig showed a demo of his game at FAU’s Undergraduate Research Day:
FIL4703/DIG6546: Visual Design & Pre-production Projects
Fall 2018 My Visual Design for Film, Animation, and Games undergrad course is cross-listed with my Preproduction, Prototyping and Previsualization grad course in the fall semester. In this course students go through the entire preproduction pipeline, including planning a media concept, designing aesthetics, painting concept art, drawing storyboards, and creating animatics. Here are samples of student concept art:
DIG4703 student Shannell Tebeau presented a research poster of her history inspired pre-production work at FAU’s Undergraduate Research Day (click poster for details):
Students use a tablet and stylus to digitally visualize their narrative by drawing a storyboard of their scenes: