The Center for Simulation and Synthetic Humans

at the University of Texas at Dallas

Our Team

Dr. Marjorie Zielke (seated second from right) and her team of researchers, artists, and developers at The Center for Simulation and Synthetic Humans at UTD. Walter, one of the lab's virtual humans, is seated to the left of Dr. Zielke.

Dr. Marjorie Zielke is the Director of the Center for Simulation and Synthetic Humans (The Center). Dr. Zielke is the principal investigator on several innovative simulation research projects in the military/law enforcement, health/medical, and education sectors. The Center is a cross-disciplinary team including full-time and student computer and behavioral scientists, human computer interaction experts, animators and modelers. The Center is comprised of full-time staff and students.

Dr. Marjorie Zielke Featured In Dallas Innovates Magazine

(12/22/20) Our Director, Dr. Marjorie Zielke, and her work involving Virtual Humans were featured in the December issue of Dallas Innovates magazine, alongside fifty other influential Dallas based Innovators. Article

The Center Wins 5G Tech Titans Challenge

(10/7/20) Dr. Marjorie Zielke, our director, has been selected as the winner of the Tech Titans 5G Grand Challenge. Dr. Zielke proposed the Emergent Virtual Teacher Platform (EVTP) which would use advanced sensors and Internet of Things (IoT) data transmitted over 5G networks to drive virtual teachers in virtual reality (VR). The Center team is now eligible for a $20,000 award from AT&T, Cisco, Nokia and Ericsson. The EVTP was selected by Tech Titans Judges entrepreneur Mark Cuban; Thaddeus Arroyo, CEO of AT&T Consumer and Charlotte Jones, Dallas Cowboys Chief Brand Officer. Article

NSF Grant Announcement

Community Integration Platform for Health Science Education of Social Emotional Intelligence through Collaborative Mixed Reality

(10/1/20) The Center announces its award of a $149,967 planning grant (NSF Grant No. 1952163) from the National Science Foundation (NSF). The Center will explore creating an educational platform using collaborative mixed reality (MR) and other advanced intelligent technologies such as high-speed networks to enhance technical education and social emotional learning (SEL) in Richardson Healthcare Careers Academy (RHCA) Students. We will examine the use of networked virtual experiences such as serious games featuring virtual humans to address gaps in SEL education. Our SEL efforts will focus on helping students develop the interpersonal and emotional skills needed to work in a clinical environment, as well as examine the impact equal peers and knowledgeable experts can have on a student.

The Center Selected as Finalist in Air Force (AFWERX) Challenge

(7/28/20) The Center for Simulation and Synthetic Humans was among 370 teams chosen out of 1500 applicants to participate in the AFWERX Base of the Future online conference. The Team collaborated with the Multi-scale Integrated Interactive Intelligent Sensing and Simulation lab (MINTS), led by UTD Physics Professor Dr. David J. Lary to present a solution in the Empowering Airman and Family Wellbeing Challenge Category.

NSF Grant Announcement

Exploring Social Learning in Collaborative Augmented Reality with Virtual Agents as Learning Companions

(4/27/20) The Center for Simulation and Synthetic Humans at the University of Texas at Dallas announces its award of a $750,000 grant (NSF Grant No.1917994) from the National Science Foundation (NSF). This grant will be supporting the Emotive Virtual Patient (EVP) project in its latest study titled: "Exploring Social Learning in Collaborative Augmented Reality with Virtual Pedagogical Agents as Learning Companions." With the aid of this grant, this study will explore the social impact of collaborative AR in social learning environments and help lay the groundwork for future learning. News Article

AMA Grant Announcement

(4/27/20) The Center for Simulation and Synthetic Humans at the University of Texas at Dallas (The Center) at UT Dallas, in collaboration with the Office of Medical Education at UT Southwestern Medical Center (UTSW) has received a grant from the American Medical Association (AMA). The Center has developed the Emotive Virtual Patient (EVP), which is designed to allow medical school students to practice interviewing with a virtual patient in an augmented reality experience using the Microsoft HoloLens. This grant will support The Center's ongoing EVP research by integrating the EVP into UTSW's Pre-Clerkship Objective Structured Clinical Skills (OSCEs) curriculum. The Center will integrate EVPs into OSCEs and measure impact on learning through: 1) student confidence measures through qualitative self-assessment; and 2) objective metrics such as the OSCE Skills Clinics scores.

Virtual Human Research

• Using Design-Based Research to Develop a Virtual Human Interface for Police Nystagmus Training

  Marjorie A. Zielke, Djakhangir Zakhidov, Joel Rizzo, Erik Defries, Laay Trivedi,Cecelia Marquart,Matthew Dusek,James Prochaska

• Exploring Medical Cyberlearning for Work at the Human/Technology Frontier with the Mixed-Reality Emotive Virtual Human System Platform

  Marjorie A. Zielke, Djakhangir Zakhidov, Gary M. Hardee, Jithin Pradeep, Leonard Evans, Zahra Lodhi, Kevin Zimmer, Eric Ward

• Developing Virtual Patients with AR/VR for a Natural User Interface in Medical Teaching

  Marjorie A. Zielke, Djakhangir Zakhidov, Gary Hardee, Leonard Evans, Sean Lenox, Nick Orr, Dylan Fino, Gautham Mathialagan

• Using Qualitative Data Analysis to Measure User Experience in a Serious Game for Premed Students

  Zielke, Marjorie A.; Zakhidov, Djakhangir; Jacob, Daniel; Lennox, Sean

• Game-Based Virtual Patients — Educational Opportunities and Design Challenges

  Zielke, Marjorie, Ph.D.; LeFlore, Judy, Ph.D.; Dufour, Frank, Ph.D.; Hardee, Gary

  Translating the proper use and settings of medical equipment into immersive gameplay that fulfills educational objectives presents multiple design challenges. This paper presents ten parameters for a model to create virtual medical equipment used in nursing education and other fields. Rationale behind virtual medical equipment simulation, as well as justification for the proposed model, based on lessons learned is included. The method for creating the model is explained by a two-part examination of the need for medical equipment simulation and the heuristics of the model itself. The model proposed is part of the design and research of an interactive course, NursingAP.com, a full online curriculum for graduate nursing students seeking a nurse practitioner degree in neonatal healthcare. The data collection methods for quantification of learning objectives through gameplay in the virtual ventilator are also discussed as a mechanism to improve design. Conclusions about the model and future enhancements for validation and investigation are detailed.




Game-Based Medical Simulations

• Beyond fun and games: Toward an adaptive and emergent learning platform for pre-med students with the UT TIME Portal

  Zielke, Marjorie A.; Zakhidov, Djakhangir; Jacob, Daniel; Hardee, Gary

• A Serious-Game Framework to Improve Physician/Nurse Communication

  Zielke, Marjorie; Houston, Susan; Mancini, Mary Elizabeth; Hardee, Gary; Cole, Louann; Zakhidov, Djakhangir; Fischer, Ute

• The promise and challenge of virtual gaming technologies for chronic pain: the case of graded exposure for low back pain.

  Trost Z; Zielke M; Guck A; Nowlin L; Zakhidov D; France CR; Keefe F

• A Model for Creating Simulated Medical Equipment in a Situational Gameplay Context

  Zielke, Marjorie, Ph.D.; LeFlore, Judy, Ph.D.; Broderick, Valarie F.; Ziegler, Ryan

  High-fidelity virtual patients afford unlimited opportunity for practice in a virtual clinical setting and present multiple design challenges for developers of game-based simulations. Virtual patients allow students the opportunity to experience varied medical conditions as well as interact with patients with wide-ranging socio-economic and cultural factors. Practice with virtual patients also can improve both the accuracy and speed of cognitive, behavioral and psychomotor tasks. Virtual patients can be designed and adapted to simulate high-risk, low-incidence variations of medical conditions as well as altered to reflect regional, social, economic, behavioral and cultural factors. As opposed to simulations that use physical clinical settings and high-fidelity simulators, virtual-patient simulations provide ubiquitous, asynchronous learning and practice opportunities. Game-based simulations can provide instantaneous assessment and debriefing, both individually and in group settings. For these reasons, there is a growing interest in virtual patients for a broad range of applications, including academic, professional and military.

• A Composite Adult Learning Model for Virtual World Residents with Disabilities: A Case Study of the Virtual Ability Second Life Island

  Zielke, Marjorie A.; Roome, Thomas C.; Krueger, Alice B.

  Many benefits are available to people with disabilities who wish to participate in a virtual world. These include self-efficacy and the ability to share in virtual world community support. Further, many disabled residents of virtual worlds can vicariously experience physical activities through their avatar such as dancing, walking, and running – actions sometimes not possible in real life. However, learning the technology in a virtual world can be daunting for many new residents. Virtual Ability Island in Second Life® offers a platform for adults with disabilities to learn the functionality necessary to enter a virtual world. This article lays out a learning model based on the andragogy theory of Malcolm Knowles, often called the father of adult learning, and complemented by other recent research on eempowerment and virtual designs for those with disabilities. A composite construct is then created as a framework to study the development of Virtual Ability Island as a destination where new residents to Second Life® with disabilities can learn basic functionality and enjoy social and physical e-empowerment.




First Person Cultural Trainer

• Game-based Simulation for Philippine Post-Typhoon Stability Operations

  Marjorie Zielke, Ph.D.; Djakhangir Zakhidov, MFA; Gary Hardee, MA; Michael Kaiser, MA

• Developing a Platform-Flexible Game-Based Simulation for Cultural Training

  Zielke, Marjorie, Ph.D.; Linehan, Thomas, Ph.D.

  This paper analyzes the design and development opportunities and challenges of creating a virtual cultural training simulation whose core features can be deployed across diverse media platforms such as personal computers, mobile devices, game platforms, the web, and Cave Automatic Virtual Environments (CAVE). This paper describes progress on a platform-flexible design approach for The First Person Cultural Trainer (FPCT), a high-fidelity simulation in which the user must build rapport with culturally rendered Afghan and Iraqi characters in order to unlock key information. FPCT was developed for Army Training and Doctrine Command Intelligence Support Activity (TRISA) at Fort Leavenworth. FPCT is part of the larger Hybrid Irregular Warfare Improvised Explosive Device Network-defeat Toolkit (HI²NT) program of federates and must operate within these architectures. In addition to design and development considerations, this paper explores the additional challenge of balancing the financial costs and performance benefits of developing for yet undetermined game engines and platforms. This paper offers suggested approaches to flexible design that can be applied to the future reuse of assets, training narratives, and core functionality in other simulations.

• The First Person Cultural Trainer

  Zielke, Marjorie A. Ph.D.; Linehan, Thomas E. Ph.D.

  This paper will describe the First Person Cultural Trainer (FPCT), sponsored by TRADOC G2 Intelligence Support Activity. FPCT is a 3D interactive simulation that trains soldiers on the values and norms of a specific culture in order to facilitate missions. The environment acts in a nonlinear way, as a Middle-Eastern geographic area would. The interactive simulation utilizes unique technology that gives soldiers the ability to read non-verbal communications of the non-player characters (NPCs) in the game. The project is currently focused on Iraq and Afghanistan, but has applications in many other cultural and geographic situations. FPCT includes four clearly defined stages which lead to the successful completion of missions. The project has adopted a challenge to develop extremely high-fidelity representations, using the living-world construct, to create an environment that serves as a training tool for cultural training before or during actual deployment.

• Serious Games for Immersive Cultural Training: Creating a Living World

  Zielke, Marjorie A.; Evans, Monica J.; Dufour, Frank; Christopher, Timothy V.; Donahue, Jumanne K.; Johnson, Phillip; Jennings, Erin B.; Friedman, Brent S.; Ounekeo, Phonesury L.; Flores, Ricardo

  Supported by funding from US government sponsors, the Institute for Interactive Arts and Engineering (IIAE) at the University of Texas at Dallas (UT Dallas) has created a serious game that lets players increase their cultural expertise in simulated Afghan rural and urban environments. Toward that end, we developed the 3D Asymmetric Domain Analysis and Training (3D ADAT) model, a recursive platform for the development and visualization of dynamic sociocultural models. This model integrates visualization, sound design, and behavioral and cultural modeling with recursive assessment tools to create a living world that is sensory and culturally realistic. Figure 1 (next page) illustrates the rich detail and character development possible with the 3D ADAT model.

• Creating Micro-expressions and Nuanced Nonverbal Communication in Synthetic Cultural Characters and Environments

  Zielke, Marjorie A.; Dufour, Frank; Hardee, Gary

  Understanding how to observe and analyze nonverbal communication of virtual humans in synthetic environments can assist warfighters with determining source credibility, deception detection, behavior observation, process training and other training objectives. The First Person Cultural Trainer (FPCT) is a high-fidelity, game-based simulation that trains cross-cultural decision making and is also a platform for the creation of synthetic characters which display culturally accurate facial and micro-expressions. These facial and micro-expressions serve as a channel for factors such as invariants and affordances within the synthetic environment that also communicate critical nonverbal information. This paper will discuss technical challenges and solutions in creating facial and micro-expressions and environmental invariants and affordances such as those needed in our high-fidelity training projects. The paper will also provide an overview of the state-of-the-art for creating nuanced nonverbal communications from the gaming industry. We conclude with a description of our solution to developing facial and micro-expressions and nuanced nonverbal communication that we feel best suits our needs at this time.

• Combining Constructive Models with a 3D Game for Enhanced Immersion

  Zielke, Marjorie A.; Gonzalez, Joe R.; Hardee, Gary

  Incorporating constructive models into a 3D game is an effective, realistic and nonlinear way to prepare warfighters for operational environment complexities. In its third development spiral, The Hybrid Irregular Warfare Network-defeat Toolkit federation, or HINT, sponsored by TRADOC G2 Intelligence Support Activity, combines an immersive 3D game-based simulation - The First Person Cultural Trainer (FPCT) - with the One Semi-Automated Force (OneSAF) and the Joint Non-kinetic Effects (JNEM) models in a hybrid model framework federation. This paper drills down on this methodology, focusing on benefits, challenges and lessons learned from integrating the OneSAF and JNEM models with the FPCT 3D game. In FPCT, the player discovers IED and insurgent networks through populace relationship building. The player wins by positively affecting the mood and cooperation level of the virtual population, thereby facilitating the collection of “golden nuggets” of information. To create this gameplay, FPCT exchanges and incorporates OneSAF data, which simulates force-on-force activity, and JNEM data, which provides well-defined civilian influences such, as mood and cooperation of a heterogeneous population in an area of operation. The player then can positively affect the JNEM-driven mood and cooperation state through culturally correct populace interaction. If populace interaction is effective, critical information golden nuggets are collected during gameplay and published back to OneSAF and JNEM to model second-and third-order effects of ongoing military stability operations. Multiple data exchanges from OneSAF, JNEM and FPCT simulate dynamically changing operational conditions. The HINT methodology for combining constructive models with a 3D game provides a platform for lessons learned and also suggests other potential ways to develop simulations in this way. FPCT won the IITSEC Serious Games Competition government category in 2011, the Innovations in Department of Defense Gaming competition at the 2011 GameTech Conference, and the NTSA Cross-Function award in 2010.

 

Videos

VR/AR Simulation Research

Media

TAMEST January 2020 Annual Conference



Joel Rizzo and Kira Lowe present a poster at The Academy of Medicine, Engineering, and Science of Texas (TAMEST) Innovating Texas Conference to academic and industry professionals from across the state. The poster was one of two selected as semi-finalists from among submissions from UT Dallas and highlighted work on the Individual Nystagmus Simulation Training Experience (INSITE™), a virtual human-based training system which provides on-demand practice for law enforcement officers learning to identify horizontal gaze nystagmus (HGN), an eye condition associated with alcohol impairment.




The Center in the News! EVP February 2020




The Center in the News! INSITE March 2019





Super Computing 2018 Conference



Dr. Zielke presents her research on virtual humans and US-Ignite high-speed networks at the Super Computing 2018 conference in Dallas, TX.





Smart Cities Connect Conference / US Ignite Application Summit 2019



DJ Zakhidov positions Walter, a holographic Emotive Virtual Patient (EVP) in augmented reality, for demonstration of the EVP System App using the Microsoft HoloLens at the Smart Cities Connect Conference & Expo 2019 / US Ignite Application Summit in Denver, CO.





News Articles

• Dallas Innovates Magazine 2021
• Simulation Lab Works To Expand Potential of Virtual Instruction
• Students Develop Patient Simulator
• Augmented Reality: Next Generation of Medical Training?
• Virtual Patient Project Aims To Develop Smarter Medical Students
• UT Dallas Lab Helps Create Virtual Impaired Driver for Identifying DUIs"
• Team Creates Medical App
• ATEC Team Receives Healthy Dose of Grants for Virtual Medical Work
• ATEC Team Developing Virtual Teachers to Help Dyslexic Students
• ATEC Game Wins First Place at Medical Simulation Conference
• ATEC Project is Finalist at International Serious Games Competition
• ATEC Nurse Training Simulations Singled Out for Awards
• Virtual Texans Celebrate Centenary of Birth of Alan Turing
• Video Game to Help U.S. Troops Wins New Award
• Virtual Medical World Has Real-Life Value
• Game Trains Soldiers in a Virtual Iraq or Afghanistan

Awards




First Place - Education/Human Performance Category, December 2018:
"INSITE," Interservice, Industry Training, Simulation and Education Conference (ITSEC), Principal Investigator.




First Place - Poster, February 2015:
"Using a Game-based Simulation to Complement Face-to-Face Medical Education - Preliminary Research Findings." Zielke, M., et al. University of Texas System Innovations in Healthcare Education Conference. Austin, Texas.




First Place - Faculty Game, February, 2015:
"The UT Time Portal." International Meeting of Simulation in Healthcare (IMSH), Principal Investigator.




First Place - Faculty Game, February, 2014:
"GLIMPSE." International Meeting of Simulation in Healthcare (IMSH), SITE Principal Investigator.




Finalist - Serious Games Competition, October, 2013:
"GLIMPSE." Interservice, Industry Training, Simulation and Education Conference (IITSEC), SITE Principal Investigator.




Honorable Mention - Game (Second Place), February 2013:
"NursingAP.com." 13th International Meeting on Simulation in Healthcare (IMSH 2013) Society for Simulation in Healthcare (IMSH), Site Principal Investigator, with Dr. Judy LeFlore.




First Place (Tie) - Demonstration, February 2012:
"NursingAP.com." Eighth Annual Innovations in Health Science Education Conference, Site Principal Investigator, with Dr. Judy LeFlore.




First Place - Government Category, Serious Game Competition, December, 2011:
"First Person Cultural Trainer." Interservice, Industry Training, Simulation and Education Conference (IITSEC), Principal Investigator.




First Place - Department of Defense, March 2011:
"First Person Cultural Trainer." Gametech. Principal Investigator.




First Place - Emerging and Innovative Technologies & Methods, January 2011:
"Can Game Play Teach Student Nurses How to Save Lives -- An Undergraduate Training Proposal for Student Nurses in Pediatric Respiratory Diseases with a Living World Gaming Construct." 11th International Meeting on Simulation in Healthcare (IMSH 2011). Site Principal Investigator, with Dr. Judy LeFlore.




Cross Function Development Award, January 2010:
"The First Person Cultural Trainer." National Training and Simulation Association (NTSA). Principal Investigator.




Honorable Mention - Poster (Third Place), January 2010:
"Pediatric Respiratory Distress." Society for Simulation in Healthcare (IMSH). Site Principal Investigator.




Top-ten Finalist - Governors Cup, December, 2009:
"The First Person Cultural Trainer." Interservice, Industry Training, Simulation and Education Conference (IITSEC). Principal Investigator.

Resources

• NursingAP.com

  The NursingAP production site.