Brown CS Blog

by Bruce Donald Campbell (RISD, Brown CS Visiting Scientist)

What do you get when you bring together a seasoned popular art school illustrator, an accomplished computer graphics scientist with a new virtual reality surround theater, a respected scientist with a compelling field of study, and twenty-one eager participants in a semester-long course focused on how to support science through design in virtual reality? You get plenty of material for a Conduit column at a minimum. And, if the you is a year-long visiting scientist to Brown CS, you aren't about to pass up the opportunity to observe as a participant.

A VR Design For Science Course

I had heard of the cross-listed course known as VR Design for Science from a “Visualization Viewpoints” article Dan Keefe wrote for CG&A in 2005 [1]. In that article, Dan hypothesized:

“[Virtual Reality is] one of the technologies that can most benefit from artistic insight, since guidelines for good visual depiction are far less developed in the unconventional visual space of VR than in more traditional media”.

Having watched artists make huge contributions in many research projects at the University of Washington's Human Interface Technology Lab in the 1990s, that hypothesis is certainly one I continue to feel confident is evident through observation, at least when providing narrative or illustration is useful. Based on Dan's article and discussion with the course's long-term providers from Brown CS and the Rhode Island School of Design’s (RISD’s) illustration department, I expected the class to facilitate further investigation.

David Laidlaw and Fritz Drury co-taught the 2005 class around the science of arterial blood flow and a winged mammal's ability to fly. For 2015's class, they enticed paleontologist Stephen Gatesy to share his work on foot movements in Triassic theropod dinosaurs. The leap upgrade of the class' VR reality theater, called the Yurt for the shape of the physical projection space, added to the excitement of a new science focus.

Needless to say, the course reading list provides a large bibliography of relevant material to exploring the science of bird and dinosaur tracks in VR. I found the most useful readings to include historical perspectives on visualization from art [2][3], science [4][5], and computer science [6]. To weave those themes together, useful reads included publications on human perception [7], human-computer interaction [8], and the hardware/software affordances of the new Yurt. In front of that background, science publications presented perspectives relevant to dinosaur science and related bird science [9][10].

Figure 1: Students created weather maps to explore multivariate data representation. In this example, Timothy Blaine-Kuklo represents temperature, wind, cloud cover, pressure, and front delineation on a single map. The Yurt allowed the artist to explore intermediate steps and stack them to try out encoding combinations while also then discussing them interactively in critique. All article photos courtesy of Johannes Novotny.

By having experts and emerging experts in these subjects co-present in a shared studio (with a minimum of two, two-hour required sessions a week), the course also provided an ideal configuration for distributed cognition: tangible objects for interaction (casts of dinosaur tracks and bird anatomy), artifacts of embedded cognition (diagrams and multimedia exhibits), and rich social processes for creating new objects and artifacts to support a scientific pursuit (as a well-designed studio class readily provides).

Through Steve's introductory presentation, I quickly confirmed my expectation that dinosaur science is hard. While many uncover and describe new fossil skeletons, his work involves the tracks dinosaurs of varying anatomy made in varying substrates using varying kinematics. Steve studies modern day birds like turkeys and guineafowl and sets up experimental environments where they walk through designed substrates – all done with an eye on optimal x-ray recording of foot movement underground. Track simulations that can be generated from reconstructed foot motion involve a large number of variables. Visualizing the data involves choosing among the variables and encoding them to co-exist in a meaningful perceptive presentation.

To help develop their own personal process for making visual trade-offs, participants first grappled with a popular visualization opportunity: weather map generation (see Figure 1). Discussions about scope and relevance shed light on choosing which weather characteristics to consider including in a visualization. Discussions about human perception and cultural implications shed light on how to represent those characteristics visually and texturally.

Figure 2: Students created visualizations that honed in on an aspect of foot movement in order to verify their understanding of what active bird footprint generation data should provide for visualization. In this example, Johannes Novotny traced points of foot anatomy through a walk cycle and used color to represent displacement of the substrate. Orange represents displacement of the substrate at one point in the down motion and yellow represents displacement slightly later. Red and purple were added later to illustrate upward motion. As in many cases, the resulting visualization appeared as an attractive sculpture floating in the Yurt's 3-D projection space.

Students were asked to bring forward the lessons learned from weather visualization to create hypothetical dinosaur foot movement visualizations aimed at providing useful science support – helping scientists rethink their hypotheses, clarify their experimental goals, and even alter the way they collect data. The potential of possible support became evident through the diversity of solutions participants suggested, sketched, created in the Yurt, and then critiqued over many weeks. Participants critiqued visual reasonableness based on watching bird walking movies and data simulation videos that demonstrated foot movement and substrate deformation. Incorporating those hints, some students created visualizations that focused on structure, while some focused on motion, and some focused on representing the forces involved in the interaction of both (see an example in Figure 2).

During most studio sessions, class participants filled the Yurt to participate in critique sessions. Heightened by the retinal visual fidelity, bold color representation, and a sense of being immersed inside data and narrative representations, I appreciated that some students had tremendous eye-hand coordination in creating detailed immersive visualizations. I noticed others compensate through a control of descriptive language they used to communicate about missing or deformed features to the mind's eye. I found some had an inherent ability to sketch ideas out quickly, describe fuller implementations verbally, and then readjust their sketches to consider feedback – even catching themselves mid-statement to adapt to a new train of thought that came from describing an idea to us. Over time, as students improved their ability to dynamically manipulate the point of view, scale, and orientation of their creations, we noticed their ability to promote ideas that required manipulation with VR peripherals (wand, tracker, button-based devices) to fully integrate.

The final project assignment required students to expand their visualization ideas into science-support tool prototypes. Students created storyboards to explain the tool user's experience, identified menus and virtual objects that would allow the user to make meaningful choices, and then built a visual representation of a practical state of the tool in action in the Yurt. Figure 3 shows one example of a final project. Before reaching this state in the tool, the user has selected from different bird anatomy options, created a substrate shape and consistency, chosen a locomotion speed, and suggested a tile density for data collection purposes.

Figure 3: A scientific support tool prototype provides a data visualization for a hypothetical running guineafowl's footprint sequence. Three depths of substrate are disrupted by foot interaction and uniquely colored tiles are displaced to represent their final resting spot. Visual filaments show the paths taken by each tile and tile orientation identifies the direction of last movement. The initial concept offered for critique is inset in the upper-left.

I noticed each student's evolution in developing a unique confident personal style as they iterated upon the role of lead commentator when considering an idea in critique. Confidence seemed facilitated by mastering all the aspects of the course suggested by lectures and reading list and incorporating the more effective aspects of each participant's emerging abilities. Confidence became evident in the growing comfort and ease demonstrated when suggesting a relevant contribution to critique. A social comfort came with familiarity of each other and the process. While some strengths of each participant could be anticipated from their previous experiences and areas of expertise, the willingness of students to offer critique outside of that expertise while among those who were their senior gave me the ultimate satisfaction as an observer. When one student struggled with a personal weakness, other participants assisted to promote their perspective, often by temporarily taking over the presentation. The supportive demeanor of the career accomplished scientists, artists, and computer scientists created a positive environment in which the students could evolve their confident style.

Notes On The Technology

Fritz consistently provided a perspective that the materials involved in illustration offer strengths and weaknesses and that an artist needs to explore first-hand to find the strengths that best support an effective presentation. David consistently reminded participants that technologies provide affordances that can only be taken advantage of when perceptible and cognitively accessible. The Yurt's highly responsive 360-degree retinal resolution provided by 69 high resolution, high luminosity projectors, and the CavePainting software that runs within it [11], provided a unique technology not readily found in any close configuration outside of the class – the most extensive use of the Yurt since it completed construction in September 2015. I watched as students took ideas they sketched out with traditional artist materials (often paper-based) and mature software tools on common computing workstations (Photoshop and 3-D modeling tools), and mapped those sketches to useful representations in the Yurt. Students found constraints in the CavePainting software a relief at times given the enormity of choices and their personal time constraints. Students found the fidelity of the experience expressive in describing what 'could be done' with different software available. The Yurt's ability to simulate the look (and even suggest the feel) of a wide variety of materials seemed critical for the class' success. Through tackling opportunities and constraints, participants provided valuable feedback for considering the evolution of the Yurt going forward.

Figure 4: Emily Schnall took advantage of the Yurt's high fidelity to immerse us in a photorealistic dinosaur habitat. Jurassic Park and other dinosaur films had provided us previous experience with dinosaur environments but the freedom to choose our own visual experience through such an environment feels different and evokes a different type of emotional engagement – one perhaps hard to put into words and perhaps more valuable in the artist's domain.

Thoughts From The Experts

Fritz reminded that the course always takes on the problem of working with a kind of immersive experience that is really new and fundamentally different from the previous territory of traditional design fields, but also from the quotidian physical world. That’s our challenge and our inspiration. When asked about what made the 2015 class memorable, Fritz called out the photorealistic background objects students included to help immerse our focus into the human relevance of the data being explored (see Figure 4). He reports that the new Yurt significantly improves the illusion of immersion provided by a color range far beyond what he had to work with in the older Cave, and that the Yurt is big enough for the everybody to comfortably get inside for critiques, an essential component of any collaborative design process. The improvements provide an increased sense of engagement and suggest added value for supporting context, metaphor, and narrative.

Fritz and David acknowledged that they were fortunate to have some graduate students in computer science who approached the technicalities of scientific visualization in a substantive way, but also some illustration students who pushed the boundaries of using the virtual space of the Yurt to create elaborate immersive environments. Another group of particularly cogent thinkers were able to construct “clean” designs that facilitated usability and visual clarity. This variety reflects positively a number of potential applications of the skill set that participants can acquire in class: visualization for scientific research; creation of immersive displays for public use; interactive interface design, and pure entertainment. Although they stress the importance of designing for the scientist user, the course embraced all these corollary skills and interests.

David echoed the success of many of these objectives he had when creating the Yurt and was still integrating the feedback provided by the class experience. In its first three months, the Yurt has been most successful when using a megapixel image viewer or high-resolution video player or building new applications upon the legacy process – the one used to evolve the previous Cave's technology. As the talented support staff looks to support new and improved methods for getting applications running in the Yurt, David wondered about the metrics to use in pursuing a promising best solution compared to an ecosystem of marginally improved ones (there are many candidate approaches the support staff has been considering). The pursuit is complicated as many applications benefit from specialized trade-offs in data size and format considerations, messaging strategies, performance considerations, and interaction facilities.

Steve reminded me that currently a track simulation involves over 3 million particles. He was fascinated to see the object of his research through the eyes of bright, creative students. Sometimes a key point that he thought was obvious from his papers and presentation was not communicated well enough. In other cases, an extremely subtle aspect that he never expected anyone to understand was picked up perfectly. Although it's always instructive to get feedback from non-scientific audiences, that feedback is typically verbal. Artists creating images and material for the Yurt provided visual feedback that was extremely powerful.

At an intermediate level, Steve was repeatedly struck by how art students were comfortable working with approximations and caricatures of real data. Sketches, cartoons, and inexact representations usually conveyed relationships just as well, and often even better, than perfectly scaled and anatomically accurate objects. The freedom to exaggerate or ignore certain elements made him see his data differently. He really appreciated the class' attention to color, line and perceptual cues – and yet they varied their approaches far more than expected.

Steve added that

“Seeing our project in the Yurt was invigorating. 2-D sketches and concept drawings that looked promising did not always pan out in 3-D. Yet other examples that I thought might go nowhere shocked me in the Yurt. Sometimes the impact was simply a novel perspective (standing above a lifesize trackway or plunging down and into the depths of a magnified track). The stereo and textures often combined to offer visual overlap that I've never had before, seeing through multiple layers without losing spatial context.”

Conclusion

In addition to the experts, student commentary both implicitly, and explicitly via Tim, overwhelmingly suggested that “the class gave me a lot to think about”. Upon watching a trajectory of confidence and effectiveness emerge by the predominantly college age participants, and experiencing their growth in being able to work together face-to-face within a technology (instead of isolated with personal gadgets as buffers), I find myself wishing that their careers include many art-tech-science collaborations where they can innovate in new ways with their unique trajectory as an age group. Or, if not, that they will continue to weave technical information from visualization research with multiple technical fields of knowledge to find innovative directions for their work.

References

[1] D. F Keefe, D. B. Karelitz, E. L. Vote, D. H. Laidlaw, Artistic collaboration in designing VR visualizations, Computer Graphics and Applications, IEEE 25 (2), 18-23

[2] R. M. Kirby, H. Marmanis, and D. H. Laidlaw. “Visualizing Multivalued Data from 2D Incompressible Flows Using Concepts from Painting.” In VIS '99: Proceedings of the Conference on Visualization ’99, 20, pp. 333-340, Los Alamitos, CA: IEEE Computer Society Press, 1999.

[3] D. H. Laidlaw. Loose, artistic “textures” for visualization. IEEE Computer Graphics and Applications, 21(2):6–9, March/April 2001

[4] P. S. Stevens, Patterns in Nature, Little, Brown and Company; 1st edition (1974).

[5] N. C. Hale, Abstraction in Art and Nature, Chapter 3, pp. 57-85, Dover Publications; Reprint edition (1993).

[6] D. H. Laidlaw, R. M. Kirby, C. D. Jackson, J. S. Davidson, T. S. Miller, M. da Silva, W. H. Warren, and M J. Tarr, Comparing 2D Vector Field Visualization Methods: A User Study, IEEE Transactions on Visualization and Computer Graphics, Vol 11, No 1, Jan-Feb 2005.

[7] B Martinez and J. Block, Perception, Design, and Practice, Chapters 1-5, Prentice Hall College Div (1985).

[8] J.J. Gibson, The Ecological Approach to Visual Perception. Chapter 8, Boston: Houghton Mifflin, (1986).

[9] P. L. Falkingham, S. M. Gatesy (2014), The birth of a dinosaur footprint: Subsurface 3D motion reconstruction and discrete element simulation reveal track ontogeny, Proceedings of the National Academy of Sciences

[10] S. M. Gatesy, K. M. Middleton, F, A. Jenkins Jr & N, H. Shubin, Three-dimensional preservation of foot movements in Triassic theropod dinosaurs, Macmillan Magazines Ltd letters to nature, Nature, Vol 399, May 13, 1999

[11] D. F. Keefe, D. Acevedo, T. Moscovich, D. H. Laidlaw, and J. LaViola. Cavepainting: A fully immersive 3D artistic medium and interactive experience. In Proceedings of ACM Symposium on Interactive 3D Graphics 2001, pp 85–93, March 2001