Vasion Case Study: Graphic Recording in Virtual Reality
“We want something that no one has ever seen before!” When Jennifer from Vasion Software called to ask us to graphic record their upcoming tech demos, she presented an interesting challenge. We had talked about the difference between graphic recording on paper versus digitally using an iPad, and she was leaning towards digital, but then she said, “We want to be cutting edge.”
I thought for a moment and said, “Well, there’s something I’ve been wanting to try at a live event. I’ve done it before, but always remotely for virtual meetings that I could join from our studio. Would you be willing to be a bit of an experiment?” Jennifer asked what I had in mind. “I’ve been experimenting with graphic recording in virtual reality since 2018.” I showed her some examples:
“There have been a couple of recent developments, like the Meta Quest 3, which can be run standalone without a big VR computer, and the Logitech MX Ink stylus, a pen that can draw in VR. I think it’s at a place where it’s more portable and a lot faster, so it can keep up with the content of presenters. Let me do a few experiments, and if I like how they go, we can try graphic recording in virtual reality. If they don’t go well, I can always draw on an iPad.” Jennifer enthusiastically agreed.
The first thing I did was to mock up the Vasion logo in OpenBrush, an open source 3D painting originally developed by Google as TiltBrush. One of the newer developments is the ability to draw in augmented reality (or “passthrough,” as Apple now calls it.) The Vasion logo floated in the air in our studio:
If I was going to do this, I had to build my fluency with the tools. My first attempt was with a NOVA Scribes on MITRE’s Innovation Toolkit by Dan Ward. The stakes were low and the topic was, after all, innovation, so I felt experimenting a little would be OK.
Virtual Reality Recording by Brian Tarallo of MITRE’s Innovation Toolkit by Dan Ward.
I gave myself some “I like, I wish, What if” feedback:
I liked the white background: viewing the actual room through augmented reality competed with the content.
I wish the brushes and colors rendered more accurately between OpenBrush and SketchFab, the 3D platform I’d use to share the final illustration.
I wish recording in the round wasn’t so difficult to navigate in SketchFab.
What if I simplified my approach?
For my next attempt, I simplified and recorded in “2.5 dimensions:” a format that had depth but could also be viewed and read flat. I stuck to one brush and one color, so that I could focus on content and illustration. I also added a single continuous line, a technique I’d picked up in digital scribing, to add directionality and flow to the graphic. I practiced scribing a Fresh Air interview with Jon Kabat Zinn, the same interview my students listen to when they first practice graphic recording.
Brian Tarallo Virtual Reality scribe of a Fresh Air Interview with Jon Kabat Zinn.
I thought this version was much more navigable and easier to read. But I ended up relying more heavily on reference images that I’d anticipated: something that’s hard to do when you’re wearing a VR headset and can’t search Google for a picture.
That meant this was going to be a two-person job.
Thrine Kane
I’d been looking for a challenging project to work on with Thrine Kane for a long time. I pitched her the idea, and she loved it. We spent the better part of a day practicing and streamlining a workflow:
Vasion VR Workflow
We would tether the headset to a VR-capable gaming laptop. With her ear attuned by several years as a professional graphic facilitator, Thrine would listen for key phrases and imagery. She’d drop screenshots of the words and images into a shared folder that I could see in VR. I’d then place her phrases and images into the 3D space and over time, transcribe and delete the reference images. Thrine would be responsible for listening, synthesis, text, and images. I’d be responsible for layout, placement, and drawing and lettering her references.
The day of the event, we set up (then had to tear down and set up somewhere else) and connected to a large screen as a second monitor. That was the hardest part: the “smart” screen wasn’t that smart, and the venue’s A/V team had to help us connect it to the gaming laptop.
I warmed up by tracing an image of Vasion’s corporate headquarters in St. George, Utah. The mountain background ended up becoming a joining element that I carried throughout the entire composition. I dropped a “virtual camera” into the space, so that the monitor would show a static image, rather than the motion sickness-inducing live feed directly from the headset. As a result, a black outline of the headset, my “avatar,” was visible on the second screen, and people could watch me work.
When the event took off, everything fell into place. Thrine and I got into a rhythm and tempo, keeping pace with each other, as if we were doing any tandem graphic recording on paper. We were positioned off to the side of the event, so we could whisper to each other without disturbing the presenters or participants.
Vasion Reference Text
This view shows reference text (on white rectangles) placed in the 3D space before I transcribed it. The black grid to the left is the palette of tools; the u-shape above it is the avatar of the VR headset.
When the event wrapped, I loaded the 3D model into SketchFab and sent it out to participants before they even left the venue. They excitedly pulled it up on their phones: I think, when viewing a 3D model on a phone, touch controls are more intuitive than viewing on a PC.
Many participants were even more excited to see the illustration from my vantage point and try out the VR headset. Vasion generously gave away a Meta Quest 3 headset as a door prize.
Thrine did a fantastic job of capturing content. She also made notes of things we were learning! Here’s what we figured out:
Keep a subfolder to chuck extraneous and finished images into so as to not clog up the shared folder.
The person at the computer needs to watch both the laptop screen and the secondary monitor to see what content has and has not been transcribed.
Add images and text that are grayed out so they’re easier to trace and transcribe.
Have a looooong (30’) HDMI cable.
Save often! OpenBrush doesn’t have an auto save. At one point, my foot wrapped around and unplugged the cable and shut the whole thing down… it was pure luck that I had just saved.
The MX Ink stylus has a battery life of less than two hours. I had to switch to handsets.
The gaming laptop’s fan is loud! Have a way to pump up the audio volume so we can hear the speakers.
Use OpenBrush’s flat plane grid to record on for easier 2.5-dimension viewing.
Use offset color (color fill slightly behind outlines)
Use a spectator camera.
SAVE OFTEN!
And one more thing I discovered: we worked in VR for six hours straight. I didn’t realize the toll that it would take on me! Most of the time, I can graphic record for hours and hours on end with no break and not feel fatigued. But this took a lot of stamina. I suppose my day-long gaming marathons are in the rearview mirror. Altogether, Thrine and I had a lot of fun, and I’ll be looking for the next time we can do VR graphic recording together.
You can view the final composition at SketchFab here.
Contact Lizard Brain today to learn more about our services!