Sam and Andy Rolfes self-describe their work as “overly navel-gazing, obsessed-with-layers, weird.” From visualizing songs by Lady Gaga and BLACKPINK to facilitating mind-bending, improvisational performances at MoMA, the duo are in a perpetual toggle between real life and the screen. Cleverly using VR, mixed reality, figurative animation, and motion capture tools to highlight the absurdity of life, dream up ironic characters, and make anti-capitalist statements, Sam and Andy discovered and perfected a digital fluency that’s uniquely theirs. They also happen to be brothers. 

Sam and Andy sat down with us to speak about their 3D modeling software from childhood, why improv comedy is seminal for their practice, and the game they’re designing—fingers crossed, the first of many.

Growing up, your mother ran a 3D studio. Did you take anything from that environment into your practice? 

Sam: Our mom had big books of Blender tutorials. She’d be like, Hey, we need a spaceship for our company. I remember trying for a few weeks and being like, This sucks, It’s just wireframe meshes, and you’re moving points around. I moved back to making violent stick-figure animations or something in Flash soon after. All the 3D stuff is roughly from the time we were 10 to 13. I left it behind until I was 22. It’s really full circle. 

Andy: We had the very early version of Blender and Cinema4D. It still felt so hard to get into it, so we kept to stuff that was far more expressive—2D animation and our traditional painting training.

Sam: I have vivid memories of replaying Half-Life 2 hundreds of times. I would go back into levels as a 10-year-old, and I would direct my playthroughs—almost like a speed run-through, but for style. I’d move my camera so that a certain thing would get highlighted. I role-played with myself.

Sam: Getting back to 3D didn’t happen until we came across ZBrush, a 3d sculpting program. The program’s internal logic is completely topsy-turvy. It speaks a language unlike any other 3D software. Once you get a handle at the most basic level, you can squish things. So much of what we wanted was that immediate, responsive squishiness.

Andy: The learning curve on ZBrush was harsh. I was working at the fashion agency Two Hustlers. My boss there had me use Cinema4D for making assets. I wanted to see how much I could push it. I then learned ZBrush.

Sam: I was doing these very ornate, mixed media things where I was constructing stuff and layering oils. I would spend hours on them. They became too overwrought, and the idea got buried. Going digital allowed us to riff in the moment. Finding something that was immediately responsive and expressive was the key for us. ZBrush, Unreal, mo-cap and livestream stuff—our trajectory arises from that immediate visceral response, responding at the moment to whatever external environmental stimuli. 

How did your respective educational trajectories influence the kinds of things you make?

Sam: We both went to school for painting in one way or another. Many of the seeds were planted there but we just ended up being funneled into the Juxtapoz-industrial complex. Just being into a teen boy and say, Oh, I want to do sick, graffiti, cool figures. The high school and college art school system didn’t prepare us—we didn’t have the language to work digitally in any real way up until recently. Even when I was at SAIC, I barely did anything with 3D. A class taught Unity, but I stayed far away from it because all the kids were just making the same recreation of a 3D gallery. You could walk around, but it was like, Okay, we’re reconstructing the white box gallery that I already hate the experience. It doesn’t have the good parts, which are the alcohol and the socializing. It’s the only reason people go to these things in the first place. 

Andy: I went to the University of North Texas. I got a major in watercoloring before they closed that down. With watercoloring, you have to plan things out a bit, but you’re still dealing with water and pigments, so there will be a randomness. That laid a nice groundwork for being more reactive with artwork than just trying to assign a single pathway.

How does your collaboration generally work—especially as it pertains to projects foregrounding real-time embodiment?

Sam: Andy comes from a photographic background. He has a consideration of light that I don’t, he also has, but you also have a more realistic figurative nature. 

Andy: I remember Sam sketching out post-humanist, distorted work like back in high school. He follows through with a much more loose sense of form. I am interested in the more tradesman, craftsman-y stuff. I learned proper anatomy and focus on these much more formless elements to break it down later.

Sam: Andy keeps us grounded in reality. He keeps me from budgeting our project too insane and yelling at me when I get too excited on a call. I promise a lot of work to a client. That’s part of the dynamic. 

Andy:  It comes down to our history—me going to state school, Sam going to art school. Sam making a way earlier as a freelancer, and I bounced from internship to internship, to in-house job before coming over to freelance. We’re similar enough where it counts and likewise, different enough where it counts. 

Sam: Our skill sets complement each other, but they’re close enough that we can be pretty unified with the vision. Often we’ll concept together about what the performative perspective? What is the core mechanic of performing through the space that is going to be inherent to the concept? We can brainstorm that together, and Andy’s very helpful in keeping it current.

We’ll trade off designed to the characters and then as it goes into production, often I’ll start handling the environments and the arc through the environments, like the choreography, functionally of the camera and the characters and stuff, I’ll start building the environments, blocking them out in VR and walking through them and certain construct, different elements.

While I’m doing that, Andy and Alexander Bauman start the whole process of rigging, which is essential to how the characters move. Then, we bring them into the scene and we’ll all together start rigging them. Typically, I’ll do the VR puppetry and maybe—and usually if it’s not a musician or different performer, I’ll be wearing the mo-cap suit and then in the camera.

The physical performance is often me; sometimes me and Andy. It’s a matter of improvising within the space and I’ll have roughly mapped out roughly the trajectory through the space.

Sam: Spatially, the performance or video ends up being a one-shot. Functionally, it’s like a game playthrough. For us, that gameplay perspective is natural. We’ll start improvising in VR, playing around with the characters, finding what their personalities are. You think they’re going to move a certain way depending on how they’re rigged up. It might be, Okay, they can’t move very fast before they explode. Or, They look really cute when they do this shoulder-shrugging thing.

It ends up being a choreographed dance between both characters. The POV is set up so that we are embodying the camera, so there’s a dialogue between the camera and the characters. That develops throughout 20 takes. I’m recording, honing things in—sometimes I’ll block it out. Often, I’ll perform as one character, then we’ll save that. Then I’ll put on a VR headset with mo-cap and then play as the other character against myself, and then build the pieces from there.

We set up the conditions for something to happen—we’re open enough and have enough technical articulation to accomplish that. We feel out what the best thing is for the space. Then, we render it. Andy might help with lighting, and then once we get the scene blocked out, he’ll go in and do passes. Then we render it. We’re editing in-engine, and often, it’s a lot of single takes.

For your projects with a real-time component, how do you think about the relationship between what’s happening on screen versus in physical space?

Sam: We get a suggestion from the crowd—that serves as a catalyst for them to riff on and makes it clear to everybody that this is happening live. We choreograph it in a way where you have to slowly unpack and elaborate the performance language for the audience.

It’s very easy to just drop in, do a bunch of crazy shit that people can’t really understand how it’s being generated. So, we take a lot of care to build the performative mechanics directly into the concept and have that story come from there. 

I always direct the performers. If I’m wearing the suit myself, I do large arm and leg movements. Everybody’s on stage in front of a big LED screen—they’re backlit, in mo-cap suits. I’ll be doing VR camera and moving us through. Andy will be conducting the scenes, triggering the same progression, toggling the lights live, programming transitions.

We start out basic. A body moves the thing on screen. So you see the silhouette of my body, you see that it ripple across onto the screen. We got that relation; that’s the introduction. And then we start to abstract it. There are two bodies tied together, with the camera focused on one body.

So you’re watching your avatar on screen, which results in a series of levels of increasing abstraction. That allows us to environmentally build in the narrative around them. The characters, the increasing intensity, the environment…all of that is present. Then, it’s all about responding to the music, to each other, and to the audience, and letting that guide us in the moment. 

For our Berlin Atonal performance, it was myself and another dancer. It was like, Okay, it’s not just about the one-to-one relationship of our bodies on screen. Let’s code in a mechanic where they’re a figure in the center...its expressions are tied to our closest together as we’re dancing. When it’s abstracted that way, a body that is the result of your movements and another person’s movements defines the movements you do. With the choreography that arises, you’ll start hitting different poses and angles with your body—movements that are completely different than you would otherwise do, inherently from a dancing perspective.

Andy: We rely on “show, don’t tell.” You can’t directly communicate with the audience because there’s music playing. We heavily rely on visuals to convey something, but we utilize the relationships to convey and explore these various scenes and set-ups. Each performance has its climax, and then we fall into the denouement. 

Sam: That becomes a method of moving through the scenes, and a way of heightening things as we go. In improv comedy, they establish base reality. They raise the stakes on one absurd element that they’ve written. It’s a continual process of agreeing on a reality, then raising this absurdity. What if two bodies within this type of space reacts this way, then we heighten it again? You can build dramatic action this way. In improvisational work, you are given a large latitude to do your thing. 

Andy: Our process has that similar throughline to improv—laying the groundwork that elevation than exploring that. A good adage for environmental 3D work is 90% based in reality, 10% based on whatever is going just pure creativity—

Sam: —or heightening it. We have both come through the art world, which has its own arguably outmoded values in terms of permanence and auteurism—all this shit that I’m trying to unlearn. Also, we grew up as snobby turntablists—we’re all about technique; we’re anti-pop.

Andy: We grew up with street art, ‘lowbrow’ art. We are pushing away from an old head mentality and being able to grow into new things. How can we apply that into our work? Maybe we can have a nice balance of trying to be serious about a topic or a narrative and keep things fun where we need to so it can actually engage people in more than one way.

Sam: It’s conversational, dynamic. The broader world doesn’t do that; but we have similar shared interests of culture, music, art, gaming. So where can we situate ourselves in terms of contextual equal footing and be able to do our thing?

We’re pretty stubborn, to our financial detriment. There’s a lot of corny stuff that we probably could be doing that we don’t do. Still, we want to speak to a lot of people, and a culture that’s not just a series of micro-niches. How is that sustainable in a world where all communication media are completely captured by capital?

On that note, what spaces have you found the most successful, gratifying, or accessible for your work?

Sam: Music videos have been the entry point for us to the world of characters and narrative and speaking to people more directly and having specific concepts we try to communicate.

Often, it’s about vibe and you’re trying to convey a narrative without dialogue, or if with dialogue, it’s lyrics trying to convey something that may heighten the music. It comes to the added benefit of the current cultural configuration. Music and games—people care about them more, which is understandable. It’s more of a utility throughout their day than visual art. 

In your livestreams and performances, what is the role of audience input or feedback? 

Sam: The audience is the chaos element. That lends itself to comedy. A lot of our live shows are intermediated by the internet. We hit a dry patch when we just start figuring out what this character’s name is.

The audience just submits a bunch of trolly shit. Still, it gives us an idea of that character. They were just trying to be an asshole on the chat, but it gave us something to go off of.

Andy: Foundational work came from interacting. For Superdeluxe, Daniela Hamilton wrote and produced this telenovela series. They did it four times over an hour. The audience dictates how the story would play out. That laid the groundwork for our understanding of the audience and how we can play with them as we get better at our own tools. I want to see how we can apply it in different places and keep their spirit going.

Sam: We feel best in this experimental space. We’ve been doing so-called ‘virtual production’ since before Unreal had a way to animate like this. For our first videos, I was palming a prototype Vive or Oculus headset in my hand and screen capturing, because there was no sequencer in Unreal to make animations. 

We could be in a golden age of non-gaming, interactive stream right now. It just hasn’t happened. I blame that on market consolidation, vertical integration, mega-corporations buying up these studios, and then consider then I understand them thinking that they’re redundant and then asking them. Many people are really, really good, who are internally working at these different companies, and they get fired and have their stuff turned over. It’s not breaking news that a monopolistic con consolidation of these markets is bad for creativity, but that’s been the story for us these last few years.

The relationship of doing experimental art within a commercial market of any kind is fraught. And I’m coming to terms with that — we’re really just trying to make a living doing our weird thing. It’s just a matter of making it substance sustainable.

You’re working on a game! What can you tell us about this project? 

Andy: We gave in!

Sam: Functionally, all of our videos or games, right? They’re all playable to a certain extent. It’s just only we can play them, and they’re not very optimized. Games impact people in a much deeper way right now than many other forms—commercially, monetarily, sustainably. 

Right now, we’re in production building up this game that’s going to be audience participatory lore component where they can determine where the story and functionality goes. Every month we’re going to drop a new update of the game—potentially allowing people to mod it.

We can bring it into the main canon arc by allowing people to submit characters or environmental things in that model. And building out a thing throughout however long—build increasingly functional weird experimental game.

We don’t like the idea of asking people to just work on spec just for the coolness of seeing their stuff from there. It’s way more interesting to have everyone truly participate and feeling like they have an investment in this story we’re building together. The hope is that anybody who contributes—develop some lore or is really active in voting—buys into a heavy percentage of the game’s overall profits.

The hope is that we build up functionality so that how much you get out of it depends on how much you put in. We’re trying to set up something that’s sustainable for everybody to make a fun thing together.