Artist Ben Grosser’s Computers Watching Movies shows what “a computational system sees” when it watches films. Presented as a series of temporal sketches and videos, Grosser tracks the system’s “eye” movement to propose a contrast between our human “culturally-developed ways of looking” and computational watching.
It was pretty fascinating seeing the lines appear, tracking a trail from across the screen to the correlating sounds of a scene from The Matrix, but we wanted to know more about the “computer vision algorithms” are employed to the system “some degree of agency.”
“There are a number of algorithms at play in the work,” Grosser tells ANIMAL. “I’m using a computer vision library along with stochastic processes to allow the system to watch a variety of types of objects, whether its faces, buildings, etc. What the system chooses to look at in any one moment varies from clip to clip and within a clip, but in general, it looks for patterns, areas of prominence, repetitions, etc.”
“For example, in the explosion part of the Inception scene, the systems shifts its attention to the rapidly moving bits and pieces of the buildings as they fly across the screen,” as seen above.
Would the artist contrast this visually to human eye-tracking? “I haven’t paired it with eyetracking software, but I definitely recommend the great work by Tim Smith and David Bordwell on eyetracking movie watching,” Grosser says. “They’ve done studies where they correlate a number of eyetracked subjects, and use that data to create clips so you can see where everyone focuses their attention.”
But are we watching when we’re watching the watchers?
“I think Computers Watching Movies allows the viewer to do this a bit for themselves; as the viewer watches a clip they engage their memory of the scene and try to compare what they recall looking at with what the computer is looking at.” We’re literally superimposing our memories; it’s a meta sort of synesthesia. There’s almost a feedback loop. Woah.
Speaking of, Grosser’s choices seem very appropriate — key scenes from 2001: A Space Odyssey, The Matrix and Inception, specifically referencing to artificial intelligence (HAL), a malleable, artificially intelligent environment (the structure of “the Matrix” itself) and the dream environment version of the same thing — all could really just serve as metaphors for programming gaining agency.
And then he threw in Annie Hall, American Beauty and Taxi Driver. Drama, bang, bang bang.
