using an FFT analysis

Nik and I created this mobile sensor that broadcasts light levels and accelerometer data to an online database via text message. The above visualization was created by running around for five minutes. Light levels are represented by the opacity of blue and faster changes in tilt (when one is running with the sensor in hand, for instance) are represented by longer lines. 

We are currently working on a proper write up that will include code and schematics. 

Nik Hanselmann and I installed a sensor in the Sesnon Gallery that collects environmental data (light level, sound level, proximity and temperature) from the space and uploads them to an online database every two seconds. We developed software that creates a generative drawing that changes with the data over time. Above are screenshots of the drawings at different points in time.

We exhibited the live generative drawing in the Porter Faculty Gallery and the DANM Lounge. An online version can be viewed here: http://transmogrify.me (let it run for a few hours for a rich, complex image). 

I projected some live reactive video filters for the Dragging an Ox through Water show last week. Above is a photo of the “Bubble Filter” in action. 

Above are stills from four of the filter modes. Modes can be selected by pressing the 1-5 keys or the “b” and “w” keys. The modes are as follows:

1: Unfiltered video
2: Black and White Bubble Filter
3: Colored Bubble Filter
4: Grey/Red Frame Difference Filter (based on the “Frame Differencing” sketch by Golan Levin)
5: Grey/Red Frame Difference Filter w/noise (based on the “Frame Differencing” sketch by Golan Levin)
B: Black screen
W: White screen

If you have a built in video camera on your computer, you can load up the sketch in Processing and start playing with live filtered video. 

Processing Code

In this program, each bubble has a note (in C major) associated with it. When a silhouette in the video feed overlaps with the bubble, the note is sent to an audio program (Reason, in this case) in the form of a midi note. The bubbles can be moved left and right using an Arduino equipped with an accelerometer.

Processing Code

Similar to the above, but now two video feeds are incorporated. Now bubbles can only be played if they overlap with red sections of the video. Red sections are created when two silhouettes from separate video feeds overlap, so cooperation between people in the two video feeds is necessary to create notes. 

Processing Code

Both these programs build on the programs from my previous two posts. 

In this project, people in two separate rooms are videotaped. The two video feeds are fed through Processing, converted into silhouettes and superimposed over each other. Both rooms receive the same projection of the superimposed silhouettes. When silhouettes from the two rooms overlap, the overlapped area turns red and a midi note is sent to an external audio program (Reason, in this case).

The video feeds are initialized using a background subtraction technique, so only new objects in the space (ie: people) are fed back as silhouettes. The program sends midi notes using the RWMidi library. The program contains four different modes, some in which participants hear the same sounds, some in which they hear different sounds. All notes are currently in the C major scale. 

My version sounds like this (all sounds can be modulated in the sound program you choose):

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

Processing Code

For the 24h Thesis I also created live video filters with Processing. These are composite frames of filtered video. 

Processing Code: [1] [2] 

One of my projects for last week’s 24h Thesis was this noise-generating box. It has two speakers, four buttons and two potentiometers mounted on a cardboard box and connected together with a breadboard and an Arduino. Each button has a different sound, which can be modulated with the two potentiometers. It sounds like this: 

Audio clip: Adobe Flash Player (version 9 or above) is required to play this audio clip. Download the latest version here. You also need to have JavaScript enabled in your browser.

I’m still learning how to play it!

Arduino Code