Virtual ConductingAndy Lin and Brandon YoshimotoProject Overview Description: An interactive music player which allows the user to control the sound of a composition through hand movements. The user conducts holding a blue LED in each hand Movements are interpreted as beats and affect the playback of music Music qualities controlled:Volume: left hand controls low frequencies, right controls high frequenciesArticulation: left hand controls low frequencies, right controls high frequenciesTempo: right hand controls the tempo of the piece Inputs: Camera Video Music Outputs: Visualization on Monitor SpeakersHigh-Level Description Two main units: Video ProcessingInputs: Camera dataOutputs: Video display and movement qualities Audio ProcessingInputs: Movement qualities and audioOutputs: Processed audioVideo Unit 3 Main Parts: Camera Input Storage and Retrieval:Retrieves data from the camera Visualization and Video Processing: Calculates position of the hands and displays on monitor Motion Analyzer: Interprets hand movementsCamera Input Storage & Retrieval Each pixel stored as 8-bits: 5 bits for Y, 3 bits for Cb Store four pixels per location in ZBT ZBT Memory usage: (729 wide x 487 tall)/4 = 88755 locations per frame 65 Mhz clockCamera ImageConductorInputVisualization and Video Processing Calculates position of the user’s hands: left in the left half plane, and right in the right half Displays hand positionsDetails Video Interpretation: For noise reduction: Requires at least three successive pixels to be blue before registering a pixel as part of the hand Position Calculator: Calculates a running sum of x and y positions for the blue pixels in each half of the screen. Uses Xilinx Pipelined Divider v3.0 to divide this sum by the count of pixels of the desired color to get the average coordinates of the hand Display Output: 3 components Displays blue pixels detected by the camera, leaving other colors as black Sprite to follow left hand movement Sprite to follow right hand movementMotion Analyzer Determines the start and end of a beat When coordinates stay within a certain distance for more than 10 frames, beat ends After a beat ends, when movement starts again, a new beat starts Methods for calculating qualities: Amplitude: difference in the x and y coordinates of successive beat starts Acceleration: average second difference of the 10 frames following a beat start Beat period: number of samples counted between two beat starts.Audio Processing Beat-by-beat processingOne beat stored in SRAM at a timeEnd/start of beat in RAM identified by Beat detectorBeat signal instructs system to move onto next beat. TimingsAudio read in from ROM every clock period (27 Mhz)Final audio output at 48 Khz LP/HP Filter15 segment convolution Beat detectorLPAudio[7:0] exceeding a threshold amplitude signifies a beatBeatAudio[15:0] contains LP signal in BeatAudio[15:8] and LP signal in BeatAudio[7:0]Flash ROMData fed in directly through RS232 interface on PCFlash ROMLPF HPFBeat DetectorAudio[7:0]LPAudio[7:0]ZBT SRAMBeatAudio[15:0]Tempo ModulatorArticulation and Volume ModulatorAC 97 InterfaceMain FSMMusicBeatFlash_reset_bFlash_addess[23:0]Flash_data[15:0]Flash_byte_bFinalAudio[7:0]RAMRdata[35:0]OutAudio[15:0]Ram_data[35:0]beat_startacceleration_left[10:0]amp_left[10:0]AC 97Command_dataac97_readyAc97_synchAc97_bit_clockAc97_sdata_outCommand_addressCommand_validBeatPeriod[10:0]beat_period[10:0]AccL[10:0]VelL[10:0]ROM FSMSRAM FSMRAMWdata[35:0]RAMWEOBeatPeriod[10:0]TRdata[35:0]TWdata[35:0]TWERam_address[18:0]Taddr[18:0]We_bRam_cen_bacceration_right[10:0]amp_right[10:0]AccR[10:0]VelR[10:0]HPAudio[7:0]Audio Processing Modulebeat** Note: all modules will include a 27 Mhz CLK and RESET inputsbeat_startLastAddr[18:0]FirstAddr[18:0]BeatPeriod[10:0]RAMAddr[35:0]MusicBeatTempo Modulator TheoryDivides Audio signal into indivisible “divisions” whose time period is greater than that of the lowest audible sound.Scientifically, sounds less than 20hz are inaudibleWe will use 15 Hz divisions –3200 samples (48 kHz)Divisions removed or added to change tempoHas been tested on MatlabTempo Modulator Division Converter Rounding by truncation Simplifies original signals into 3-bit representations Interval[2:0] will be the truncated version of OBeatPeriod[10:0] Skip[2:0] will be the positive difference between the two truncated versions of beat periods Add <= (BeatPeriod > OBeatPeriod) Division counter Counts the number of divisions. Takes an enable signal from Address Counter which is triggered every 3200 address counted. If division count = interval[2:0], skip[2:0] is added or subtracted from the accessing address via Addr_mod[2:0] Special Cases Problem when approximated speed increase is inaccurate, or the beat period suddenly changes. If beat arrives prematurely, rest of beat that needs to be played is cut off If beat arrives late, repeat last few divisions until beat arrives.Tempo Modulator Division ConverterBeatPeriod[10:0]OBeatPeriod[10:0]skip[2:0]interval[2:0]addDivision counterAddress Counterenable Addr_mod[2:0]OutAudio[15:0]beatTempo ModulatorOutAudio[15:0]BeatPeriod[10:0]OBeatPeriod[10:0]beatTRdata[35:0]TWdata[35:0]TWETaddr[18:0]LastAddr[18:0]FirstAddr[18:0]TRdata[35:0]TWdata[35:0]TWETaddr[18:0]LastAddr[18:0]FirstAddr[18:0]Articulation and Volume Modulator Mathematical multiplication in time domain of tempo-modified audio and “Articulation Function” Separate Articulation and Volume Modulating for LP and HP signals Articulation Function 3rddegree polynomialA(t)t(One beat duration)Address CounterArticulation FunctionAddr[18:0]XOutAudio[15:8] Result[7:0]Articulation and Volume Modulator UnitAddress CounterArticulation FunctionAddr[18:0]XArticulation and Volume Modulator Unit+Result[7:0]FinalAudio[7:0]acceleration_left[10:0]amp_left[10:0]beat_period[10:0]acceleration_right[10:0]amp_right[10:0]beat_period[10:0]RAM FSM Access Control Tempo Modulator and Main FSM share access to ZBT RAM Tempo Modulator at 48 kHz, Main FSM at 27 mHz Data from Main FSM delayed 3 clock cycles Allocation Alternator Insures data from previous beat is not overwritten by next beat Addresses fed into SRAM FSM are “virtual addresses.” SRAM FSM adjusts these addresses
View Full Document
Unlocking...