A Hand Controlled Digital Audio Synthesizer6.111Behram MistreeAlexander SanchezOverview of Functionalityz Hand Controlled– LEDs detected by camera– LED’s on hands correspond to musical notes and volumesz Synthesizer– 2 Proposed Methods– Looping Capability– Single Instrument6.111Proj.CameraHand Position DetectionMonitorKeyboardSynthesizerDetected X PositionDetected Y PositionNumber of HandsType of HandPlay Back RecordXYNumber of HandType of HandAudioAC97SpeakerReadyHand Controlled Audio Synthesizer8 707070 703330MemoryVolumeNoteAddressData InData Out8 8 Write5107719ReadyHand Detection Block Diagramcamerahand_detectntsc_decodewithin_distanceYCrCb2RGBtest_acceptable10 10 10YCr Cbclkrst88 8BGRX 71010newmeasuredacceptableX 2888RG Bacceptable65 mhz clkget_hcount_vcounthcountvcount1010YCrCb30data validfvh3vclktv_in_ycrcb10Hand Detection Essentialsz Checks each pixel for coloring (test_acceptable).z If vector of last 10 pixel colors all contain correct color value, then identified as LED (hand_detect).z Differentiates between distinct LED’s by making sure sequences are far enough apart (within_distance).= LED?=???= Same LEDFar enough apart -> Different LED’sHand Detection Statusz Currently, recognizing horizontal motion of red LED. z Need to test and retest thresholds.z Better interface to monitor module.Plan A for the Synthesizer…Method of Synthesizingz Subtractive Synthesize– Create Oscillatory signals rich in harmonicsz Sines, Square waves, Saw Tooth– Pulse Wave Modulation– Combine signals– Apply an Attack, Decay, Sustain, Release (ADSR) envelope– Low Pass FilterOscillator1PulseWidthModulatorADSRMultiplierFilterOscillator2PulseWidthModulator+xxReadyReadyReadyReadyVolumeVolumeNoteNoteNoteNoteNoteInstrumentReadySynth_OutReadySubtractive SynthesizerModule8888885513131410101010SubtractiveSynthesizerModuleSubtractiveSynthesizerModule+MemoryModuleNote1Note2Volume1Volume2+Voice_In+Audio_OutLoop01LoopSynthesizerModuleMight only produce one note at a timeReady88888888101055And Plan B for the Synthesizer…Sine WaveGeneratorSquare WaveGeneratorSawtooth WaveGeneratorNeural NetworkADSRFSMSamplerVolumeVibratoErrorCalculatorNoteNeural NetworkSynthesizerSynth OutAttack Decay Sustain ReleaseVolumeReadyReadyReadyModulating FrequencyTimerΔiά888888888 8 810Note10488Building Up the Network: The Neuronz Neuron– Takes weighted sum of inputs– Applies the Sigmoid functionz High if in surpasses the threshold, otherwise low– Outputs a single value that can drive multiple neurons∑aiginiai= g(ini)ActivationFunctionInputFunctionOutputBias WeightW0,ia0 = constantajWj,iOutputLinksInputLinksA Mathematical Model for a NeuronWeight GeneratorInput FunctionActivation FunctionGenerator88Δi888888 888888άΔjg’(in)ing(in)Neuron88ForwardAdjustForwardInputs from previous levelForwardBuilding Up the Network: Making Connectionsz Cascaded layers of neurons– Outputs of one layer drive inputs of next layerz Weights applied to inputs at each levelzForward PropagationOutput units aiHidden units ajInput units akWj,iWk,jA Simple Multilayer Neural Networks8888AttackDecaySustain Release888Sine Wave Saw Tooth Wave Square WaveOutput LevelInput LevelLevel 2Level 3Level 4Level 5W2,1W3,2W4,3W5,4W6,5Neural NetworkSome Formulasz Compare output of network to a sampled audio file of instrument to get squared error vector – E = 1/2 ∑(yi–ai)2, where y is desired output vectorz Ei= ith component of error vectorz Modified Error Δi= Eix g’(ini)z Weight update rule for output neurons in output layers:– Wj,IWj,i+ ά x ajx Δiz Propagation rule for Δ values:– Δj= g’(inj) ∑ Wj,iΔiz Weight update rule for neurons in hidden layers:– Wk,jWk,j+ ά x akx ΔjBackward Propagationz Find error vectorz Find Δ values for output unitsz Start at output layer and repeat for all layers– Propagate Δ values to previous layer– Update weights between layersz Training– Continue to adjust weights via backward propagation until network produces satisfactory outputSchedulez 11.3.2006: Several LED hand devices built.z 11.11.2006: MONITOR module completed (x, y coordinates can be displayed tomonitor).z 11.15.2006: HAND POSITION_DETECTION module completed (Camera capable of extracting xy coordinates of LED's).z 11.15.2006: Code for KEYBOARD module (found online) is incorporated into project. Code for interfacing to AC 97 codec (also found online) is incorporated into project.z 11.15.2006: Decision on methodology for synthesizing notes finalized.z 11.17.2006: NOTE and VOLUME modules completed and tested.z 11.22.2006: Interface to ZBT for playback and recording completed and tested.z 11.29.2006: SYNTHESIZER module completed and tested.z ****Debug, Debug, Debug****z 12.7.2006: Work on possible extensions. (See final section of proposal.)z 12.11.2006: Report completely written.Possible Added Featuresz Sound Effects: – In researching how to synthesize sounds, we observed that there were several other effects that we could incorporate into our project. Examples include an echo and slowing or speeding up the sound during playback.z LED Control Tower: – Our synthesizer can be controlled by any LED movement. Therefore, if we build a device that turns sequences of LED’s on and off, the synthesizer should play notes that correspond to those LED sequences. If we make this LED device controllable by circuitry, we should be able to play complicated music precisely.z Voice In– Our synthesizer allows users to make music using body movements. It would be nice to add a voice elementz More Instruments– Our lab report calls for only synthesizing a piano. If our efforts are successful, we may be able to add on other instruments such as guitar, saxophone, and bass guitar.z Display– Any musical instrument is difficult to learn. If our synthesizer proves particularly nonintuitive, we may make a more sophisticated display that makes our synthesizer easier to play.Referencesz Norvig, Peter; Stuart, Russel. Artificial Intelligence: A Modern Approach. Second Edition. 2003. Pages
View Full Document
Unlocking...