KnoteBoxProject ObjectivesSystem OverviewAlgorithm OverviewHardware – Audio InHardware – ADC/FiltersHardware – FPGAHardware – MicrocontrollerHardware – USB ControllerHardware – USB OutSlide 11FPGASlide 13Frequency ExtractionSlide 15Testing FFT IP CorePeak PickingSlide 18Slide 19MCUSlide 21Peak DetectionNote DetectionNote Detection- Update NotesNote Detection- New NotesMIDI over USBBudgetDivision of LaborScheduleMilestonesRisks and MitigationsSlide 32Slide 33Slide 34QuestionsKnoteBoxJoe Kramer, Leo Ovanesyan, Jimmy ThompsonProject Objectives• Recognize musical notes from an analog sound input in real-time and output MIDI data to a computerKnoteBoxMIDIA4 B7 F5 G3 A6System OverviewFPGA MCUUSB ctrlADC/filtersPower ctrlUSB outAudio inExternal PowerAlgorithm OverviewFPGAMCUHardware – Audio In•Audio jack–6.5mm connector•Typical for electrical guitars and professional audioHardware – ADC/Filters•ADC from Texas Instruments–12-bit–20 KHz sampling rate•Can still get one harmonic from highest noteHardware – FPGA•FPGA breakout board from Dallas Logic–Altera Cyclone III•24,624 logic elements•594 Kbits of memory–Programmer and power includedHardware – Microcontroller•ARM M3-Cortex from STMicroelectronics–32-bit–72 MHz–37 I/O pins–Single-cycle multiplication and divisionHardware – USB Controller•FTDI Chip–Serial to USB–Output seen as MIDI using:•Virtual COM Port driver•Roland MIDI driverHardware – USB Out•Mini USB connector from Hirose–USB 2.0 compliant–Standard connection used in camerasSystem OverviewCyclone IIIARM M3-CortexFTDI chip20 KHz ADCMini USBAudio inPower CtrlExternal PowerFPGA•We will use the FPGA for the frequency extraction portion of our project as well as to some the peak picking stage.FPGAFPGAFrequency ExtractionPeak PickingA/D Converter MicrocontrollerFPGA•Altera Cyclone Series:–Cyclone II C35 on DE2 board for prototyping–Cyclone III C25 from Dallas Logic on PCBFrequency Extraction•Using Fast Fourier Transform (FFT)•Done in FPGAFrequency Extraction•Altera FFT IP core–Transform Length – 4K–Data Precision – 12 bits–Input – Stream of input data (sound signal)–Output – Stream of FFT dataResources AvailableEP2C35 EP3C25Logic Elements (LEs) 33,216 24,624Memory Bits 420,000 594,000Testing FFT IP Core•FFT IP Core test being done in SimulinkPeak Picking•We will be picking peaks using the “Neighbor Method”•Neighbor Method - focuses on a sample and then compares it to the samples around it (its “neighbors”)•For each bin or collection of points we will find the slope of the line in the bin relative to its neighbors and the max power in the bin.Peak Picking•How this will help us:–Will be able to see if each bin is going up or down and with what intensity–Find local maximums and minimums–Then we will correlate the data to find actual peaks opposed to noisePeak Picking•What will be done in the FPGA?–Find slope and local max of each bin–Then we will send that data to the Microcontroller–The Microcontroller will use the data to find actual peaksFPGA MicrocontrollerPowerSlope[ ]MCU•The microcontroller receives frequency information from the FPGA over SPI•It then recognizes musical notes and sends MIDI events to the USB outputMCUMCUPeak PickingNote RecoginitionFPGA SPI MasterUSBMCU•STMicroelectronics ARM Cortex M3–Header board from SparkFun for development–Raisonance R-Link Programmer/Debugger with RIDE7 IDE based on gccPeak Detection•Pick peaks based on frequency data–Power relative to some threshold–Separation from neighbors•Peaks over some threshold are put in a bufferNote Detection•State of notes being played–Frequency and harmonic content•State changes trigger MIDI events•State updated for each time sliceUpdate current notesFind new notesGet peak infoSend MIDI events for note on/offNote Detection- Update Notes•From the frequency data, possible notes are given a score based on peaks near ~10 harmonics•First, the notes in the on state are checked to make sure their score is high enough•If a note is still on, its peak content is subtracted from the peak data•If a note is no longer on, it’s state changes to off and a MIDI note off event gets sentNote Detection- New Notes•Goal: find the set of notes that best fit the peak data•Two iterations over peak data–Generate possible notes based on fundamentals–Pick combination of notes with highest score•Notes that turn on are added to the notes state table and a MIDI note on event is generatedMIDI over USB•USB devices can conform to defined “classes”–i.e. Mass storage, Communications, and Audio•Conforming to the MIDI USB class allows for compatibility on PC, Mac, and Linux•MCU has USB onboard with several drivers, but no MIDI class•FTDI chip would simplify USB interface by providing USB MIDI from serial interfaceBudgetItem Description Price/Unit Units Total PriceADC TI Stereo Audio A/D $7.75 5 $38.75FPGA Dallas Logic Cyclone III Board $249.00 1 $249.00Microcontroller STMicro ARM Cortex $9.60 3 $28.80PCB 4-Layer board from Advanced Circuits $66.00 4 $264.00Audio TRS Connector 6.35 Female Audio Connector $2.77 4 $11.08Casing $30.00 1 $30.00Power $50.00 1 $50.00Dev Boards $175.00 1 $175.00Misc PartsLEDs, OpAmps, Headers,Wires, Resistors, etc. $150.00 1 $150.00 Total $996.63Division of LaborTask Joe Leo JimmyFeature Extraction x x xPattern Recognition x x xPCB DesignXxFPGA Prototyping xXMicrocontrollerXxPC Software x x xScheduleTask Joe Leo JimmyFeature Extraction x x xPattern Recognition x x xPCB DesignXxFPGA Prototyping xXMicrocontrollerXxPC Software x x xMilestones•Milestone I–Have 1st iteration of PCB ordered–FPGA – Demonstrate the FFT core working on the DE2 Board–Microcontroller – Be able to send USB data•Milestone II–Interface between FPGA and Microcontroller working–Have 2nd iteration of PCB ordered•Expo–Full System Operating CorrectlyRisks and Mitigations•Inaccuracy with picking out notes and peaks–Perform Matlab testing at the beginning to find and verify the best method–Perform FPGA testing on a test board before building our own hardware•Speed (Speed vs. Accuracy)–Consider parallelizing the system–Store information instead of having the system be real-timeRisks and Mitigations•Noisy signal (especially at Expo)–Have filters in place to remove the noise–Consider having several