E4896 Music Signal Processing (Dan Ellis) 2014-01-22 - /21Lecture 1:Course Introduction!!!!Dan EllisDept. Electrical Engineering, Columbia University[email protected] http://www.ee.columbia.edu/~dpwe/e4896/!11. Course Structure"2. DSP: The Short-Time Fourier TransformELEN E4896 MUSIC SIGNAL PROCESSINGE4896 Music Signal Processing (Dan Ellis) 2014-01-22 - /21Signal Processing and Music•Music is a very rich signal"•Signal Processing can expose some of it"•We want to get inside it!2freq / kHz01234freq / kHz01234freq / kHz01234freq / kHz01234time / sec0 2 4 6 8 10 12 14 16 18 20E4896 Music Signal Processing (Dan Ellis) 2014-01-22 - /211. Course Goals•Survey of applications !of signal processing to music audio"music synthesis!music/audio processing (modification)!music audio analysis!•Connect basic DSP theory !to sound phenomena and effects"•Hands-on, live investigations !of audio processing algorithms"using Matlab, Pd, ...!3E4896 Music Signal Processing (Dan Ellis) 2014-01-22 - /21Course Structure•Two weekly sessions"Monday: !presentations, practical exposition, discussion!Wednesday: !presentations, practical, sharing!!•Grade structure"20% practicals participation!10% one presentation!30% three mini-project assignments!40% one final project!4E4896 Music Signal Processing (Dan Ellis) 2014-01-22 - /21Flipped Classroom•Video Lectures are required viewing !before Monday session"!about 30 mins each!!recorded last year!!bring one question !(at least) to class!5E4896 Music Signal Processing (Dan Ellis) 2014-01-22 - /21Hands-on Practicals•Music Signal Processing involves !connecting algorithms with perceptions"•Our goal is to be able to ‘play’ with algorithms to feel how they work"In class, on laptops, in small groups!•We will use several platforms:!6Matlab ProcessingPureData (Pd) PythonE4896 Music Signal Processing (Dan Ellis) 2014-01-22 - /21Projects•There will be 3 “mini projects” assigned"approx. week 3, week 5, week 7!provide basic pieces, but open-ended!involve implementation & experimentation!•Also, Final project"on a topic of your choice!implement some kind of music signal processing!due at end of semester!•Good projects ..."... start with clear goals!... apply ideas from class!... evaluate performance & modify accordingly!•Could continue into summer...!7E4896 Music Signal Processing (Dan Ellis) 2014-01-22 - /21Project Examples•Music !Transcription!8Figure 1: Screenshot of the PD patch implementing the Scheirer algorithm.Much of the logic is hidden inside the “combfilterbank” object at the b ot t om.2.2 Tempo SelectionEach of the six onset pulse signals (one for each band) is fed into its ownbank of 150 comb filters – you can think of the se filters as mapping to 150p os si b le tempos that we want to detect. A comb filter is essentially just a delayimplemented a circular buffer. The delay time on each filters corres ponds tothe tempo that that filter is meant to detect – so for example, the filter thatis detecting a tempo of 90 bpm has a delay time of9060=1.5 Hz, which yieldsabuffer length of441001.5= 29400 samp l es. The tempo selector then, for eachbpm, sums the energy across all 6 subbands at that te mpo. The tempo withthe hi ghe st total energy is chosen as the tempo of the piece.Because of the sheer number of filters (150  6 = 900), it was impossible toimplement this using standard PD objects. Therefore I created a PD Externalwritten in C [3] and used this within my PD patch. The external has six inputs,one for each subband, and has two float outlets – one indicating the detectedtempo, and the other indicating the phase. I added a signal outlet as well for2•Beat Tracking•Mood !ClassificationFig. 4. Comparison of effectiveness of different combination of features.the most effective combination of features. The result is shownin Figure 4 Although there are some features especially useful,I also observe that the number of features enhance the overallclassification accuracy. Therefore, the best combination is touse all the features.Fig. 5. Composition of hits in each class.VI. EVA L UAT I O N A N D RESULTSIn the evaluation part, I will use 100 test samples (20samples for each mood class) to evaluate the effectivenessof my classifier. Essentially, there are two tasks I wouldlike to investigate in the evaluation. First, I use the bestcombination of feature found in the feature selection part,which is to use all of the features. The average hit rate is0.64. However, I noticed that the accuracy of each moodclass are dramatically different. The hit rates for angry andFig. 6. Comparison of effectiveness of different features for distinguishingtwo classeshappy are almost 1 while hit rate for calm is 0. Thus I furtherinvestigate the composition of hits in each class. The result isshown in Figure 5It is quite intuitive that happy songs will bemistaken as angry songs since they both have fast tempo. Inthe same logic, it is also intuitive that calm songs, sad songsand scary songs are easily obfuscated. However, it is hard toenvision that calm songs would be largely classified as angry.This phenomenon is still left to be investigated. Second, Iwould like to investigate which features are capable to tellthe difference between any two different classes. Differentcombination of features will be applied in the above two tasksto find out the most effective combination of features. Theresult is shown in Figure 6. Previously we have known thattimber features are very useful. In this investigation, we furtherfound that the harmony features are useful when the two songsare melodically different. For example, harmony features areeffective in terms of classifying calm songs and happy songs,while they are not useful when distinguishing calm songs andsad songs.VII. CONCLUSIONIn this project, I implemented and evaluated a music moodclassifier. Specifically, I focus on investigating the effective-ness of each feature. Compared with other feature selectedworks, we found that tempo feature does not always work wellwhen classifying music moods, while timbre features play asa key role of recognizing music moods. Also we found thatnumber of features also affects classification accuracy. Whenfurther analyzing the classification results, we found that someclasses are easily recognized to another one, and then thefeatures that are suitable for distinguishing two mood class arefound. For example, harmony features