Data and Time Series Analysis TechniquesControlsCalibrationSample StatisticsGaussian (Normal) DistributionData and Sensor QualityTime and Frequency DomainTime Resolution in Sampled SystemsFiltering of SignalsMassachusetts Institute of Technology 12.097Data and Time Series Analysis TechniquesMassachusetts Institute of Technology 12.097• Does the work stand up to scrutiny?– Use of controls– Calibration– Data quality– Data processing– Documentation and record-keeping!Ask Yourself:Massachusetts Institute of Technology 12.097Controls Did you really measure what you thought?• Rat Maze: Is the maze acoustically navigable? (R. Feynman)• Mass Spectroscopy: When you put in a sample of known composition, are the other bins clean?• When measuring electrical resistance, touch the probes together.Check a precision resistor too. • Resonance in load measurement rigs?• When measuring hull resistance, does zero speed give zero force?Take every opportunity to eliminate doubt!Massachusetts Institute of Technology 12.097Calibration• More time can be spent on calibration than the rest of the experiment! • Sensors should be calibrated and re-checked using independent references, such as:– Manufacturer’s specifications– Another sensor with very well-known calibration ÅÆ– A tape measure, protractor, calipers, weights & balance, stopwatch, etc..• Calibration range should include the expected range in the experiment.• Some statistics of the calibration:– Precision of fit (r-value or σ)– Linearity (if applicable)• Understand special properties of the sensor, e.g., inherent nonlinearity, drift, PWM outputMassachusetts Institute of Technology 12.097Sample Statistics• Sample mean m:• Sample standard dev. σ:σ = sqrt [ ( (x1-m)2+ (x2-m)2+ … + (xn-m)2 ) / (n-1) ] • Error budgets for multiplication and addition (σA is standard deviation of A):(A + σA)(B + σB) ~ AB + AσB+ BσAExample: (1.0 + σ0.2)(3.0 + σ0.3) ~ 3.0 + σ0.9(A + σA) + (B + σB) = A + B + σ(A+B)Example: (1.0 + s0.2) + (3.0 + s0.3) = 4.0 + σ0.5Massachusetts Institute of Technology 12.097Gaussian (Normal) DistributionProbability Density Function f(x) ~ Histogramf(x) = exp [ - (x-m)2/ 2σ2] / sqrt(2π) / σThis is the most common distribution encountered in sensors and systems.+/- 1σ covers 68.3% +/- 2σ covers 95.4%+/- 3σ covers 99.7%Area under f(x) is 1!Massachusetts Institute of Technology 12.097Data and Sensor Quality• Signal-to-Noise Ratio (SNR): compares σ to the signal you want• Repeatability/Precision: If we run the same test again, how close is the answer?• Accuracy: Take the average of a large number of tests –is it the right value?Massachusetts Institute of Technology 12.097Time and Frequency Domain• Fourier series/transforms establish an exactcorrespondence between these domains, e.g.,Xm= s cos( 2π m t / T ) z(t) dt * 2 / T, m = 0,1,2,…Ym= s sin( 2π m t / T ) z(t) dt * 2 / Tz(t) = X0 / 2 + Σ Xmcos( 2π m t / T ) + Σ Ymsin( 2π m t / T )00TTMassachusetts Institute of Technology 12.097Time Resolution in Sampled Systems• The Sampling Theorom shows that the highest frequency that can be detected by sampling at frequency ωs= 2π/∆t is the Nyquist rate: ωN= ωs/ 2.• Higher frequencies than this are “aliased” to the range below the Nyquist rate, through “frequency folding.” This includes sensor noise! Åanti-aliasing filters• The required rate for “visual” analysis of the signal, and phase and magnitude calculation is much higher, say ten samples per cycle.Massachusetts Institute of Technology 12.097Filtering of SignalsFilterxxfUse good judgement!filtering brings out trends, reduces noisefiltering obscures dynamic responseCausal filtering: xf(t) depends only on past measurements – appropriate for real-time implementationExample: xf(t) = (1-ε) xf(t-1) + ε x(t-1)Acausal filtering: xf(t) depends on measurements at future time – appropriate for post-processingExample: xf(t) = [ x(t+1) + x(t) + x(t-1) ] / 3Massachusetts Institute of Technology 12.097A first-order filter transfer function in the freq. domain (where jω is the derivative operator):xf(jω) / x(jω) = λ / (jω + λ)At low ω, this is approximately1 (that is, λ/λ)At high ω, this goes to 0magnitude, with 90degrees phase lag (λ/jω = -jλ/ω)Time domain equivalent:dxf/dt = λ (x – xf)In discrete time, tryxf(k) = (1-λ∆t) xf(k-1) +λ∆tx(k-1)Massachusetts Institute of Technology 12.097• BUT linear filters will not handle outliers very well! • First defense against outliers: find out their origin and eliminate them at the beginning!• Detection: Exceeding a known, fixed bound, or an impossible deviation from previous values. Example: vehicle speed >> the possible value given thrust level and prior tests.• Second defense: set data to NaN (or equivalent), so it won’t be used in calculations. • Third defense: try to fill in. Example:if abs(x(k) – x(k-1)) > MX,x(k) = x(k-1) ;end;Æ Limited