\documentclass[../book.tex]{subfiles}
\graphicspath{{\subfix{.}}}
\begin{document}

\chapter{Smoothing Data Responsibly}
\label{11SmoothCrim}

\setcounter{section}{-1}  % Start subsections from 0

\input{prereqs/11SmoothCrim_prereqs}

\newpage
\section{Problem}
\setcounter{subsection}{-1}  % Start subsections from 0

In the process of working up the FTIR data in the last chapter, you and your classmate notice some small peaks that appear to come and go in the spectra.  Your classmate remarks, ``I wonder if these are signatures of the intermediates in this reaction? I've always wanted to get some more detail about the mechanism.  We could integrate these, too, and see if they track the kinetics of an intermediate species.'' Looking closer, you think, ``They're so small that it's hard to tell if they are there at all---much less to know where the center of the peak is, what the limits of integration should be, or how to subtract the solvent background.''

You say, ``We can integrate these peaks with the same basic code as the azide peak before, but we will probably have to smooth the data first to make it easier to set up the parameters.''  Your classmate looks nervous.  ``My advisor always complains about people smoothing data.  He says there's rarely a good reason to do it, and it can make you see things in the data that aren't really there.''

This is not a bad point, but you need to do something because you'd just be making educated guesses with the noise you have in the data as recorded.  Not only that, but since the solvent background is so much more intense than the actual peak, any error in background subtraction will be a large error in the estimate of the integral.  This also seems bad.

``Let's give it a shot. Your advisor is correct to be cautious about smoothing, but maybe it will make sense in this case, and maybe we can also prove to your advisor that it's okay in this particular case.  We can still integrate the raw data, we just need some well-founded numbers to put into this script.''

\subsection{Understanding why implementing smoothing requires caution}
\label{Understanding_why_implementing smoothing requires_caution}
%\setcounter{subsection}{-1}  % Start subsections from 0

How bad can it possibly be to smooth?  And how do you even do it?  Looking through the documentation of the Scipy \cite{library}, there are a bewildering number of smoothing options to choose from.  You also don't see the most common option you can think of, the moving average.  What is going on?

You track down \first and ask, ``What do you know about smoothing? And how do you do it in Python.''

``Oh, smoothing is super interesting!'' They stand tall and gives you her best noble look. `` `With great power comes great responsibility.'  Why do you want to smooth something?  Hopefully not to `make it look better!'"

You explain what you want to do, but then get curious: ``What is wrong with making it look better?''

``Let me show you something.''  They take your laptop, turns it around so that you can't see the screen, and in a minute, turns it back around to show you Figure \ref{fig:SmoothCrim:SmoothRand}.

\begin{figure}[h]
    \centering
    \includegraphics[width=4in]{Figures/SmoothRand.png}
    \caption{Data that \first shows you, wondering how many peaks there are.}
    \label{fig:SmoothCrim:SmoothRand}
\end{figure}

``How many carbonyl stretches do you see in this spectrum?''

You squint at the screen.  ``Four?''

``Nope!  This is just an array of normally-distributed random numbers that has been smoothed.  There's nothing there. That's the issue.''

You are slightly irritated, but the point is taken.  ``I can see how it can go wrong, and we will have to keep this in mind, but it must still be valuable in some cases.  So how do you do it?''

\First explains that the easiest and most common ways to smooth are moving weighted averages and local regressions.  Moving weighted averages basically take each $y$ point and average them together with a set number of adjacent $y$ points.  The moving average that you are familiar with is actually a special case of a more general averaging approach.  In the moving average, every point in the average has the same weight, but that need not be the case.  You can weight the points however you like: using a binomial distribution, a normal distribution, an exponential, or really anything you can imagine.  

Local regression-type methods are a little more complicated, but basically boil down to fitting a subset of your data to a function that ignores the noise, and then using that function to produce the new smoothed $y$ point for each $x$ point.  In fact, one of the most common of these methods, Savitzky-Golay smoothing, was invented by chemists who wanted to smooth noisy HPLC chromatograms!

``There are many other approaches, but moving averages and local regression should cover most of your bases without having to learn a lot of new math.''

``So how do we know which one we should use?''

\First says that more smoothing will reduce noise, but it can shorten and broaden your peaks.  Different methods will cause different amounts of peak distortion.  Luckily, you can just make up test data to see how these effects would impact your real data.

\subsection{Comparing smoothing approaches using test data}
\label{Comparing_smoothing_approaches_using_test_data}

\subsubsection{Binning data to preserve meaningful statistics}
\label{Sec:N:11SmoothCrim:BinningBackground}

The simplest approach to smoothing is to simply combine adjacent points into one point---so called ``binning.''  This has the advantage of not fundamentally altering your data: you are essentially zooming out from it, letting fluctuations cancel one another out and seeing the broader trends.  It also has the advantage of being mathematically simple, which allows for relatively straightforward rigorous error propagation.  The problem with binning is that the ``smoothing'' effect is not as significant as with other methods that actually \textit{change} your data.  Since binning is essentially summing multiple measurements, signal to noise improvements with binning scale as $\sqrt{N}$, where $N$ is the number of points being binned together.\footnote{See, for instance, Bevington, P.; Robinson, D.R. \textit{Data Reduction and Error Analysis for the Physical Sciences}, 2002}  Figure \ref{fig:SmoothCrim:BinCompare} shows the effect of increased binning on data.

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{Figures/BinCompare.png}
    \caption{As we increase the binning on the histogrammed data, the apparent noise decreases.  Note that as the bin width increases, the number of points that fall within each bin increases, so the $y$-axis scal increases.}
    \label{fig:SmoothCrim:BinCompare}
\end{figure}

Binning is most naturally applied to distributions of data that could be plotted as a \cite{histogram}; that is data which could be visualized in a bar chart with a continuous independent variable.  In these types of charts, we often use as the dependent variable the ``frequency'' that a measurement appears between the edges of an individual bar or bin, or the total number of measurements between the edges.  Thus, we can simply make the bin limits wider to include more measurements, thereby reducing the apparent noise.  Spectra, which are just histograms (the probability of the sample absorbing photons between 405 and 406 nm, for instance), x-ray powder diffraction patterns, and images can also be straightforwardly binned even though we do not typically think of them as histograms.

\subsubsection{Understanding the common moving average}
\label{Sec:N:11SmoothCrim:MovingAverageBackground}

The moving average is likely the smoothing technique with which you are most familiar.  In a simple moving average, $N$ adjacent points are averaged together to determine the value of a single point.  This is similar to the binning process outlined above, except for two important points: the moving average has the same number of points as the original signal, and the moving average \textit{changes} the data in a way that may make it complicated to derive the uncertainty in the smoothed data from the uncertainty in the measured data.  For the simple $N$-point moving average, our smoothing function is just an array of $N$ points that is normalized such that its sum is 1, i.e. each point is simply weighted by $1/N$.

Most moving average smoothing methods can be cast as convolution problems.  Convolution as a mathematical and numerical concept is ably covered in many resources online, and thus will only be briefly outlined here for the specific context of smoothing.  Figure \ref{fig:SmoothCrim:MovingAverageExplain} graphically depicts the process of convolution.  Essentially, convolution for smoothing is the process of taking a signal that you would like to smooth, multiplying it by a function that you would like to smooth by (for a simple moving average, that function is just a flat top function that is constant for the data points to average and 0 everywhere else), and integrating the resulting product.  Translation of the smoothing function against the original signal specifies the range of data over which you are averaging.  

You may say, ``So what if moving average smoothing changes the data?''  One particularly problematic example is the smoothing of relatively narrow peaks.  Figure \ref{fig:SmoothCrim:SmoothCompareWidth} shows an example of this.  As we increase the number of points to average, the data does indeed get smoother.  However, we can also plainly see that the peak is reduced in intensity and is significantly broader.  If you require high resolution, like in FTIR or NMR spectra, or if it is important that peak intensities are quantitative, then this smoothing approach is probably not the best one.

\begin{figure}[H]
    \centering
    \includegraphics[width=4in]{Figures/MovingAverageExplain.png}
    \caption{A graphical representation of simple moving averages.  The dashed lines represent the smoothing function.  The smoothing function is moved one point at a time across the noisy signal (top), and after each movement, multiplied by the signal.  This product is then averaged, and the average is used as the new data point (plotted vertically to the left).  Each new point corresponds to a shift of the smoothing function across the noisy data.}
    \label{fig:SmoothCrim:MovingAverageExplain}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{Figures/SmoothCompareWidth.png}
    \caption{As we increase the number of adjacent points to average, peak widths increase and amplitudes decrease.}
    \label{fig:SmoothCrim:SmoothCompareWidth}
\end{figure}

Of course, you are not limited to simple moving averages; you can choose any weighting of points for smoothing when you use the convolution approach.  It is common to weight points such that the current point (middle point of the collection of points being averaged) is carries the most weight, and then each nearest neighbor carries less and less weight.  Binomial and Gaussian distributions of weights can often be used to minimize the broadening effects described here.

\subsubsection{Smoothing by local regression}
\label{Sec:N:11SmoothCrim:LocalRegressionBackground}

The basic problem with the class of moving average smoothing approaches that we just covered is that they do not explicitly attempt to describe the underlying trend in the data; the trend is just revealed by averaging away noise.  We could instead attempt to fit the data to some function,\footnote{As we did in Chapters \ref{04EqFit}--\ref{05CompFit}} which explicitly treats the underlying trend in data with a model described by a well-chosen mathematical function.  However, this function often does not exist, and we often do not even have a well-formed have a numerical function to use as the fit model.\footnote{Like we did in Chapter \ref{06AnalFit}}

We can instead choose a relatively simple function, only try to fit a small region of the data to it, and use the best-fitting function to calculate the smoothed $y$-value for that specific $x$-value.  We can then shift the small region of data to fit to the next point, and do it all again.  This is the idea behind \textbf{local regression}-type smoothing algorithms.  Figure \ref{fig:SmoothCrim:LocalRegressionExample} shows a simplified, intuitive view of this process for a linear local regression.  Clearly, this smoothing approach also changes the data.

\begin{figure}[h]
    \centering
    \includegraphics[width=.75\textwidth]{ChapNarrative/11SmoothCrim/Figures/LocalRegresionExample.png}
    \caption{A simplified view of local regression smoothing.  At each point, a function is fit to a range of surrounding points.  The best fitting function is then used to calculate the ``smoothed'' value for that point.  This example shows a linear function, but low-order polynomials are also often used.}
    \label{fig:SmoothCrim:LocalRegressionExample}
\end{figure}

One of the most well-known (and chemistry-relevant) methods of local regression smoothing is the Savitzky-Golay filter.\footnote{See Savitzky, A.; Golay, M. J. E. 
Smoothing and Differentiation of Data by Simplified Least Squares Procedures, \textit{Anal. Chem.} \textbf{1964}, \textit{36}, 1627–1639.}  This method was developed by Savitzky and Golay to smooth noisy HPLC data and remains to this day a very commonly used method.  The approach is to fit a polynomial of order $n$ (where the order is the largest power in the polynomial, i.e., 2 for a parabola) to an $m$-point-wide span of the data.  There are a wide range of possible values of $n$ and $m$.  As $m$ gets larger, more points are used, and there is a larger smoothing effect.  As $n$ gets larger, more complex curvature can be reproduced, which is a double-edged sword: on one hand, more significant changes in the data can be handled, like smoothing narrow peaks in a spectrum, but this can also make specific noise patterns look like real peaks by accident.  In practice, the choice of $m$ and $n$ can be made by modeling artificial data in which the underlying curve is known, as shown in Figure \ref{fig:SmoothCrim:SmoothCompareSavgol}.

\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{Figures/SmoothCompareSavgol.png}
    \caption{Comparison of a five-point moving average (left) with Savitzky-Golay smoothing using two different sets of $n$ and $m$ (center, right).  With Savitzky-Golay smoothing, you can typically achieve more noise reduction while maintaining peak widths and amplitudes better.}
    \label{fig:SmoothCrim:SmoothCompareSavgol}
\end{figure}

More generally, these kinds of approaches are classified as locally estimated scatterplot smoothing (LOESS) or (if the data is weighted during fitting) locally-weighted scatterplot smoothing (LOWESS) methods.  These methods are computationally more intensive than moving averages, and also make uncertainty estimation even more difficult (except for in some special cases with analytical solutions), but they are often quite effective once validated.

\subsubsection{Smoothing by more advanced methods}
\label{Sec:N:11SmoothCrim:OtherSmoothingApproaches}

You have only scratched the surface of smoothing here, but beyond these methods, you mostly find approaches that are mathematically much more complex or are only used in niche cases.  A quick survey reveals that two more approaches are quite common.

Spectral methods make use of the fact that a signal often lies in a specific frequency range (very often, below a specific cut off frequency).  Since noise spans all frequencies that can be represented in the data, it is possible to filter out the high-frequency part of the noise to achieve a smoother curve.  These approaches are usually classified as digital filters.  Their implementation typically leverages fast Fourier Transforms, which are commonly used in chemical instrumentation like FTIR and NMR spectrometers.\footnote{This class of approaches is mathematically more complex and beyond the scope of this book.}

Smoothing splines are similar in concept to (and in some cases, a special case of) LOESS or Savitzky-Golay smoothing, but are often separately implemented (for instance, in the Scipy \cite{library}.  These methods usually use cubic functions for fitting, as cubic spline-based numerical techniques are particularly well developed.\footnote{You will see another use of splines in Chapter \ref{12Interp}.}

\subsection{Deciding when smoothing is appropriate}
\label{Deciding_when_to_smooth_data}

Now that you have more background on smoothing, you still don't quite know when you should use it.  You go back to \first to pick their brain, and they explain, ``Usually, you should not smooth your data.  Smoothing data typically fundamentally changes the data, sometimes in a way that will lead you to make qualitatively wrong conclusions.  Some situations in which you probably should \textbf{not smooth}:
\begin{enumerate}
    \item \textit{To make your data look better.}  If your data is too noisy to find the trend/peak/feature in it that you expect, then smoothing may create a trend/peak/feature out of pure noise as we saw in Figure \ref{fig:SmoothCrim:SmoothRand}.  If you are worried that other researchers will not believe your data, it is better to prove them wrong using rigorous statistical methods!
    
    \item \textit{Before fitting.}  You should essentially never fit smoothed data.  Just fit the noisy data, the math still works fine!  Since smoothing changes your data, your fit result will likely not be the same.  Furthermore, typical goodness-of-fit metrics will be misleading, giving you more confidence than you should have in the statistical relevance of your results.
    
    \item \textit{Before integrating.}  Again, smoothing changes your data.  Just integrate the noisy data.  If the noise is actually random, it will mostly cancel out during integration because you should have a roughly equal number of points that are too high as you do peaks that are too low.
\end{enumerate}

\noindent So why do we spend all of this time learning how to smooth?  There are times where smoothing is a very good (or maybe the only) idea:
\begin{enumerate}
    \item \textit{As an intermediate step in data processing.}  Sometimes you need to determine parameters, limits, and models for exploratory data analysis.  Smoothing can give you an idea of starting points for these analyses, then you can do the full analysis on the unsmoothed data.  Furthermore, when you are automating data analysis, your automated algorithm may be fooled by noise.  Smoothing data first, then applying automated methods to set up initial guess, to find peaks, etc. can get your algorithm moving in the right direction.  Then, it will be more likely to be robust when applied to the unsmoothed data.
    
    \item \textit{When you want derivatives of noisy data.}  If your data is noisy, then derivatives of the data will show significantly more noise due to the fact that the slopes between adjacent noisy peaks are much larger than those of the underlying signal.  Particularly when you do not have a model to fit to your data, smoothing can allow you to more reliably estimate the value of the derivatives of your data.\footnote{To learn more about derivatives, see Chapter \ref{14NumDer}.}
    
    \item \textit{When you have a pattern of variations in the data that you already understand and want to remove.}  As long as you know where the noise is coming from, and its frequency (or period), then you can smooth it away to more easily see the underlying trend.  However, it is usually still better to model the noise, if you can, than to smooth it out.
\end{enumerate}

These are both not exhaustive lists, but will provide you food for thought as you run into situations in which you think smoothing might be an effective tool."

\section{Solution}
\setcounter{subsection}{-1}  % Start subsections from 0

What you need to do is pretty clear:
\begin{enumerate}
    \item You will modify your \cite{script} for integrating FTIR peaks to smooth the data first.\footnote{See Chapter \ref{10NumInt}}
    
    \item From the smoothed data, you will determine the background and the limits of integration.

    \item You will take the background you got from the smoothed data and subtract it from the raw data.
    
    \item You will then perform the integration on the background-corrected unsmoothed data in order to avoid any numerical errors from the smoothing.
\end{enumerate}
First, though, you need to figure out what is the right smoothing approach.  You decide to test different smoothing methods on simulated data.  You can accomplish this by:

\begin{enumerate}
    \item Creating the $x$- and $y$-values of a simulated absorption peak.
    \item Adding noise to each $y$-value.
    \item Smoothing the noisy data.
    \item Comparing the smoothed peak height and width to the ``clean'' one.
\end{enumerate}
Creating the ``clean'' peak will be no problem, you have already calculated a Gaussian curve.\footnote{See Section \ref{Generating_test_data_to_evaluate_the_accuracy_of_numerical_analysis}.}  How do you make it noisy?

\subsection{Generating noisy test data using random numbers from \texttt{numpy.random}}
\label{Generating_noisy_test_data_using_random_numbers_from_numpy.random}

In its purest form, noise comes from random errors in measurement and should be characterized by a random normal variance (a.k.a., Gaussian, Bell curve, etc.) from the ``true'' value.  You can simulate noise by having the computer generate random numbers.  For a measurement like FTIR, most of your noise is in the $y$-values, so you can simulate noise by adding positive or negative random numbers to your $y$-data.  Numpy provides a suite of random number generators that will do the trick.

You notice that the \textit{standard} random number generator in Numpy produces numbers \textit{evenly} distributed across the range $0 \leq x < 1$.  This won't do, since you need numbers that (1) greater than and less than 0, and (2) have a probability distribution described by the Gaussian probability distribution.  A little more digging shows that Numpy has functions that allow for this. In particular, the \lstinline{normal()} function of \lstinline{numpy.random} will produce the random numbers you want. This function accepts arguments that allow you to specify the center and width of the distribution as well as the size of the array that it returns.  You give it a shot:
\begin{pyin}
noise = np.random.normal(loc = 0, scale = 0.2, size = 100)
\end{pyin}
and check to see if it gives you the expected results:
\begin{pyin}
np.mean(noise)
\end{pyin}
\begin{pyout}
0.0042539097714522165
\end{pyout}
\begin{pyin}
np.std(noise)
\end{pyin}
\begin{pyout}
0.20823387191838444
\end{pyout}
Not exactly the numbers you gave it, but hey, these are random!  You try a couple more times and see that the numbers are different each time, but close to the values you expect.\footnote{Sometimes, you are testing some code and want the \textit{same} ``random" numbers every time it runs.  For this, you can set a ``seed" the random number generator (which normally is seeded by a clock on your computer's processor).  See the documentation for the Random Generator in Numpy for more details.}

So, all you have to do is add a random number to each of your $y$-values and you have a rough estimate of noise.\footnote{In reality, noise often deviates somewhat from a perfect Gaussian distribution due to the presence of noise sources with specific frequencies and filtering effects of data acquisition electronics, but a Gaussian distribution will get the job done most of the time.} Now that you can make a noisy peak, it is time to figure out how to smooth it!


\begin{mdframed}
    \textbf{Random Numbers and Monte Carlo Techniques}
    
    \noindent A number of numerical problems can not be efficiently solved using standard numerical techniques. This is especially true for problems with large numbers of dimensions, like the structure of liquids or proteins, where creating a grid of positions of all atoms leads to too much data and too many calculations to do in a reasonable amount of time on a computer. In some of these cases, so-called Monte Carlo techniques, in which a problem is solved indirectly by statistical analysis of a large number of solutions for individual ``trial" conditions (for structure, a given set of atomic coordinates).
\end{mdframed}


\subsection{Smoothing by moving averages using \texttt{scipy.signal.convolve()}}
\label{Smoothing_by_moving_averages_using_scipy.signal.convolve}

You could compute the moving average explicitly by working out the algebra, but you will instead use a more general approach that employs convolution.  The convolution approach also avoids a tedious feature of algebraically computing the moving average: dealing with edge cases.  For instance, if we are doing a 5-point moving average, then obviously the first point does not have 2 points ahead of it.  Thus, we are forced to deal separately with these points.  This is not a problem, but it is somewhat of a hassle.

Luckily, the convolution-based smoothing process is quite simple to code!  Scipy's \lstinline{signal} \cite{module} contains the \lstinline{covolve()} \cite{function}, which does everything that you need.  It takes two required \cite{argument}s, the data that you would like to convolve, and the function with which you are convolving (which should be normalized, i.e. the array should sum to 1).  There is one important thing to note, however: mathematically, when we convolve an $n$-length \cite{Numpy_array} with an $m$-length convolving function, we will get an $(n+m)$-length array as a result.  This is usually inconvenient, so \lstinline{scipy.signal.convolve()} features a helpful \cite{keyword_argument} \lstinline{mode}, which when set to \lstinline{'same'}, produces a resulting array that is equal in length to the original data.  So, a simple 5-point moving average can be done in a single line (assuming we have already run \lstinline{import scipy.signal as sps} and \lstinline{import numpy as np}:

\begin{pyin}
smoothed_y = sps.convolve(data, np.array([.2, .2, .2, .2, .2]), mode = 'same')
\end{pyin}
or more generically,\footnote{See Section \ref{Generating_arrays_of_all_ones_or_zeros_Numpy} for details on \lstinline{np.ones()}.}
\begin{pyin}
smoothed_y = sps.convolve(data, np.ones(5)/5, mode = 'same')
\end{pyin}

%Here, \lstinline{np.ones()} is a shortcut \cite{function} that creates an \lstinline{ndarray} where each element is 1.  The length of the array is specified by the first positional argument. In this usage we also then we divide by the length of the array to get the desired Numpy arrray.

\subsection{Smoothing using local regression with \texttt{scipy.signal.savgol\_filter()}}
\label{Smoothing_using_local_regression_with_scipy.signal.savgol_filter}

Implementing the Savitzky-Golay filter is straightfoward. The \cite{function} \lstinline{savgol_filter()} has three \cite{positional_argument}s:
\begin{enumerate}
    \item \lstinline{x}: an array holding (confusingly) the $y$-values that you want to smooth.
    \item \lstinline{window_length}: the number of points that you want each polynomial to span.
    \item \lstinline{polyorder}: the order of the polynomial to use for smoothing (i.e., 3 for cubic).
\end{enumerate}
There are a number of \cite{keyword_argument}s that you are unlikely to use.  \lstinline{savgol_filter} returns a \cite{Numpy_array} of the same length as \lstinline{x} containing the smoothed $y$-values.

Now that you have a survey of some smoothing methods, you are ready to write the code to smooth the weak peak and determine the baseline to subtract from the integral.  



\section{Code}
\setcounter{subsection}{-1}  % Start subsections from 0

You decide that you want to use Savitzky-Golay smoothing, but you can choose among any combination of $n$ (polynomial order) and $m$ (number of points).  What to do?  You write a quick test script that will allow you to try different combinations of $n$ and $m$ to see how the data responds.  You sketch out a quick flowchart to organize your thoughts:

\begin{figure}[H]
    \centering
    \includegraphics[]{ChapNarrative/11SmoothCrim/Figures/FlowChartCh10.pdf}
    \caption{Flowchart for code that will read in spectral data and then apply smoothing, so that a baseline can be established.  Much of this solution is the same as in Chapter \ref{10NumInt} and is omitted here for clarity.}
    \label{fig:SmoothCrim:ActivityDiagram}
\end{figure}

\noindent and from this, produce the following code:

\lstinputlisting[language=Python]{../../Distribution/Code by Chapter/Ch10 - Smoothing Data Responsibly/FindAzideSmallPeak.py}

You run the code, it asks you for a csv file with an FTIR spectrum, and you select the first FTIR time point ``azide\_0062.csv" in your ``Azide" folder. It produces the following figure: 

\begin{figure}[h]
    \centering \includegraphics[width=.8\textwidth]{ChapNarrative/11SmoothCrim/Figures/SmoothIR.png}
    \caption{The results of your test code for smoothing.  The small peak that you wanted is centered at 2520 cm$^{-1}$.  You can see that your baseline estimate appears to be reasonable, so you feel confident that the procedure works.}
    \label{fig:SmoothCrim:SmoothIR}
\end{figure}

You try several combinations of $n$ and $m$, finding that including more points in the smoothing (increasing $m$) leads to the small peak and the large peak staring to merge, which means your baseline would be incorrect.  Using fewer points adds more noise.  Changing $n$ to 2 does not make the smoothing much more effective, while changing it to 4 degrades the smoothness, again raising the question of whether the baseline will be correct.  You decide that $n=3$ and $m=100$ seems like a happy medium.

\subsection{Line-by-line explanation}

\begin{footnotesize}
\textbf{Lines}: \textbf{Explanation}
\begin{itemize}
    \item [7--11:] Import \lstinline{numpy} to get \lstinline{genfromtxt()}, \lstinline{scipy.signal} to get \lstinline{savgol_filter()}, and \lstinline{quickScatter()}, \lstinline{quickOpenFilename()}, \lstinline{quickPopupMessage()}, and the \lstinline{symbols} module from \lstinline{codechembook}.
    \item [14\&15:] Get the name of the csv file that we want to smooth.
    \item [18:] Read the data file.
    \item [21:] Smooth by Savitzky-Golay using 100 points and a third-order polynomial.  You can use this script to try out different combinations of these values to see what seems reasonable.
    \item [24--29:] Compute the baseline:
    \begin{enumerate}
        \item [24:] First, set the integration limits for the peak, i.e. where you expect the peak to return to the baseline.  We do this in one line using unpacking.
        \item [25\&26:] Now, determine the $y$-values at these points, much like in the last chapter.
        \item [27--29:] Finally, compute the slope and intercept using the $x$- and $y$-values determined above and calculate the baseline points.
    \end{enumerate}
    \item [32\&33:] We will make the figure using \lstinline{quickScatter()} to save time, but don't want to show the figure yet, so we'll use \lstinline{output = None}.  We make use of the fact that we can pass \lstinline{quickScatter()} a single array as the argument for \lstinline{x} and a list of arrays as the argument for \lstinline{y} and it will figure it out.
    \item [36--38:] There's no easy way to set trace attributes using \lstinline{quickScatter()} (hey, it's supposed to be \textit{quick}!)  So, we'll update them here.    
    \item [41\&42:] The only minor detail here is that we want to zoom in only on this peak by setting the $x$-axis range, but the $y$-axis range will still be the full range of the spectrum.  So, we have to calculate the maximum and minimum \textit{y}-values for just the $x$-values of interest.
    \item [45--49:] Format the figure and show it.    
\end{itemize}
\end{footnotesize}

Looking at the integration code that you already wrote for the azide peak,\footnote{In Chapter \ref{10NumInt}} you realize that it will be very easy to adapt to this smaller intermediate peak.  All you have to do is \lstinline{import scipy.signal as sps}, create a smoothed version of the spectrum before the baseline determination on lines 47-49, use this smoothed spectrum to determine the baseline, and change the limits of integration, and you're finished.  So, you replace lines 46-48 with the following lines of code:

\lstinputlisting[firstline = 47, lastline = 51, firstnumber = 47]{../../Distribution/Code by Chapter/Ch10 - Smoothing Data Responsibly/IntegrateAzideSmallPeak.py}

\noindent then add \lstinline{import scipy.signal as sps} and change the limits of integration to 1790 to 1870 cm$^{-1}$ and add the import statement.  Satisfyingly, the code just works!

\begin{figure}[h]
    \centering \includegraphics[width=.8\textwidth]{ChapNarrative/11SmoothCrim/Figures/IntegrateAzideSmallPeak.png}
    \caption{The spectra that you are analyzing (left), a zoom of the small spectral component in question (center), and its kinetics (right).  It appears to be a byproduct of the reaction, rather than an intermediate.}
    \label{fig:SmoothCrim:IntegrateSmallPeak}
\end{figure}

You take the results to your colleague, who says, ``Well, it looks like it is some small intermediate that forms relatively quickly. This will really change the way that we think about the mechanism! I'll have to go look for it in the NMR and see what it is telling me.  In any case, thanks!  I have some clear data tell my advisor and include in the paper."

\hrule
\input{Exercises/Chapter 10 Smoothing}
\hrule

\end{document}
