\documentclass[../book.tex]{subfiles}
\graphicspath{{\subfix{.}}}
\begin{document}

\chapter{Automating the Plotting of Data}
\label{01OnePlot}


\setcounter{section}{-1}  % Start subsections from 0

\input{prereqs/01OnePlot_prereqs} %prereqs go here

\newpage
\section{Problem}
\setcounter{subsection}{-1}  % Start subsections from 0

You have completed the experiment outlined in the prior chapters and have titration curves.  You have identified the ionic strength that you wish to work at moving forward. However, you would like to verify that the titration involves only two chemical species: the protonated and deprotonated form of the catalyst. You are lucky that your catalyst has a nice UV-vis spectrum, and so you can follow the titration using UV-vis and look for an isosbsetic point. So, you run another experiment in which you obtain the UV-vis spectra at different p$H$ values.

You now have 20 different UV-vis spectra, stored on your computer as csv files. Your advisor has asked that you send them a plot of the \textit{final} spectrum, just so they can verify that the end point makes sense before you work up the rest of the data. While it is nice they are taking an interest in your training, and you appreciate your advisor wants to verify you are not wasting your time working up nonsensical data, it is also true that your advisor is particular about the appearance of plots. It always takes you some time to get Plots formatted correctly in Microsoft Excel, which seems silly for what should be a simple task. In addition, your advisor likes to have images as .png files, but you like to have your own copy as a \cite{scalable_vector_graphic} image (.svg), since you like the fact you can rescale them however you like without a loss in resolution. This disagreement in file preference means you have to run through the image export file twice for each image.  

You mention these frustrations to \first and they just say, "Oh I just have a Python script that automatically generates the plot and then saves it as both a .svg and a .png, formatted correctly. I would be happy to help you make a similar one.  It might take a bit longer to learn this than to make a single plot in Excel, but once you have this script, you can make figures in just a few seconds---if not less." 

You agree, and then sit down to get started. 

\section{Solution}
\setcounter{subsection}{-1}  % Start subsections from 0

''You have data,'' \First says, ''So, you've already done the hard part!  Now you simply need to get this data into Python---what we would call `loading' or `reading' the data.  Once you have done so, you can plot that data, and then save an example of that plot---what we would call 'outputting' the plot. This input/output is what people call I/O, and we will do it using files on your computer, so people call that 'file I/O'.'' 

\subsection{Planning}

Thinking at a more granular level, you can see that the desired outcome can be broken up into five major steps:
\begin{enumerate}

    \item \textit{Identify a file that stores the data to be plotted.}\\In this case, the file is a .\cite{csv} file, which stands for `comma separated values.'  Essentially, the file contains a series of rows, each with a series of values.  Each value is separated by a comma (,). If you open your file in a text editor,\footnote{It is important to note that programs in the Microsoft Office Suite, like Word or Excel, will not render the csv correctly, so it is best to use a text editor instead.} like Notepad (Windows) or Textedit (MacOS), you will see something like the following: 
    \begin{figure}[H]
        \centering
        \includegraphics[width = 2in]{./Figures/csv.png}        
        \caption{Screenshot of a csv file opened up in a text editor. The csv file has data, separated by commas.  This is a common means by which to store data, and in this chapter you will learn how to read such files in Python.}
        \label{fig:OnePlot:CSV}
    \end{figure}
    You can see that each row has two values, separated by a comma. You can also see that the top row has text that function as labels for each column---in this case nm (the unit for wavelengths) and absorbance. We call such text rows `\cite{header}s.' This is information that is not the data, but intended to help the person looking at the file interpret and understand the data. They can be very useful for documenting what the data actually represents.
    
    \hspace{2em} We will point to the location of this file on your computer using its \cite{path}, which is the location on your computer. \First points out that there are different \cite{file_system}s for different operating systems, but that the solution you will use will work with any of these systems, so we don't really need to worry about it too much. 

    \item \textit{Read the file and store the data in \cite{memory}.}\\Ideally, you would want the data as \cite{Numpy_array}s, since you already know how to use these to hold multiple values.  This means making one array that holds the $x$-values (wavelengths) and one array that holds the $y$-values (intensity) of the UV-vis spectrum. 

    \item \textit{Construct the plot}\\You need to turn the data arrays into a graphical representation of their values.  In this case, you would like a line plot, since that is the standard choice for UV-vis spectra. You want to be able to control the presentation of the spectrum, such as the range of data shown and the general appearance of the figure. 

    \item \textit{Display the figure.} \\Once you have the figure constructed, you need display it. This step is ultimately optional, since you can save the plot without showing it.  However, it is nice to have Python display your data, so that you can quickly see what it looks like, rather than needing to open up the saved file. 

    \item \textit{Output the figure to image files}\\ By saving files, you have something you can send to your advisor or something you can use yourself in other applications.  Because you have different preferences, you need to save both .png and .svg files. 
\end{enumerate}

% When thinking through these steps, you also doodle out a diagram of the process, which can serve as a map for the code, and looks like:  
% \begin{center}
%     \includegraphics[width = 4in]{./Figures/01 Planning.png}
% \end{center}

\First says that there are Python \acite{libraries} that handle navigating file structure on your computer, libraries that handle importing data, and libraries that handle the plotting and outputting the plots. Though there are many choices for each of these tasks, we are going to use the following:
\begin{itemize}
    \item [1:] \textit{navigate the file structure using the Pathlib library}\\
    The reason for using this library is that it does a great job \cite{parsing} and manipulating the components of a file path. That is, it is aware of what folders and filenames are, among other things.  This will make it much easier to work with our computer's file structure in Python. Additionally, this library can be used to produce code that automatically works the same on different operating systems with different file systems, like Windows versus MacOS.

    \item [2:] \textit{input the data using the } \lstinline{numpy.genfromtxt()} \cite{function}.\\
    We have already used the Numpy library when making Numpy arrays by specifying them directly. However, the library also provides functions for creating Numpy arrays from data read from files and \lstinline{genfromtxt()} is one such function.

    \item [3--5:] \textit{making, displaying, and saving a data plot using the \cite{Plotly} library.}\\
    There are a large number of libraries that handle making, displays, and saves data visualizations in Python, but this one has three advantages for our work.  First, it automatically makes interactive plots.  Second, the way it composes .svg files is convenient for later modifications using software (such as PowerPoint, Illustrator, or Inkscape).  Third, the way that Plotly makes, combines, and provides access to plot \cite{object}s is very convenient, once you understand how to use it. 
    %The Plotly library also contains the ability to write both \cite{vector_graphic} (e.g., .svg) and \cite{bitmap} images (e.g., .png) to files.
\end{itemize}
However, before we can use all of these correctly, there are a few new concepts that you need to know.




% from <library> import
\subsection{Importing subsets of libraries using \texttt{from}}
\label{Importing_subsets_of_libraries_using_from}

%We have already introduced the idea of \ul{libraries}---things that contain different functionalities for Python. 
``Some \acite{libraries} are huge, and we don't need or want to import them all. Luckily, we don't have to.  Libraries themselves can be broken up into parts.  In Python, there are three major components of libraries: \cite{module}s, \cite{class}es, and \cite{function}s, and each can be imported separately when you don't want to import the whole library. We already talked a little about classes and functions, but modules are really just collections of classes and functions that are topically related.  We can access modules directly using the syntax \lstinline{<library>.<module>}.
Modules can also have submodules, which are accessed by \lstinline{<library>.<module>.<submodule>} and so on... 
Similarly, we can access specific classes and functions using \lstinline{<library>.<class>} and \lstinline{<library>.<function>}, respectively.  When a library is very complex, this can lead to rather lengthy constructions to access a particular component."

``But, if you only want to import a component of a library, you can use a different type of import statement that goes like \lstinline{from <library> import <component>}, where \lstinline{<component>} is the name of the module, class, or function that you want to use.  You can keep going, too, with things like \lstinline{from <library>.<component> import <subcomponent>}."

%A \ul{module} is simply a file that contains code (i.e., a file that ends in .py, like our code from the last chapter), but which is intended to be imported through the \lstinline{import} statements. It is a way to hold a series of functions, definitions, etc. 

%\ul{Classes} can be part of a .py file, and are custom \textbf{data types} that specify both how their data can be created and stored and a set of functions defining how the data can be modified or operated on. Though there is much more that can be said about both \ul{modules} and \ul{classes}, for this book, it is sufficient to understand that \ul{modules} and \ul{classes} can be components of \ul{libraries}. It is also important to understand that:
% \begin{itemize}
%     \item Classes are essentially templates that contain predefined structured ways to organize and store data and functions to operate on that data.
%     \item Classes and modules have their own names, which we use to refer to them when we use the \lstinline{import} statements. 
%     \item If a class or module is part of a library, we can access just the component we want by naming the library, adding a '.' and then the name of the component: <library>.<component>
%     \item Classes and modules can have functions, and you access the functions by adding them, following a '.': <library>.<component>.<function>
%     \item Modules can contain submodules. Modules can contain classes.  Classes can can contain subclasses.\footnote{There is one thing missing here: classes do \textit{not} contain modules} So that you can have: <library>.<component>.<subcomponent>.<subsubcomponent>.<function>, etc.
%     \item When we create a variable based on a class, we call it an \ul{object}, which is a single \ul{instance} of the class.  Using this variable gives access to the data stored in the object and the associated functions defined in the class.  In Python, every \ul{variable} that you create is an instance of a \ul{class}, whether built in such as the numbers or strings introduced in Chapter \ref{} or defined in \ul{libraries} or other Python files.
% \end{itemize}
% ''This last point is important to understand for working with Python.'' \First says. ''So, let's see a quick example.''

% We have already seen one example of an object that is an \ul{instance} of a class.  A \textbf{Numpy array} is an instance of the \lstinline{ndarray}\footnote{\lstinline{ndarray} stands for n-dimensional array} \ul{class} in Python. So, when we create a \textbf{Numpy array}, we create a new object containing the data we define when we create it.  This object also contains the functions defined in the \lstinline{ndarray} \ul{class}, which we can access using the variable name assigned to the object. 

% When we access and use functions that are associated with an \ul{objects} (i.e., <object>.<function>), we call them \ul{methods}, which is a bit of terminology that is good to know when reading about Python online. We will be using \ul{methods} when we make our plot.

``Not only does this save your computer the overhead of having to load the whole library, but it makes your code easier to write and read.  Suppose we wanted to use a function that was defined in a submodule of a module of a library.  If we wrote,"

\begin{pyin}
import <library> as <alias>
\end{pyin}
``then every time we wanted to use the function, we would have to type,"

\begin{pyin}
<alias>.<module>.<submodule>.<function>()
\end{pyin}
``You can image that this would be slow to write and difficult to read.  Instead, we could just write,"

\begin{pyin}
from <library>.<module>.<submodule> import <function>
\end{pyin}
``Now, when we want to use the function, we can just write,"

\begin{pyin}
<function>()
\end{pyin}

``This is so much nicer! And we will use this kind of importing twice in the code we will produce later on."






% So far, when we have imported a \ul{library}, we have imported the \textit{complete} \ul{library}.  But we just learned that \ul{libraries} are broken up into \ul{classes} and \ul{modules}.  It turns out that we can import these individual components if we like. This is done using the construction: \lstinline{from <library> import <component>}

% Interestingly, the \lstinline{<library>} component could itself reference a \ul{class}, for example: \\\lstinline{from <library.component> import <subcomponent>}

% Depending on what the component is, the subcomponent can be a \ul{class}, \ul{module}, or function.

%The reason to use such a construction is two fold.  First, the importing of the smaller section can be faster, especially for very large libraries. Second, it simplifies our usage of the functionality we want.  As an example of something we will see in the code below, the \lstinline{make\_subplots()} function is part of the \lstinline{subplots} class in the \lstinline{Plotly} library.  If we use \lstinline{import plotly} we could access this function using \lstinline{plotly.subplots.make\_subplots()}, however, if we use \lstinline{from plotly.subplots import make\_subplots}, then we can directly use \lstinline{make\_subplots()}


% \subsection{Customizing the behavior of functions using keyword arguments}
% \label{Sec:N:01OnePlot:Keywords}

% We have already used several functions in our exploration of Python. We have seen examples where we pass functions zero, one, or more \ul{arguments} to operate on.  But in each case, we have passed the \ul{argument} by simply stating the \ul{argument(s)}, separated by commas if there are more than one, for example, \lstinline{print("Hi", name)}.  Such \ul{arguments} are known as \textbf{positional arguments} because they are required to be there and their order as written in the code defining the function matters.  However, there is another type of argument known as \textbf{keyword arguments}.  These arguments are passed using the syntax: \lstinline{<keyword> = <object>}. We will see many examples using such \ul{arguments} below. \textbf{Keyword arguments} are typically optional, and if you don't include them, they will be set to some default value specified in the function definition.

% Let's return to the \lstinline{round()} function that we saw last chapter. This worked well for single numbers, but if we want to round all elements in a Numpy array, then we need to use a different function: \lstinline{numpy.round()}. This also accepts two arguments, and we can use it the same way we used round: with two \ul{positional arguments}.  However, we can also specify the number of decimal places we want using a \ul{keyword}: \lstinline{decimals}. This specifies the decimal place to round to.  The default for this is 0, so if you omit it, then implicitly you are writing \lstinline{np.round(<number>, decimals = 0)} without having to type it all the way out. Instead, you could specify \lstinline{decimals = 2} to round to the nearest hundredths digit, or \lstinline{decimals = -2} to round to the nearest hundred.

% \begin{pyin}
% np.round(127.652)
% \end{pyin}
% \begin{pyout}
% 128.0
% \end{pyout}
% \begin{pyin}
% np.round(127.652, decimals = 2)
% \end{pyin}
% \begin{pyout}
% 127.65
% \end{pyout}
% \begin{pyin}
% np.round(127.652, decimals = -1)
% \end{pyin}
% \begin{pyout}
% 130.0
% \end{pyout}

% Whether you specify the number of decimals using the \ul{positional argument} or \ul{keyword argument} approach, the argument goes to the same place in the function. Many functions that you will encounter have several keyword arguments.  If you only want to use some of them, then you will have to use the keyword approach, since the positional approach does not tell Python which keywords you intend to pass arguments to.



\subsection{Performing multiple assignment operations in one line by unpacking}
\label{Performing_multiple_assignment_operations_in_one_line_by_unpacking}

``We have been using the assignment operator (\lstinline{=}) in an intuitive way: there has been a single \cite{variable} name on the left, and a single \cite{expression} on the right. However, Python allows more complex usage of the assignment \cite{operator}. For instance, you can have more than one variable on the left, as long as there is a matching number of values on the right:"
\begin{pyin}
one, two = 1, 2
\end{pyin}
This will create a variable \lstinline{one} with the value 1, and a variable \lstinline{two} with the value 2.

``You can also take a single object that holds multiple values and pass it to the same number of variables, such as:"
\begin{pyin}
three, four = np.array([3, 4])
\end{pyin}
``This will create two new variables: \lstinline{three} with a value of 3, and \lstinline{four} with a value of 4. These behaviors are examples of \acite{unpacking}."

\subsection{Representing answers to yes/no questions using booleans}
\label{Representing_answers_to_yes/no_questions_using_booleans}

``So far, we have seen a few different \cite{data_type}s, such as \cite{string}s or \cite{Numpy_array}s. We are going to need to learn another now: \cite{boolean}.  This is a data type with only two possible values: \lstinline{True} and \lstinline{False}, which functionally behave a lot like `yes' and `no.'  These values are often used to tell Python when you want something done or not, or to test whether two things are equal. In Python, capitalization matters, so \lstinline{TRUE} and \lstinline{true} are not valid values for booleans. The first letter (and only the first letter) of these values \textit{must} be capitalized."


\subsection{Exploiting built-in methods to manipulate data stored in objects}
\label{Exploiting_built-in_methods_to_manipulate_data_stored_in_objects}

``One of the nice things about Python is that pretty much everything (\cite{variable}s, \cite{function}s, etc.) is an \cite{object}, which makes it so that you have to write much less code yourself.  This is because the object 
%\footnote{More precisely, it holds the address of an object in \cite{memory}. For now, this distinction is not really needed, but it will become an important distinction much later, when we get to the Python concept of cite{lists}. If you want more information, you can read the technical Chapter \ref{DataTypes}}
contains not just the data that you assign to it, but also a bunch of code that other people wrote that helps you transform the data into something more useful.  This means that the object lets you use bunch of complex code without having to write it!"

``Consider an analogy: you could build your own computer. You could figure out what parts you need, make sure that they're all compatible, buy them, put them together, install an operating system, install software, update everything, and you'd have a working computer but you would have also spent a lot of time on it.  Alternatively, you could just buy a computer from a company.  You don't have to know what is inside the computer, you just have to turn it on and use it and not worry about all the work that went into making it."

``When we assigned a \cite{Numpy_array} to a variable, what happened is that the computer made an \cite{instance} of the \cite{class} \lstinline{ndarray} (this class is the official \cite{data_type} of a Numpy array).  Don't worry about what a class is for now, just know that it is the collection of code describing how an object works---in this case stored in the Numpy \cite{library}.  You will probably never have to even look at the code describing this class, much less understand in detail how it works.  But that code describes two things you need to know about: \cite{method}s and \cite{attribute}s.  Methods are \cite{function}s that are built into the object,  have access to the data in the object, and act on that data.  Attributes are variables within the object that provide information about the object.''

``Let's just worry about methods for now."

``Methods are not much different than any other function in Python, like \lstinline{print()}, except that they live within objects and can access data in the object without your intervention.  Since a method is not usable without the object in which it resides, you use it a little differently than the functions we have used so far.  Instead of \lstinline{<function>()} like we've already seen, you call a method by first identifying the object containing it, and then giving the method \cite{call}, separated by a period. This construction looks like: \lstinline{<object>.<method>()}.  That way, if we have multiple instances of the same object with different names, Python knows which one we want to operate on.  It also means that we do not have to pass the object variable to the method as an argument, since it's identity is built in.  Beyond this, \cite{argument}s and \cite{return}s work pretty similarly to any other function."

You open the Spyder IDE,\footnote{This was introduced at the end of the last chapter.} go to the console window, and type the following:

\begin{pyin}
import numpy as np
\end{pyin}
\begin{pyin}
testArray = np.array([1, 5, 3])
\end{pyin}
``You now have created an \lstinline{ndarray} \cite{object}.'' \First continues, ``In the \lstinline{ndarray} class definition, there are functions that can report statistics of your array: \lstinline{mean()}, \lstinline{std()}, \lstinline{sum()} and so on.  Normally, we might think that we need to use a construction like \lstinline{numpy.ndarray.sum(testArray)} to access these functions.  However, since our \lstinline{testArray} is an instance of the \lstinline{ndarray} class and contains these functions, we can just use \lstinline{testArray.sum()}"

\begin{pyin}
testArray.sum()
\end{pyin}
\begin{pyout}
9
\end{pyout}
You can see that using the \lstinline{sum()} method of the Numpy array class returned the sum of all the elements in the Numpy array... all without you having to know how the method worked.  

\subsection{Describing the location of files or folders on your computer using Pathlib}
\label{Describing_the_location_of_files_or_folders_on_your_computer_using_Pathlib}

``Files in your computer are organized in a hierarchy of containers intended to make it easy to logically group them and indicate their relationships.  The typical representation of this hierarchy is that files are grouped into folders (also known as \acite{directories}).  Any folder can also be in another folder, and so on.  Eventually, you have one primary folder that is not contained in any other folder, but contains all other folders, subfolders, sub-subfolders, etc. This primary folder is called the \cite{root} folder, and is typically represented by \texttt{C:\\} on Windows computers and by \texttt{/} on Macs and Linux computers."

``We might describe the location of a file in the hierarchy as <root><folder1><folder2><folder3><file>.  We call this the \cite{absolute_path}.  The collection of files and folders is called the \cite{file_system} and the exact organization (names and locations of the files and folders) that it employs is called the \cite{file_structure}."\footnote{See Chapter \ref{Files} for more details.}

\begin{mdframed}
\textbf{**KNOWLEDGE CHECK**: Do you understand the computer's file system?}\\
It is \textit{absolutely critical} that you understand file structures when coding. You will be unable to write effective code if you do not know (or learn how) to navigate a file structure. While we provide tools to try to help make this easy, you can not avoid it all together. If \textit{any} the following statements describes you, we \textit{strongly} recommend you go read the technical chapter on files (Chapter \ref{Files}) or refer to the tutorials posted on the companion website: \website.
\begin{itemize}
    \item You do not know what a folder on a computer is.
    \item You do not know what a file is on a computer is.
    \item You do not know what a file extension is.
    \item You do not know what a path is.
    \item You cannot tell us what the path to your documents folder is.
    \item You do not know how to determine the number of folders that lie between your documents folder and downloads folder.
    \item You do not know what a directory is.
\end{itemize}
We also want to stress that it is ok if you don't \textit{yet} know these things---we can help you learn.  But you \textit{will} need to learn them. 
\end{mdframed}

\begin{wrapfigure}{r}{0.5\textwidth}
    \begin{mdframed}
        \textbf{Why Pathlib?}
        
        \noindent The `original' way to handle files and file structure in Python was using a \cite{library} called \lstinline{os}, and which is still included in Python (same as Pathlib). However, Pathlib adds many additional capabilities for handling \cite{file_structure} that make it worth using instead. For instance, Pathlib is aware of what a file \cite{extension} is, as well as the file hierarchy. For this reason, we will only use \lstinline{os} when it can perform a function that Pathlib cannot. 
    \end{mdframed}
\end{wrapfigure}

``Python has a built in \cite{library} for exploring and creating paths called \cite{Pathlib}. It provides a function \lstinline{Path()}, which accepts a \cite{string} \cite{argument} that specifies a path on your computer and converts it into an \cite{object} that Python can work with (a Pathlib Path object)---recognizing things like folder, files, extensions, etc. More details can be found in Chapter \ref{pathlib}. However, for the solution to our workflow problem we need to understand how to change the \cite{extension} of a file using the \lstinline{with_suffix()} \textbf{method} of \lstinline{pathlib}."


``Now, extensions are the way in which computers know how to treat files, and they are short codes that follow the filename. For instance, we have been considering \cite{csv} files.  The "csv" that follows the dot is the extension. Likewise, if you have a modern excel file, you might notice that it ends in .xlsx, which is the extension associated with Microsoft Excel. Microsoft Word documents use the extension "docx." In Pathlib, extensions are termed `suffix,' and the \lstinline{with_suffix()} method allows us to control the suffix (extension) associated with a file."

``We can see this behavior by creating a path object, with extension, and then changing this extension."

\begin{pyin}
from pathlib import Path
\end{pyin}
\begin{pyin}
new_path = Path('C:/Users/me/Documents/Research/fakeFile.csv')
\end{pyin}

\noindent``You see that this is a Windows-style path with `C:' as the root, but we used forward slashes instead of backslashes.  This is because the backslash character in strings has special significance: remember that \lstinline{\n} is the newline character, and Python knows that it isn't just the letter n because of the backslash.  So, pathlib knows about this, and can accept forward slashes instead, or if you prefer, two backslashes in a row to signify a backslash.  If we then enter this value, it will return a path object:"
\begin{pyin}
new_path
\end{pyin}
\begin{pyout}
WindowsPath('C:/Users/me/Documents/Research/fakeFile.csv')
\end{pyout}
``An important thing to remember is that the path is \textit{not} the file or folder. Creating a path does not create a file or folder.  And just because you write a path doesn't mean there is a corresponding file or folder.  The path is like directions to a location: it will get you to a place, but there may be nothing there when you arrive. 
You can also write directions to fictitious places---a fact regularly exploited by authors and screenwriters. 
Paths just specify a location in your file structure.  Also, if you are on a Mac, then ``WindowsPath" is replaced by "PosixPath".  Both are indicating that this is a path object."

``We can then take this path and change the extension, using the \lstinline{with_suffix()} method:"
\begin{pyin}
new_path.with_suffix('docx')
\end{pyin}
\begin{pyout}
WindowsPath('C:/Users/me/Documents/Research/fakeFile.docx')
\end{pyout}
``Here you can see everything about the path is the same, but it ends with ``docx'' rather than ``csv''."

``Now there are two very important things to note about what just happened:
\begin{enumerate}
    \item Just because a file has an extension, doesn't mean that a program can open it. If you have a random file that ends in ``docx'' and try to open it, your computer will try to open it in Microsoft Word (because the extension is telling it to), but unless the file is formatted correctly, Microsoft Word will fail to make sense of it (or might not even open it). 
    
    \item The \lstinline{with_suffix()} is a method of a path object.  You cannot just use it on any thing you want.  For instance, if you try to use this on a string, you will get an error:"
    \begin{pyin}
    'C:/Users/me/Documents/Research/fakeFile.csv'.with_suffix("docx")
    \end{pyin}
    \begin{pyout}
    AttributeError: 'str' object has no attribute 'with_suffix'
    \end{pyout}
    ``Where this error message is telling you that strings do not have this capability."
\end{enumerate}





\subsection{Making paths convenient and portable by specifying relative paths}
\label{Making_paths_convenient_and_portable_by_specifying_relative_paths}

``Paths can get very long and annoying, because your computer can have a lot of folders," \first says.  ``Luckily, there's a shortcut that makes it much easier. You can specify a different folder in which to read or write files one of two ways.  First, you can explicitly type out the full path to where you want it saved---often called the \cite{absolute_path}  You can also type out the path \textit{relative} to the \cite{working_directory}.  The working directory is the place where the computer is currently active.  You can think of it like this: if you were to create a file without specifying any location, the place it would be created is the working directory."

``Specifying the location of a folder or file relative to another is known as a \cite{relative_path} and it uses two symbols: `.' and `..'.  The symbol '.' represents the working directory and '..' indicates the directory containing the working directory. Imagine that our current working directory was \lstinline{<folder2>} above.  Then \lstinline{./<folder3><file>} would refer to the file of interest, while \lstinline{..} would refer to \lstinline{<folder1>}.\footnote{You can read more about relative paths in Chapter \ref{Files}.} If you already have absolute paths, then \cite{Pathlib} makes it easy to automatically find the relative path between them.  For example, let's say we have a Python script in directory \textbf{A} on our computer, and a data file in directory \textbf{B}.  We can create absolute paths from \textbf{A} to \textbf{B} by using the \lstinline{.relative_to()} \cite{method} of the Path \cite{class} like this:"
\begin{pyin}
relative_path = pathB.relative_to(pathA, walk_up = True)
\end{pyin}
``This yields a new \lstinline{Path} object that points from your Python script to the data file, which you can use in your code to avoid having to type out the full path. The \lstinline{walk_up} keyword allows Python to navigate your computer's file structure more completely.\footnote{Basically, the default behavior is to explore only up to the `base' directory given (i.e., a place to start), but setting \lstinline{walk_up = True}, Python can go beyond that folder. } Note that \lstinline{pathA} and \lstinline{pathB} must both be absolute paths for this to work! You can use the \lstinline{.resolve()} method to get an absolute path from a relative path:"
\begin{pyin}
abs_path = relative_path.resolve()
\end{pyin}
``In this case, we are just recreating the absolute path to the data file.  A possible usage of \lstinline{.resolve()} is as an alternate way to get an absolute path to the current working directory or one close by: \lstinline{path_above = Path('..').resolve()} gets the absolute path of the folder containing the folder we are currently working in."

\subsection{Adding \texttt{codechembook} to your Python installation}
\label{Adding_codechembook_to_your_Python_installation}

``Now, it is important to understand how the \cite{file_system} works, and how to specify locations in the file system by writing \cite{path}s," \first says, ``but it can also be tedious at times.  We are a bit more familiar with finding files using our computers file explorer. Because of this, I wrote a bit of code that I tend to use when I simply want to identify a file to open. This saves me time and effort when trying to specify a specific file location.  I put this code in my own \cite{library} named \cite{codechembook}, so that I can call the functions in that library just like any other program. Since I thought others might find some use from it, I have also made it so that you can install it on your computer. %However, since this is not an official Conda package, you need to open up the anaconda prompt and type: \lstinline{conda install -c conda-forge codechembook}"
\footnote{You learned how to do this in the last chapter, but if you need help, you can refer to our website (\website) for detailed instructions.}

``Nothing we are going to do will \textit{require} that you use this library, but I think you fill find it adds a great deal of functionality for working scientists, while also saving time and reducing frustration---and isn't that why you are learning to code in the first place?"

\begin{mdframed}
    \textbf{Who, or what, is codechembook for?}
    
    \noindent codechembook is a Python library written and maintained by the authors of this book. It originally started as a set of shared tools we used to help us in our own research workflows. For instance, we have modules that automatically format data visualizations for common chemistry journals. When writing this book, we thought we could share these tools with a wider audience.  Additionally, we wrote a few tools specifically for this book, which help navigate some of the input/output tasks that might otherwise be frustrating to people new to coding. It is important to note that using codechembook is not strictly required to do any of the work in this book---we have ensured that we provide sufficient information to accomplish any of the tasks without it.  However, we also hope you will find this library useful in accomplishing the work in this book, and beyond!
\end{mdframed}



\subsection{Locating files and folders graphically with \texttt{codechembook.quickTools.quickOpenFilename()}}
\label{Locating_files_and_folders_graphically_with_codechembook.quickTools.quickOpenFilename}

``Now that we have \cite{codechembook} installed, we can access its functionality. In this case, we are going to use a function that I created to identify a single file to open: \lstinline{quickOpenFilename()}. We can see how this works using the following:"
\begin{pyin}
from codechembook.quickTools import quickOpenFilename
\end{pyin}
\begin{pyin}
quickOpenFilename()
\end{pyin}
``This will open up a native file explorer that you can use to select any file on your computer.  What it will \cite{return} is a \cite{path} \cite{object} of the \cite{Pathlib} \cite{library}, which contains the path to the file you selected.  Basically we are getting the filename for the file we wish to open. We can take this output and assign to a variable for later use, or pass it to another function, or anything else we do with the output of an \cite{expression} in Python. Also, because this function returns a Pathlib path object, we can work with it using the Pathlib library. For instance, we can use the \lstinline{with_suffix()} method on this object.''

``Another thing to be aware of... even though \lstinline{quickOpenFilename()} returns a pathlib object does not mean that the script automatically has access to pathlib.Path. All this means is that you will get back a Path object. You can operate on that object using Path methods, because the instructions for how to do so are contained in the object.  But if you want to use funtions from pathlib.Path, elsewhere in the script, you will still need import it as normal. 

%``Another thing to note... because the function \lstinline{quickOpenFilename()} is returning a Pathlib path object, this  means that this function must have loaded \lstinline{Path} from Pathlib.  Thus, if we use the \lstinline{quickOpenFilename()}, we may not need to also load \lstinline{Path} from pathlib. If you want to make a new \lstinline{Path()} object, you will, but if you just want to use the methods for the object that is returned by \lstinline{quickOpenFilename()}, then you don't because they are automatically part of that object.''

``At any rate, the \lstinline{quickOpenFilename()} function is a way to get the path to a \textit{single} file on your computer. The next time we meet, I can show you how to get the path to \textit{multiple} files at once, but this is sufficient to solve our current workflow problem.''

``One more thing!  If you click cancel in the file dialog, \lstinline{quickOpenFilename()} will return the value \lstinline{None}, which is a special data type that represents literally nothing.\footnote{For more information, see Chapter \ref{DataTypes}.}  Your codes will mostly be expecting strings or Path objects, so getting \lstinline{None} will probably cause an error and the \cite{Python_interpreter} will stop executing your code.  For now, we'll just live with this possibility, but later we'll talk about how to handle these situations gracefully."\footnote{See Section \ref{Handling_errors_without_crashing_using_try-except-finally_}.}



\subsection{Accessing data in structured text files using Numpy}
\label{Accessing_data_in_structured_text_files_using_Numpy}

``Now, we need to talk about reading data files,'' \first says. ``Structured text files are a convenient way to store human-readable numerical data.  You are already familiar with the generic term ``\cite{csv} file."  This is a very common structured text data file format, in which data (often, but not exclusively, numerical) is presented in rows and columns.  Strictly speaking, for comma-separated value files, each column is \cite{delimited} by commas.  Delimiting is necessary because computers can not easily tell the difference between numbers, white space, and other characters, and thus do not necessarily know when one value ends and the next one begins.  With a comma \cite{delimiter}, a program can determine that one value should exist between each pair of commas (as well as between the beginning of a line and the first comma, and between the last comma and the end of a line), which makes it much easier to parse the data in the file.  Other common delimiters are tab characters `\lstinline{\t}' and spaces.''

``Because this manner of representing is so robust, standard \cite{function}s can be written that automatically load data from structured text files such as a csv file into \cite{variable}s in your code.  One very convenient one is built into \cite{Numpy}, the function \lstinline{genfromtxt()}.  This function has only one required \cite{positional_argument}, the file path (as a string), but you will generally want to pass several additional optional \cite{keyword_argument}s, which are:
\begin{itemize}
    \item \lstinline{delimiter}: the delimiter character (as a string).  Common values are \lstinline{','} for commas, \lstinline{' '} for spaces, and \lstinline{'\t'} for tabs.
    
    \item \lstinline{skip_header}: the number of rows to skip before reading in values.  Often, these files have one or more rows with information in them other than the values that you want to load into arrays.  For instance, the headings of each row (wavelength, absorbance for a UV/vis spectrum).  In this case, you would want to skip one row, since you can't load text into your $x$ and $y$ data arrays.  You can optionally attempt to capture these headings into separate variables using the \lstinline{names} keyword. Headers can sometimes contain several additional rows of metadata, so make sure you look at your data files before loading them.
    
    \item \lstinline{unpack}: by default, this function returns all of the data as a 2D numpy array in which the outer array holds each row and the inner array holds the individual values in a row.  It can save you some time to unpack the data and having multiple variables on the left hand side of the assignment operator to accept each column separately.  You can do this by setting \lstinline{unpack = True}.
\end{itemize}
Of course, \lstinline{genfromtxt()} has a few useful behaviors and many additional options.  Most importantly, it gracefully handles missing values from files, allowing you to specify what goes in their place. More can be found using Python's \lstinline{help()} function.''




\subsection{Making appealing and interactive plots using Plotly}
\label{Making_appealing_and_interactive_plots_using_Plotly}

``Now, we are ready to talk about plotting! There are an abundance of \acite{libraries} in Python that can be used to plot data. We are going to be using the \cite{Plotly} library. In Plotly, the instructions for data visualizations are held in a \cite{figure_object}. These Figure objects have two basic components: data and layout. The data component contains information that includes the data points you are plotting, the type of representation that you wish to use (e.g., scatter, bar, etc.), how these representations should be displayed (e.g., color, shape, etc.), and where the representation should be drawn (e.g., if there are multiple subplots, which one to add the representation to). The layout object contains information about how the axes are formatted, the color of the plot background, any annotations you wish to include, and so on. We will explain elements of both the data and layout components of the Figure object as we need them, but for now, it is enough to understand that the basic usage of Plotly is to create a Figure object, specify the representations you wish to include, and then format the elements of your plot.  Finally, you can display this plot in a few different ways, but we will largely either be showing static images in the Spyder IDE or making interactive plots that open in web browsers. The interactive functionality is a particular strength of Plotly, and provides a easy means for exploration of your data. All of these steps will be shown in the code at the end of the chapter.''\footnote{Many details on how to understand and use Plotly can be found in the technical chapter devoted to this library (Chapter \ref{ch:plotly}). At this point, it is probably too early to read this chapter in detail, but it is worth keeping in mind that this resource exists. }

"Ok, we are now in a position to create the code that solves our workflow problem!"

\begin{mdframed}
\textbf{Why not use Matplotlib?}\\
If you have any prior experience with working up data in Python, you have likely encountered Matplotlib before.  If you are new to Python, you will very quickly encounter Matplotlib in examples and discussions online. It is probably the most popular plotting library in Python. However, we prefer Plotly over Matplotlib for five reasons:  (1) Plotly creates prettier images, (2) Plotly creates better interactive plots by default, (3) interactive plots in Plotly are easily sharable with others, (4) Plotly provides better access to the elements of the figure, and (5) Plotly figures persist and are accessible even after showing them.  These last two points means that Plotly is easier to use in the \cite{object_oriented} approach to coding that we use in this book. While there is nothing wrong with using Matplotlib, we honestly believe that you will be better off in the long run using Plotly.  
\end{mdframed}

\begin{mdframed}
\textbf{Why not use Plotly express?}\\
As you learn to use Plotly, you will encounter Plotly express in many examples. We do not use it here, because we are interested in presenting you with a general tool. Plotly express is good for making graphs that have single plotting areas, but as soon as you want to have multiple subplots (which we will do as soon as we start fitting data), you will find that Plotly express makes this harder than our approach.  Our approach, while not as well represented on the internet, is just as simple as Plotly express, while being more flexible. 
\end{mdframed}




\subsection{Outlining your solution before coding using comments}
\label{Outlining_your_solution_before_coding_using_comments}

You sit down and open up the \hyperlink{https://www.spyder-ide.org/}{Spyder IDE} to start coding. \First points out that it is going to be very important to \cite{document} your code with \cite{comment}s, and so you might as well plan things out using them.  They suggest that you can use the outline you produced at the very start as a basic map. Doing so, you arrive at the following initial structure:

\begin{lstlisting}{}
'''
V0.1 A program to plot uv-vis spectra from csv
by: you
'''
# specify the name and place for data

# import data to plot as a numpy array

# construct the plot

# display the plot
\end{lstlisting}

You also sketch out a flowchart of what you want to do:
\begin{figure}[h]
    \centering
    \includegraphics[]{ChapNarrative/01OnePlot/Figures/FlowChartCh2.pdf}
    \caption{Flowchart of the solution to plotting a single spectrum, with data read from a csv file. }
    \label{fig:OnePlot:ActionDiagram}
\end{figure}

Now, all that remains is to provide the code that carries out what your comments describe. You pass your laptop over to \first, who produces the following code:

% Insert code

\section{Code}
\setcounter{subsection}{-1}  % Start subsections from 0

\lstinputlisting[language=Python]{../../Distribution/Code by Chapter/Ch02 - Automating the Plotting of Data/PlotCatalystUVvis.py}

You run the script in Spyder, and a file dialog box opens.  You navigate to the folder `Research/Catalyst/Titration/UVvis' to select the file 0.999.csv, and after about a second, most of which is due to loading the Plotly library.\footnote{This delay will only happen the first time you run this code in a session. Once the library has been loaded once, it won't need to happen again, unless you restart the session.} However, you soon see a figure that looks something like that shown in Figure \ref{fig:OnePlot:Spectrum}.
\begin{figure}[H]
    \centering
    \includegraphics[width = 5in]{./Figures/0.999.png} 
    \caption{Plot of a UV-visible spectrum, as generated using the Plotly library and data read from the file 0.999.csv. The code in this chapter will allow you to make such plots automatically by pointing to the file of interest.  The raw data and code for it can be found on \website.}
    \label{fig:OnePlot:Spectrum}
\end{figure}
The same basic figure opens in a browser window, is displayed in the 'plots' tab of the Spyder IDE, and is saved as both 0.999.png and 0.999.svg in the same folder as the csv file.\footnote{If you are confused on how you might open up a .svg image, most internet browsers (i.e., Microsoft Edge, Google Chrome, etc.) will display them.  Also, programs like Adobe Illustrator, Affinity Designer, or Inkscape can both open and edit them.} The image displayed above was taken from the Spyder plot window. The formatting of the saved images---especially the number of tick marks and size of the font---might look different, but we will learn how to specify the exact format the figures, as we move through the book.  You can also read the technical Chapter \ref{ch:plotly} for more infomration. 


\subsection{Line-by-line explanation}

\begin{footnotesize}
\textbf{Lines}: \textbf{Explanation}
\begin{itemize}
    \item [1--6:] We start the file with a multi-line comment that states the purpose of the code, requirements for using the code, who wrote the code (and how to contact them), and then information on the versions.  This is sometimes known as a `docstring' and is used to help others understand what is going on, and (perhaps more importantly) help you remember what you were doing if you ever return to the script. The version numbers and date, in particular, are very excellent things to have. 

    \item [7:] We import \lstinline{numpy}. We will use a function from this \cite{library} in order to read our .\cite{csv} file. Because we don't want to type `numpy' all the time, we assign it the \cite{alias} `np.' This allows us to refer to \lstinline{numpy} using simply \lstinline{np}. 
    
    \item [8:] We want to use the \lstinline{make_subplots()} function from \lstinline{plotly.subplots} \cite{module}, but we don't need the entire Plotly library. We write \lstinline{from <library>.<module> import <part>} to get only what we want.  The \lstinline{make_subplots()} function creates an empty \cite{instance} of a Plotly figure \cite{object}. Once we have this object, we can add data to it and format it as we like. 

    \item [9:] We import a specific function in the \lstinline{codechembook} library, in this case the function \lstinline{quickOpenFilename()} from the \lstinline{quickTools} module. This function will open up a native file explorer, in which the user can select a file of interest, and then the function will return a pathlib Path object that points to that file.  

    \item [12:] Here we get the path to the file of interest, using the \lstinline{quickOpenFilename()} function, which returns a pathlib Path object. This path is then stored in a \cite{variable}. It is convenient to have a Path object because it is easier to parse the components of the path, such as getting folder, filename only, extension, etc. This is a functionality we will use at the end of the script (lines 37 and 38). There are other reasons to use \lstinline{Path} \textbf{objects}, which can be read about in technical Chapter \ref{pathlib}. 

    \item [15--18:] These lines call the \lstinline{genfromtxt()} \lstinline{function} from \lstinline{numpy} to read the csv data file and generate Numpy arrays from the data in the file. 
    The \lstinline{genfromtxt()} function of \lstinline{numpy} can return the data read in a row-by-row manner or a column-by-columns (we will use it by columns). \lstinline{genfromtxt()} will return a Numpy array that contains a number of other Numpy arrays equal to the number of columns we tell it to read. There are two columns in the data file, so we will get back a Numpy array containing two \lstinline{numpy arrays}. We \cite{unpack} these two arrays into two variables on the left hand side of the assignment \textbf{operator} (=). \\
    You will also notice that our usage of \lstinline{genfromtxt()} covers multiple lines of code. When we call a function, we have to place \lstinline{arguments} for that function between parentheses. In Python, we can put the \lstinline{arguments} on different lines, as long as they are between parentheses. When functions have lots of \lstinline{arguments}, we will do this in order to achieve greater clarity and readability.\\ 
    To understand this call of \lstinline{genfromtxt()}, we can look at the usage of each argument. 
    \begin{itemize}
        \item [15:] a file name as a \cite{positional_argument}, stored as a \lstinline{Path} \textbf{object} in the variable \lstinline{data_file}. The \lstinline{genfromtxt()} function will read this file and pull out the values in it. 
        
        \item [16:] We have to let \lstinline{genfromtxt()} know what character separates the data values.  This separator is called a \cite{delimiter} and, for our csv file, it is a comma. We supply this separator as a \cite{string} (',') to the keyword argument \lstinline{delimiter}.

        \item [17:] The data file we are using has \cite{header}s. In this case, the first row are labels of the columns. The \lstinline{genfromtxt()} function has no idea that the data doesn't start until the second row, but we can use the \lstinline{skip_header} keyword to tell it how many rows to skip before treating the rest as data.  Remember that Python starts counting at 0, so you can think of this number as either the number of rows to skip or the first row in which there is data. 

        \item [18:] Finally, the default function of \lstinline{genfromtxt()} is to assign each row to a \lstinline{numpy array}. For our data, which has 911 rows, this would result in 911 \lstinline{arrays}, each storing two values (each row has 2 entries).  However, we want to have the columns in our file each in an array. Using the keyword \lstinline{unpack} and setting this equal to the boolean \lstinline{True} will give us the data organized as two columns. \\
        We end the function \cite{call} with a closing parentheses.  This closing parentheses needs to match the opening one (line 15) and indicates to Python that we have provided all the arguments we are going to provide for this function. 
    \end{itemize}

    \item [21:] We use the \lstinline{make_subplots()} function to create a Figure object.  There are actually several ways to make a Figure object in Plotly, but this approach will create an object that can hold many plots if we like.  For this reason, it is the most general approach. Here we also see that some functions (if they have no positional arguments) can be used \textit{without} supplying an argument.  When you do this, they will just perform some default action.  In this case, \lstinline{make_subplots} returns a blank Figure object assigned to the variable \lstinline{UVvis} that holds a single subplot.

    \item [22-24:] We use the \lstinline{add_scatter()} method of the Plotly Figure object to create another object known as a `trace' (Plotly's term for a graphical representation of data) and add the data to the Figure object. We know this is a \cite{method} of the Figure object, because it follows the form: \lstinline{<Figure object>.add_scatter()} This trace is an \cite{instance} of the scatter plot \cite{class}. 
    
    \begin{itemize}
        \item [22:] The \lstinline{add_scatter()} method requires both $x$ and $y$ values that we specify using the associated \cite{keyword_arguments} \lstinline{x} and \lstinline{y}.
    
        \item [23:] We use the keyword argument \lstinline{mode} to tell Python that we want to plot using only lines (as opposed to using markers as well). 

        \item [24:] We use the keyword argument \lstinline{showlegend} to tell Plotly we don't want this trace included in a legend, setting it equal to the boolean \lstinline{False}. We use a closing parentheses to indicate we have completed the \ul{function call}.\\
        We have used just a few options for the \lstinline{add_scatter()} method. A more complete description of the options can be found in the technical Chapter \ref{ch:plotly}. 
    \end{itemize}

    \item [27-29:] We use methods of the Figure class to control the formatting (appearance) of the Plotly figure. 

    \begin{itemize}
        \item [27:] Using \lstinline{update_yaxes()}, we make changes to \textit{all} $y$-axes in our figure, though in this case, we have only a single $y$ axis. Later in the book, we will see how to specify only some of the y-axes, when we have more. This method accepts a large number of keyword arguments, but here we only use one: \lstinline{title}, which accepts a \lstinline{string} that will be used as the title of the $y$-axes. The default behavior of the axes in Plotly is to show a range slightly larger than the full range of the data.  So, we don't need to specify a range to show, if we are ok with seeing the full range. 
    
        \item [28:] Similar to the $y$-axes, this line formats the $x$-axes, this time using the \lstinline{update_xaxes()} method.  Here we specify the title, but also add the use of another keyword argument: \lstinline{range}, which is used to specify the limits of the axes.  Since we don't need to show all of the ultraviolet data shown, we want to only use part of the full range of the data.  We specify the start and stopping point for the axes using two numbers, between square brackets, separated by commas. 

        \item [29:] We define the appearance of the figure by using the \lstinline{update_layout()} method of the Figure class.   Again there are many possible keyword arguments, but we only use one of them: \lstinline{template}.  We pass a string containing the name of a pre-defined template in Plotly: \lstinline{simple_white}. This tells Plotly to apply the settings of this template.  This contains information regarding the tick marks, font, font size, background color, and much more. We will eventually discuss how to create your own templates, but for now, this one will work well. 
    \end{itemize}

    \item [32:] Using the Plotly Figure \underline{method} \lstinline{show()} we can display our composed, formatted, figure. We provide this function a positional argument that is a string with two words, joined by a plus sign.  Separately, 'browser' and 'png' could be used with \lstinline{show()}.  If we only used 'browser', then the figure would be opened in your computers default internet browser (e.g., Chrome, Firefox, etc.) and you could interact with it, by zooming in or hovering over data. If you only used 'png' then (assuming you are using the Spyder \cite{IDE}) you would get a static image in the png format, displayed in your Spyder IDE plot window. By using both, and joining them with a '+' sign, we are telling Plotly to do both. This approach to combining multiple options is somewhat unique to Plotly, so don't think this is a general way to do this in Python. Running this line will open a browser with an interactive plot, and you will get a static plot in the Spyder `Plots' pane. 

    You may encounter an error here, and if you do, please refer to the Plotly technical Chapter \ref{ch:plotly} to trouble shoot it. The `browser' option should work for sure, while the `png' or `svg' may not at first. 

    \item [35 \& 36:] Both of these lines use the \lstinline{write_image()} method to create an image file on your computer.  The first line will create a png image and the second will create a svg image. In both cases, we provide a positional argument that is the name and path to the file we want to save.   We are starting with the same path we created back on line 12, but are using a method of the Path object, \lstinline{with_suffix()}, to change only the \textbf{extension} of the file (\textit{i.e.}, it otherwise has the same name). So, while the name was initially `0.999.csv', it is now `0.999.png' (line 32) and `0.999.svg' (line 36). We need to change the extension so that \lstinline{write_image()} knows what format to save the image file as, so that the data file is not accidentally overwritten, and because operating systems use extensions to specify what program to use to open files.
\end{itemize}
\end{footnotesize}

And that is it!  You now have a script that you can use over and over again to plot UV-vis spectra. If you need a different data file, then you just select it.  If you want a different x-range, you change that. But that is all you need do. Of course, this will only work for files that have two columns of data, but as we move forward, we will learn how to handle cases where this might change. 

\hrule
\input{Exercises/Chapter 02 Making nice plots quickly}
\hrule

\end{document}
