\documentclass[../book.tex]{subfiles}
\graphicspath{{\subfix{.}}}
\begin{document}

\chapter{Producing Structured Data Files}
\label{08Report}

\setcounter{section}{-1}  % Start subsections from 0

\input{prereqs/08Report_prereqs}

\newpage
\section{Problem}
\setcounter{subsection}{-1}  % Start subsections from 0

It is a normal day in the lab, which means there is good news and bad news.  The good news is that your experiments have gone well and you are starting to draft a paper based on them.  The bad news is that you need to gather all your results to report for this paper.  Specifically, you need to prepare two files for the supplementary information of the paper: in one file, your advisor wishes to collect all the results into a csv file so that these files can be used by anyone on any type of computer and does not require commercial software to read.  However, your advisor also wants to report just the parameters you obtained by your fitting in a table, which will be part of a Microsoft Word document that explains the fitting procedure and results. Thus, you need to make a Word table of containing these parameters.

Before learning to code, the above would have meant that you would spend an afternoon copying and pasting the exact same data between at least two programs, then formatting these files by hand. 

You now know that such formulaic processes can be handled well by coding.  You ask \first about it, and they smiles. "Absolutely! We can automatically produce complex csv and Microsoft Word files.  Let's start with the csv files and then move on to Word."


\section{Writing a csv file}
\label{Sec:N:08Report:WritingCSV}
\setcounter{subsection}{-1}  % Start subsections from 0

\subsection{Planning}

"There are lots of reasons to use the \cite{csv} format." \first says. "It is a \cite{plain_text} file, so you don't need specialized software to open it. It is human readable, so you can just open the file in any text editor and read the data. Both of these points means it is a reasonably transferable and future proof format. This makes it a nice choice when you are sharing it with other people that may have software or workflows you don't know about.  Like in the the supplementary information of a paper."

"Also," they continue, "the structure is simple: it is just the data you want, as text, separated by commas. We saw this the first time we worked with a spectrum.\footnote{See Chapter \ref{01OnePlot}} So, you basically already know everything you need to about the structure. We only need to introduce a few new ideas in order to make the code to produce it."

\begin{mdframed}
    \textbf{Binary Data Files versus csv files}
    
    \noindent While plain text files like csv files are convenient, the representation of data as text typically requires more storage space per unit of information than the equivalent representation in a \cite{binary_format}.  Many programs and instruments will, by default, store data in binary-formatted files to reduce the storage space required for the computers that operate them.  The disadvantage of binary files is that it is very difficult to retrieve the information stored in them without a program designed to do it, or at least knowing the exact formatting used to generate the file.  Because of this, many programs will allow you to export a plain text file, often csv, if you want it.
\end{mdframed}

% with open
\subsection{Writing text files with arbitrary formatting using \texttt{with ... open()}}
\label{Writing_text_files_with_arbitrary_formatting_using_with_open}

``There are going to be times that we want to save text files to the computer.  Numpy does have some functions for writing \cite{Numpy_array}s to files, like csv files, but you will want the formatting of your file to be different than what those function can easily produce. One case is when all of the data that you want to write is not in arrays that are the same length, or especially if you want to write some arrays and some single values.  So, it is beneficial to learn the general case for writing text files."

``We'll start by learning how to open a file---before you can write to it, you have to open it.  This is done using the \lstinline{open()} function, and providing a \cite{path} as the first \cite{positional_argument}. Additionally, we can supply another \cite{argument} stating if we need to read ('\lstinline{r}'), write ('\lstinline{w}'), or append ('\lstinline{a}'), the file. Read is used when you don't want to change the contents of the file, write is used to create an entirely new file, and append is used to \textit{add} to the end of an existing file. Here, we will open in read mode."

``If we simply use the \lstinline{open()} command we will also need to close the file, for instance, using \lstinline{close()} If we don't close the file, it stays open, which can use up \cite{memory}, and possibly lead to problems with the file down the line. But it is sort of silly to have to have to both manually open and close files. Thus, there is another way."

``\textbf{This other way automatically closes the file for you}, working under the assumption that we normally open a file, do something with it, and once we are done, we wish to close it. If this is true, we can use a new \cite{control_structure}, \lstinline{with}, combined with \lstinline{open()}. The construct \lstinline{with open():} starts a new \cite{block} using indented lines. When we end the codeblock (i.e., we un-indent the code) the file is automatically closed, as we have exited the control structure associated with the \lstinline{with open():} \cite{statement}. We strongly recommend using this \lstinline{with open():} structure, and we also recommend providing an \cite{alias} for the file using \lstinline{with open(<path to file>) as <alias>:}, similar to how this is done when \cite{import}ing a Python \cite{library}."

``Once the file is open, it is opened as a 'file object', which we can then write to. We can do this using the \lstinline{write()} method of the file object and providing the string we wish to write as the first positional argument of this function."  

\section{Code to produce a csv file}
\setcounter{subsection}{-1}  % Start subsections from 0
You think through the process that you need to go through to write this code, realizing that you will need a loop to iterate through each line to write, and assemble the line from the data.  You also know that the header line will have to be treated differently than the data lines.  Also, the first line will contain both the spectral data and the fit results, while the rest of the lines will only have spectral data.  You come up with a plan:

\begin{figure}[H]
    \centering
    \includegraphics[]{ChapNarrative/08Report/Figures/FlowChartCh14csv.pdf}
    \caption{Flowchart for code that will take the results from Chapter \ref{05CompFit}, and save it to a csv file.}
    \label{fig:Report:ActivityDiagramCSV}
\end{figure}

Working from this plan, you produce a code:

\lstinputlisting[language = Python]{../../Distribution/Code by Chapter/Ch14 - Producing Structured Data Files/TitrationReportCSV.py}

You run the script, select the code that you wrote to decompose the UV/vis peak into three Gaussians (DecomposeSpectrum.py, in the Ch06 folder of your Code directory) for importing, select the file 0.001.csv from your Catalyst/Titration/UVvis directory, and then in a moment, the script produces a new csv file.  If you open this in a text editor, it will be a bit hard to read, but if you open this in Microsoft Excel, you will see the following structure:

\begin{figure}[h]
    \centering
    \includegraphics[width = 5 in]{ChapNarrative/08Report/Figures/csvexcel.png}
    \caption{Screenshot of the csv file produced by the above code, opened up in Microsoft Excel.}
    \label{fig:Report:csv}
\end{figure}

%\First comments that this really isn't a well designed csv, because it has multiple headings in some columns, and columns are all different lengths, and so this would be a mess for anyone to import data from---say, back into Python. They comment, however, that there are programs that are better at handling this sort of structured sheet, for instance, Microsoft Excel, and so they will teach you how to do so at a later date (see next chapter!).

\subsection{Line-by-line explanation}

\begin{footnotesize}
\begin{itemize}
    \item [7:] Import the libraries we need.

    \item [8:] We use \lstinline{importFromPy()} to specify a Python file we wish to read and names of objects we want to import from it.  This function will attempt to open the file, and if it is not in the current working directory, then it will prompt the user to identify where it is, using the system's native file explorer. After the function executes, we will have direct access to the objects \lstinline{trimmed_wavelength}, \lstinline{trimmed_absorbance}, and \lstinline{result}, exactly as they were in the script `DecomposeSpectrum.py'.  This means that the script in `DecomposeSpectrum.py' will be run, and you will need to follow the prompts from that script.  At the end, the variable \lstinline{result} will hold the lmfit result object.

    \item [11:] We record the variables holding the $x$ and $y$ data in a dictionary.  We will add more data to the dictionary later on, and the reason for using a dictionary will become a bit more clear then (lines 15 \& 16). 

    \item [14:] We get the components of the fit, using the \lstinline{eval_components()} method of the fit object, and specify the $x$-values over which they should be evaluated.  Since our model consisted of a linear model and three different Gaussian models, the result is a dictionary with 4 items: the keys are the names of the components, while the $y$-values associated with each of these components.  

    \item [15 \& 16:] We use a \citett{for} \cite{loop} to take each component in turn and then add them to the \lstinline{spectral_data} dictionary, using the name of the component as the key. However we strip out the last character, which was a \lstinline{_}.  If you need to recall this, look back at the code produced in Chapter \ref{05CompFit}.
    
    \item [19--22:] We will want to save the csv file we are going to generate.  So, we pop up a message telling the user this is what is going on, then we use \lstinline{quickSaveFilename()} to allow them to specify the file name they wish to save the report as and the folder in which to save it. This is done using the system's native file explorer.  It returns a pathlib Path object, which is assigned to a variable for later use. 
    
    \item [25--51:] Write the file we want. We start this by using the \lstinline{with open() as <alias>:} construction, which starts a new code block in which our open file object is stored in the variable \lstinline{f}.  
    \begin{itemize}
        \item [27:] We write a string that explicitly specifies the first row, which serves as the header of the file. We are giving the header for each column, separated by commas. At the end, we include a newline character (\lstinline{\n}).

        \item [31:] We are now going to add all of the values to the file.  We want to include the wavelengths, experimental data, the data points for the best fitting spectrum and its components, and on the first row, the different parameter values. Since the spectral data is much longer than the parameter information, we need to handle these separately.  We need to iterate through all the spectra data, and so we start with that longer bit.  Since the wavelength, experimental spectra, fit and fit components are all the same length, we can just iterate for as long as they are (in this case, we use the wavelengths as a guide). If we use the \lstinline{enumerate()} function to do so, we can keep track of where we are in this list, and get the corresponding values for the other lists. 

        \item [32:] We are going to construct a string to hold the text for the line that we then write.  So, we start with a blank string. 

        \item [33 \& 34:] We then iterate through our spectral data, one at a time in a for loop, where we access the value of the data and concatenate it with our growing string. 

        \item [37--47:] We check if we are working on the first line of data. If so, that means we are we are going to write out all the parameter data as well, so we include that. Here, each line is a value (or uncertainty) for the parameters, corresponding to the order specified in the header.  Some of these values are formatted using \cite{e_notation}. If you need a refresher on how to access the results of fitting, you can read Chapter \ref{05CompFit}.

        \item [49:] We add a newline character to our string to end it. 

        \item [50:] We write our final assembled string to our file. The script ends here, because as soon as we are done with the \lstinline{for} loop, we will read the end of the \lstinline{with open} block and the file will be closed, and then the script will end. 
    \end{itemize}
\end{itemize}
\end{footnotesize}



\section{Writing a Microsoft Word file}
\label{Sec:N:08Report:WritingWord}
\setcounter{subsection}{-1}  % Start subsections from 0

You realize that you are unlikely to want to produce the \textit{entire} Microsoft Word document in Python. However, you find it a real pain to produce tables in Microsoft Word. It would be really nice if you could produce a document that had a section title and a table---in other words, the \textit{section} of the supporting information you will need.  Then you can either copy and paste it into the final document, or write around this part of the file.

``It is a bit frustrating to work with Microsoft Word in Python." \First says, "Because the finicky formatting can be a bit hard to handle. However, it is worth seeing how to create Microsoft Word documents, and the elements within them  and \textit{for sure} creating .docx tables in Python is much simpler than using the GUI in Word." 

``Really the only new thing here is to understand how .docx files are structured and then how to write them.  There is a lot of specialty stuff for how to write them, and so the best way to illustrate this is just to create one.  However, before we do this, we should consider the general structure of a .docx file."




\subsection{Making Microsoft word tables using docx}
\label{Making_Microsoft_word_tables_using_docx}

``When you open a Microsoft Word document using the Microsoft Word program, it \textit{looks} as though it is only text---albeit formatted text.  But there is much more going on behind the scenes. Microsoft has all sorts of additional information that is not shown, concerning things like the size of the paper, the font that should be used, the margin size, etc.  Even things that appear to be just text often are not." 


``Microsoft Word provides the ability to designate different types of text---for instance, we can select text and designate it a `heading' type. Though a heading (like a title) may just look like it is a bolder, larger set of text, Microsoft Word considers it to be a different kind of object.  This is why you can change the formatting for a heading type in Word, and then all the text that is in headings will change throughout the document at once. Thus, there is all sorts of `meta data' involved in a Word document. Often \cite{binary_encoding} is the most efficient way to keep track of such information. If you try to open a .docx file in a text editor, then you will just get a series of numbers---the binary encoded data."

``The main reason to point all this out is so that you realize that writing a Microsoft Word document is \textit{never} really just writing text.  We have to write text and make sure it is given the proper text type, as well as any other formatting information we may want it to have.  This also means that things that seem simple when using the GUI interface within Word actually can be quite complex constructions. For this reason, you are going to see that we import many different \cite{module}s for working with docx files. Additionally, there is going to be a significant amount of terminology associated with making these files.  For  instance, the most basic kind of text is called a `run' and we will need to use an  \lstinline{add_run()} \cite{function} to write even the most basic text."

``But, rather than try to front-load all of this, let us just jump in and make the file of interest.  Once we have the code, we can can explain how it works, and hopefully give you enough understanding so that you can read up on how to make any other changes you like like."



\section{Code to produce a .docx file}
\setcounter{subsection}{-1}  % Start subsections from 0

\begin{figure}[H]
    \centering
    \includegraphics[]{ChapNarrative/08Report/Figures/FlowChartCh14docx.pdf}
    \caption{Flow chart for code that will take the results from Chapter \ref{05CompFit} and write them to table in a Microsoft Word File.}
    \label{fig:Report:ActivityDiagramWord}
\end{figure}

\lstinputlisting[language = Python]{../../Distribution/Code by Chapter/Ch14 - Producing Structured Data Files/TitrationTableDocx.py}

\subsection{Line-by-line explanation}

\begin{footnotesize}
\begin{itemize}
    \item [7--11:]  Import the libraries we need.  All but the last of these import statements are specifically for working with docx files. 
    \begin{itemize}
        \item [7:] This will allow to create and write the document object. 

        \item[8:] 'oxml' stands for  "Office Open XML".  In turn, "XML" stands for "eXtensible Markup Language," which is a way to supply formatting and type tags to data.  Esstentially, this line lets us handle the internal stucture of the .docx. 

        \item [9:] This line allows us to handle numeric data in .docx format. 

        \item [10:] This line allows us to control alignment with in the .docx. 
    \end{itemize}

    \item [12:] Since the point of the \lstinline{importFromPy()} function is to import arbitrary Python code, we add its usage right up against the import block. We are importing the script from Chapter \ref{05CompFit}. We are importing the same script as used in the other report builder in this chapter, so refer to the line-by-line comments there for more information. 

    \item [15:] We create a document object, and assign it to the variable doc. 

    \item [18:] We add a level 1 (i.e., highest level) heading object, and assign it to the variable title. 

    \item [19:] We create a run object that will hold the text for the title. We create this up to the first special formatting we encounter, which will be a subscript '2.' 

    \item [20:] We create a separate run object, that holds just the numeral 2. Though this is added to the title run, the fact that it is also a separate object means we can format it separately. 

    \item [21:] We now take the run that has the 2, and tell it to be subscripted. 

    \item [22:] We then finish up the title.

    \item [25:] We now start a paragraph with normal text, assigning it a variable. 

    \item [26 \& 27:] We go through the same process of adding in the subscript 2 that we used in lines 20 and 21. 

    \item [28:] We finish up the paragraph. 

    \item [31:] We create a list of three strings that are the words we will use for the heading of our table. 

    \item [32:] It will prove convenient to know how many components we want to report, and so we set a variable for this.  Though we often try to avoid hard coding of variables, here it is ok, since we are only using this to generate a single specific table. We set this to 3, since we only want to report the parameters for the Gaussian components, and there are three of them. You can refer to Chapter \ref{05CompFit} for the details of the model we used. 

    \item [33:] We also set the number of columns we will need, which is 2 times the number of parameters.  We multiply by 2, because we want to have a column for each parameter value, and then a column for the uncertainty in each parameter value. 

    \item [34:] We create a table with the number of rows and columns we want. We have 1 extra row and column, since we also need headings for these. 

    \item [35:] We set the style of the table to a style pre-defined in the standard for .docx documents. 

    \item [38:] We select the cells of the first row of the table.

    \item [39:] We add text to the first cell in this row, and we add the text "Component" to the cell.

    \item [40--42:] We start a \citett{for} \cite{loop} that will iterate through our parameter names and add them to the cells in the header row of our table. We use the \lstinline{enumerate()} function to count which row we are working on, and we have to add 1, since the first column is reserved for component names. We add the parameter name, and then the parameter name + `uncertainty.' 

    \item [45--55:] We are going to apply formatting to the header row cells now.  We want the header row to be filled with a gray, and the rest to be white.  Also, the text in the top row should be bold. We are not going to explain the inner workings of this here, but you can refer to the documentation for the docx library.  Or you can have a nice conversation with an AI model, like ChatGPT.

    \item [58--66:] We now write the rest of the data. We are going to employ an approach like that used in lines 40--42 for the header, just looping through more values to write. We add 1 to the row number to account for the header row (row 0) and either 1 or 2 to the column number to account for the Component column (for value columns) or the Component and value columns (for uncertainty columns).

    \item [69:] We use a popup to tell the user that we want to save this word document. We then open up native file explorer (using \lstinline{quickSaveFilename()}) and have them identify the name and folder to save in. We restrict the filenames to those that will end in docx---since we want this to be a file that Microsoft Word will automatically recognize. 

    \item [74:] We write the docx file. 
    
\end{itemize}
\end{footnotesize}

You run this code the same way as the CSV code you just made, and see a new .docx file appear in your folder. When you open it in Microsoft Word,\footnote{At the time of writing, Google Docs and Apple Pages successfully open the file, too.} you see the following:

\begin{figure}[h]
    \centering
    \includegraphics[width = 5 in]{ChapNarrative/08Report/Figures/wordtable.png}
    \caption{Screenshot of the content of the Microsoft Word document produced by the above code.}
    \label{fig:Report:WordFile}
\end{figure}

You simply copy and paste this table into your SI document.  Now, you can just rerun this code for each file and avoid having to transcribe all of these fit results into tables by hand!

\hrule
\input{Exercises/Chapter 14 Reporting}
\hrule

\end{document}
