\documentclass[../../book.tex]{subfiles}
\graphicspath{{\subfix{.}}}
\begin{document}

\chapter{Files and Folders}
\label{Files}
\setcounter{section}{-1}  % Start subsections from 0

\section{Understanding the file system}

Your computer is essentially useless without files and folders.  Every program you use, website you visit, and piece of data that you collect, is stored in a file, that is stored in a folder, that is stored in another folder, and so on. This is true for apps on your phone as well, but your phone hides this fact from you better than your computer does. To work with experimental data, you will need to understand your file system.

\subsection{All of the information that you use is stored in files}
\setcounter{subsubsection}{-1}  % Start subsections from 0

Files make up essentially everything on your computer. Files are a container for data that makes the data easy to name, to organize, to copy, and to transfer.  They can contain almost anything: programs that you run, data that you analyze, documents that you read or write, videos of cats, or settings for your computer, to name just a few.

There's nothing special about a file, it is just an artificial construct that lets the computer keep all of these related pieces of data together. Your computer likely has tens to hundreds of gigabytes of data on it.  There has to be a mechanism to figure out what data goes with what other data, and the first layer of this mechanism is the file. Imagine that every data point of your UV/vis spectrum that you just recorded was just \textit{somewhere} different in your computer.  In principle, the spectrum could still be recalled, but the work to find all of those individual pieces of data and put them together into a useful spectrum would be immense.  Instead, why not invent a container for then and put them all in the container?  And while you're at it, you might as well give that file a name so that you can remember what it in it or what it does. The container is a file, the name of the container is the filename. 

It is worth pointing out that your computer also has files that you are not really intended to change or access.  These can be files that are integral to how your computer works, and so changing them can lead to disastrous consequences. For this reason, most operating systems have a way hide these files (called `hidden files') so that they are not visible in your file browser. However, because there can be legitimate and useful reasons to access these files, most operating systems also have ways to to temporarily or permanently unhide these files.

\subsubsection{Files have names that (hopefully) describe their content and extensions that give clues on what program to use to open them}

Files have names that are text \cite{strings}.  You may have noticed that those names often take the format of \lstinline{<name>.<extension>}, where \lstinline{<name>} is something that you get to choose and \lstinline{<extension>} is a seemingly arbitrary string of letters and numbers (or not so arbitary, like \lstinline{.py}) that the program you're using appears to choose for you.  For instance, you are probably used to seeing filenames like ``Homework5.docx" for Microsoft Word files.  Here, the filename is ``Homework5" and the extension is ``docx". Sometimes the extension of files are hidden by your computer.  There is nothing magic about the extension, it is just there to help your operating system guess which program on your computer to use to interact with the file's contents.  You can change the extension if you want, but it usually just makes your life harder.

% \subsubsection{Common file types}

% Most file types are interoperable between different operating systems by design.  The ones that aren't are usually files that contain programs, which are most often specific to a specific operating system.  Below is a table of file types that you will commonly encounter.

% MAKE FILE TYPE TABLE

\subsection{Files are stored in folders}

The name ``file system" is an allusion the old days of manila file folders and filing cabinets.  These folders were used to store related files and had a tab on which you could write some description of the contents that should be found inside. This allusion is carried forward in a visual metaphor, where the folder icon is a stylized representation of a manila folder. This is basically what folders on your computer do: they group files that you think are logically related into collections.  Folders (also often called directories) have names but have no real data associated with them and they don't do anything by themselves.  In most computers, no two files in the same folder can have the same name, and no two folders in a given subfolder can have a same name.  However, two files with the same name can quite easily be accommodated in two different folders.

Just as with files, there are folders that you either don't need to access or which accessing and changing would be dangerous for the continued operation of your computer.  To help you avoid that outcome and to avoid unnecessarily clutter, operating systems often hide these folder.  However, just as for files, you often can unhide them either temporarily or permanently.

Like files, folders can sometimes be hidden in your file system.

\subsection{Folders are an organizational hierarchy}

Folders can contain other folders, which can in turn contain other folders, and so on.  Each level of the folder hierarchy can also contain files, too.  At some point, if you track this system to the top level of the hierarchy, you will find the ``root" folder--the folder that is itself not contained in a folder.  On a Unix-like system (including MacOS or Linux (or iOS or Android)), the root folder is actually referred to as ``root" and is denoted by \lstinline{/}.  On a Windows computer, the root folder that is denoted by a `drive letter,' most often a `C', which means that the root folder is denoted \lstinline{C:\}.

Figure \ref{fig:ChapTechnical:Files:Hierarchy} presents a hypothetical piece of a file system.  We can see that the main folder is called Research---this is not the root folder, just the highest level of the hierarchy that we are looking at.  In the Research folder, there exist four more folders: Experiments, Analysis, Results, and Miscellaneous, and each of these folders contains more files and/or folders.  

\begin{figure}
\label{fig:ChapTechnical:Files:Hierarchy}
\footnotesize
\begin{forest}
for tree={
    font=\ttfamily,
    grow=east,
    edge path={
        \noexpand\path [draw, \forestoption{edge}]
        (!u.east) -- ++(5pt,0) |- (.child anchor)\forestoption{edge label};
    },
    parent anchor=east,
    child anchor=west,
    l sep=20pt, % Less horizontal space
    s sep=5pt, % Less vertical space
    anchor=west,
    inner sep=2pt, % Inner padding for boxes
    outer sep=0pt,
    rectangle,
    minimum height=12pt,
    minimum width=15pt,
    draw
}
[Research, fill=lightergray
    [Miscellaneous, fill=lightergray
        [note2.txt, fill=none]
        [note1.docx, fill=none]
        [Protocols, fill=lightergray
            [protocol2.docx, fill=none]
            [protocol1.pdf, fill=none]
        ]
    ]
    [Results, fill=lightergray
        [summary\_report2.pdf, fill=none]
        [summary\_report1.pdf, fill=none]
        [Graphs, fill=lightergray
            [plot2.png, fill=none]
            [plot1.png, fill=none]
        ]
    ]
    [Analysis, fill=lightergray
        [Spectroscopy, fill=lightergray
            [UV\_Vis, fill=lightergray
                [uv\_spectrum2.dat, fill=none]
                [uv\_spectrum1.dat, fill=none]
            ]
            [NMR, fill=lightergray
                [nmr\_spectrum2.fid, fill=none]
                [nmr\_spectrum1.fid, fill=none]
            ]
        ]
        [Chromatography, fill=lightergray
            [GC, fill=lightergray
                [gc\_chromatogram2.dat, fill=none]
                [gc\_chromatogram1.dat, fill=none]
            ]
            [HPLC, fill=lightergray
                [hplc\_chromatogram2.dat, fill=none]
                [hplc\_chromatogram1.dat, fill=none]
            ]
        ]
    ]
    [Experiments, fill=lightergray
        [Experiment2, fill=lightergray
            [Raw\_Data, fill=lightergray
                [chromatogram2.csv, fill=none]
                [chromatogram1.csv, fill=none]
            ]
            [Processed\_Data, fill=lightergray
                [analysis2.txt, fill=none]
                [analysis1.xlsx, fill=none]
            ]
        ]
        [Experiment1, fill=lightergray
            [Raw\_Data, fill=lightergray
                [spectra2.csv, fill=none]
                [spectra1.csv, fill=none]
            ]
            [Processed\_Data, fill=lightergray
                [analysis2.txt, fill=none]
                [analysis1.xlsx, fill=none]
            ]
        ]
    ]
]
\end{forest}
\normalsize
\caption{A vew of a typical file structure that might look something like yours.  Gray boxes denote folders and white boxes denote files.}
\label{fig:Text:FileStructure}
\end{figure}

\subsection{The location of a file or folder in the file system is called a \textbf{path}}

Within this hierarchical system of folders, sub-folders, and files, we need a method to \textit{uniquely} identify any file or folder in the system.  Since files in different folders or sub-folders in different folders can have the same name, it is not enough just to keep track of all of the names.  In addition, there are billions of files on a computer---to have to search through all of them to find the right one would take a long time.

The unique location of a file or folder in a file system is called its \cite{path}.  The path contains the names of every folder that must be traversed to get from the root folder to the file or folder in question.  Unhelpfully, Unix-like operating systems separate each subsequent sub-folder in the path with \lstinline{/}, while Windows uses \lstinline{\}.  However, conceptually, both systems work the same way.  So, for one of the authors (writing on a Mac) the path to the \cite{Python_interpreter}  is:

\noindent\lstinline{/Library/Frameworks/Python.framework/Versions/3.12/bin/python3}

\noindent where the first \lstinline{/} denotes the root directory, \lstinline{Library} is a folder in the root directory, \lstinline{Frameworks} is a folder in the \lstinline{Library} folder, and so on.  Eventually, we get to \lstinline{python3}, which is the actual file that is opened in order to run Python.


For the other author (writing on a PC) the path to the Python interpreter is:

\noindent\lstinline{C:\Users\Admin\AppData\Local\Microsoft\WindowsApps\python.exe}

The operating system provides all of the facilities you need to move files or folders, delete them, copy them, and more, and many of these capabilities are also included in \acite{libraries} in Python.

\subsection{Absolute paths specify the location of any file or folder in the file system}

These paths are known as \textbf{absolute paths} because they start with the root folder.  Absolute paths are the most specific way to specify the location of a file or folder, but because they are so specific, they are less flexible.



\subsection{Relative paths specify the location of a file or folder with respect to some other folder}

Sometimes, the exact specificity of an absolute path is undesirable.  Let's take the example in Figure \ref{fig:ChapTechnical:Files:Hierarchy}.  Suppose that this is the file structure required by the laboratory that you work in. You are going to write a script to work up data, and a coworker expresses interest in using it.  On your computer, the absolute path to ``hplc\_chromatogram.dat" may be 

\lstinline{C:\Users\cjj\Research\Analysis\Chromatography\HPLC\hplc_chromatogram.dat}

\noindent but if you look at the location on your co-workers machine, you find it is:

\lstinline{C:\Users\bjl\Research\ Analysis\Chromatography\textbackslash HPLC\hplc_chromatogram.dat}

\noindent so the absolute paths are not the same, which means that your co-worker will need to change the path to use the script.

However, there is a solution! Looking at these two absolute paths, you can see that see that, once you get to the folder \lstinline{Research} the rest of the paths are the same. The structure of the entire file system is the same after \lstinline{Research} folder. Thus, as long as you are thinking about the position of files from \lstinline{Research} forward, the location will be the same on both of your computers. We would say that all the paths are the same `relative' to the \lstinline{Research} folder, and this idea is known as a \cite{relative_path}.

In this case, the relative path from `hplc\_chromatogram.dat' from `Research' is:

\lstinline{.\Analysis\Chromatography\hplc_chromatogram.dat}

So, if you are producing a graph of this data, and you want to keep it in the `Graphs' folder (where you will store the plot), you can use the folowing relatie path:

\lstinline{..\..\Analysis\Chromatography\hplc_chromatogram.dat}

The first two usages of \lstinline{..} move from the folder with the script (`Graphs') back to the `Research' folder, and then the path proceeds from there to the data. 

If you write the script this way, then your co-worker will be able to use it directly, without any change. 

Relative paths are not only useful when you are trying to specify paths on different computers, they're also useful shortcuts when you're just working on your own computer.  Often, the file or folder that you want to specify is very close to the one in which you are working.  Having to specify the absolute path for one or both files or folders would require a lot of typing, while the relative path is likely to be much shorter.  In addition, if you move entire parts of your file system from one folder to another (let's say you're reorganizing your data, and move the hierarchy in Figure \ref{fig:ChapTechnical:Files:Hierarchy} from a folder called ``Active Projects" to a folder called ``Finished Projects," then the absolute paths will change, but the relative path between them will not!

\section{File system metadata provides additional information beyond file and folder contents}
\setcounter{subsection}{-1}  % Start subsections from 0

You may notice when using Windows Explorer, the Mac Finder, or any number of tools in Linux, that you can see when a file was made, when it was last modified, what its size is, and many other attributes.  This data is not actually part of the file---instead, it is collected and updated by the file system as \textbf{file metadata}.  In addition to a map of the file hierarchy, the file system itself keeps a list of all of the properties of individual files and folders and updates this list every time a change is made to any file. Thus, if you can access this metadata, you can choose between two files, selecting that which was edited most recently. 

\subsection{Metadata is not always reliable}

Because the file metadata is maintained by the file system, it is not always reliable.  For instance, if you copy a file onto a USB flash drive, and the flash drive is formatted a certain way, the date and time that the file was created may be lost.  Once you copy that file onto another computer, from that computer's file system's perspective, it is a new file.  File systems will try to reproduce each others' metadata, but it is not always possible.  Thus, for critical applications (for instance, using the file created date to determine when a UV/vis spectrum was recorded), it is best not to rely on the metadata.  Fortunately, most instrument or simulation software will note inside the files they produce when the file was made.  If you make your own files in your own scripts, it is never a bad idea to do the same.

\section{Python provides a built-in library to work with the file system, but you should usually use \lstinline{pathlib} instead}

Python includes the \lstinline{os} library which does provide many routines for interacting with the operating system and its file system, including routines to move, copy, delete, and rename files and folders.  However, because Windows and Unix-like operating systems are so different, different routines must be provided to accomplish this.  This means that, if you use the \lstinline{os} library, you will need to write your code to account for either type of operating system, or accept that it will not be usable on one of the two. For instance, the use of relative paths discussed earlier in this chapter would only work across Mac and Windows if you had a way to change between \lstinline{\} and \lstinline{/} characters as needed.  This is either a lot of extra work, or a headache for potential users, and thus it is best to avoid this.  The Pathlib library, discussed in Chapter \ref{pathlib}, alleviates most of these issues, for instance, automatically handling the change between \lstinline{\} and \lstinline{/} when you move between operating systems. 

\end{document}