\documentclass[../../book.tex]{subfiles}
\graphicspath{{\subfix{.}}}
\begin{document}

\chapter{Working With Text}
\label{Text}
\setcounter{section}{-1}  % Start subsections from 0

Most of your interactions with computer data occur through the process of entering or reading text.  This text could be letters and words, numbers, symbols, or emoji, but the computer treats them all basically the same (as binary values).  It is important to remember that the text on the screen, or that you enter via a keyboard, is just a representation of data---a pictorial indication of what the data is.  The computer does not perform mathematical operations using numerals, and indeed, has no need for them.  However, since humans are not typically adept in dealing directly with binary data, text is a useful intermediary between us and the machine.  In this chapter, we will discuss how to use text as an expression of information in a way that humans and computers both can understand, how to send text to your program and have it give you text back, how manipulate it, and how to convert between text and numerical values.

\section{What is Text Data?}

Text is just a collection of characters, which are individual symbols that humans recognize as representing letters, numbers, or other symbols,\footnote{In some languages, characters can also represent syllables, things, concepts, etc.} as well as things you don't see, like spaces, new lines, and even literally nothing (the ``NULL'' character).  Since data in a computer is binary, for your computer to understand text, some system must be devised that defines which binary numbers correspond to which characters.  This is called an \cite {binary_encoding}.  An early, widely-used encoding is ASCII, which uses 7 bits to identify capital and lower-case letters (though only those commonly used in the English language), numbers, a selection of symbols, and many non-displaying characters, for a total of 128 characters (Table \ref{tab:Text:ACIICharacters}).  Eventually, 128 characters was not enough, so many expanded encodings were developed.  The one that has become dominant is UTF-8, which is backwards compatible with ASCII and uses up to four bytes to represent (at the time of writing) over a million characters.  We will leave the details to other resources, since this is something that is mostly handled behind the scenes and so will not be critical for understanding this book.  We have, however, included a function, \lstinline{chemSymbols()}, in the \lstinline{codechembook} library at \website that simplifies the inclusion of common characters in both command line and plot outputs. Also, we note that, because text has so many specialized functions associated with encoding and rendering characters, that it is given its own \cite{data_type}: \cite{string}s.

\begin{table}[H]
\caption{Codes for ASCII characters.}
\label{tab:Text:ACIICharacters}
\footnotesize
\begin{tabular}{|c|*{16}{>{\centering\arraybackslash}p{.4cm}}|}
\hline
 & \multicolumn{16}{c|}{\textbf{Digit 2}} \\
\hline
\textbf{Digit 1} & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5} & \textbf{6} & \textbf{7} & \textbf{8} & \textbf{9} & \textbf{A} & \textbf{B} & \textbf{C} & \textbf{D} & \textbf{E} & \textbf{F}\\
\hline

\textbf{0} & NUL & SOH & STX & ETX & EOT & ENQ & ACK & BEL & BS & TAB & LF & VT & FF & CR & SO & SI \\
\textbf{1} & DLE & DC1 & DC2 & DC3 & DC4 & NAK & SYN & ETB & CAN & EM & SUB & ESC & FS & GS & RS & US \\
\textbf{2} & Space & ! & " & \# & \$ & \% & \& & ' & ( & ) & * & + & , & - & . & / \\
\textbf{3} & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & : & ; & < & = & > & ? \\
\textbf{4} & @ & A & B & C & D & E & F & G & H & I & J & K & L & M & N & O \\
\textbf{5} & P & Q & R & S & T & U & V & W & X & Y & Z & [ & \textbackslash & ] & \^{} & \_ \\
\textbf{6} & ` & a & b & c & d & e & f & g & h & i & j & k & l & m & n & o \\
\textbf{7} & p & q & r & s & t & u & v & w & x & y & z & $\{$ & | & $\}$ & \~{} & DEL \\
\hline

\end{tabular}\\
\textit{Note: Three-letter characters are non-printable and are instructions for how to display or transmit characters.}
\end{table}

\normalsize

% \section{Displaying and Inputting Text}

% Sometimes your code can run with no user intervention, but more commonly, it will need to at least display some results to the user, if not as the user for input data in order to function.  The simplest way to do this is using text.  

\section{Working with text in Python using strings}
\setcounter{subsection}{-1}  % Start subsections from 0

The Python standard library provides numerous functions for modifying, taking apart, and analyzing strings.  Most string creation is done by coding string \cite{literal}s or by \cite{function}s that read text from files, user inputs, or the internet.  Here we present an overview of common string operations, and we refer you to the Python documentation for less-common ones.

\subsection{Creating strings}
\setcounter{subsubsection}{-1}  % Start subsections from 0

While you will probably most often create string literals by enclosing text between single (\lstinline{'}) or double (\lstinline{"}) quotes, there are many other ways to create strings.



\subsubsection{Creating strings from other data types using \texttt{str()}}

Recall that there is a difference between the integer \lstinline{1} and the string \lstinline{'1'}.  To turn the number 1 into the string, you can use the built-in \lstinline{str()} function.  This function actually works on any data type, though the output that you receive from it varies tremendously, particularly for collection data types.  Particularly for floating point numbers, \lstinline{str()} will tend to but not always give you the same number of digits of precision as the floating point number can provide.

\subsubsection{How strings are interpreted and displayed}
Because text data contains characters that represent ideas that often do not need to be drawn with `ink,' Python needs a way to handle these characters. Consider the function of the Enter (or Return) key on your keyboard. We are very used to this creating a new line of text \textit{and} having this represented as... well... a new line.  However, within the data file for text, this is not the case.  Instead, the text data is simply a uninterrupted set of characters with a special character used to indicate that there should be a new line.  This 'newline character' is represented by {\textbackslash}n in Python. If we specify such a character in a string, we can see it is present:


\begin{pyin}
s6 = "One line {\textbackslash}n Second line"
\end{pyin}
\begin{pyin}
s6
\end{pyin}
\begin{pyout}
"One line {\textbackslash}n Second line"
\end{pyout}
However, if we ask to print the line, then the {\textbackslash}n character will be interpreted as a new line. \\
\begin{pyin}
print(s6)
\end{pyin}
\begin{pyout}
'One line\\
 Second line'
\end{pyout}

There are a host of such special characters.  For instance, tab is represented by {\textbackslash}t. Additionally, there are a large number of characters we might wish to display, but which are not on a standard keyboard.  For instance, in the USA, the most common keyboard has no Greek characters, though we often use Greek letters in science.  We can access such characters using the unicode designation: {\textbackslash}u+\#\#\#\#, where the \# = any hexadecimal number. The unicode for the lower case Greek delta is u03B4 so we can use it in a string as follows:
\begin{pyin}
"\u03B4"
\end{pyin}
\begin{pyout}
'$\delta$'
\end{pyout}
Here it is important to note that the capitalization of the `u' is important, and that the specification for u and U are different.  This is taken up more in the chapter on text. 

Also, because the ' and " are used in the creation of a string, they are also special characters.  This can create problems when trying to indicate quotations. 

\begin{pyin}
'If you want to use a "double" quote to indicate quotations, then start the string with a single quote'
\end{pyin}
\begin{pyout}
'If you want to use a "double" quote to indicate quotations, then start the string with a single quote'
\end{pyout}
\begin{pyin}
"If you want to use a 'single' quote to indicate quotations, then start the string with a double quote."
\end{pyin}
\begin{pyin}
"If you want to use a 'single' quote to indicate quotations, then start the string with a double quote."
\end{pyin}

The other is to use the \textbackslash character, which is also known as the 'escape' character.

\begin{pyin}
"If you want to use \textbackslash{}"double\textbackslash{}" quotes to start the string and to indicate a quotation, then you have to use a backslash to escape the quotes used for quotations."
\end{pyin}
\begin{pyout}
"If you want to use \textbackslash{}"double\textbackslash{}" quotes to start the string and to indicate a quotation, then you have to use a backslash to escape the quotes used for quotations."
\end{pyout}


\subsubsection{Modifying how strings are interpreted}
We can control how Python interprets strings by preceding the ' or " characters that start the string creation with a letter. For this book, we will consider two: r and f. 

\textit{r-strings} are those for which interpretation of the characters is removed. We will get the characters, as they are in the string.

\begin{pyin}
print(r"For r-strings, special characters like \textbackslash{}n are not interpreted.")
\end{pyin}
\begin{pyout}
"For r-strings, special characters like \textbackslash{} are not interpreted."
\end{pyout}
When we had a string with the \textbackslash{}n character in a string before, printing that string resulted in a new line.  However, in the r-string, it is not interpreted, and we see the newline character printed out. 

This can be extremely useful when we want to write the string to be interpreted by something else... like a file.

\vspace{1em}
\textit{f-strings} are those which allow direct interpretation of variables within them. We do this by adding an 'f' in front of the string declaration, and then placing variables we want interpreted byetween curly brackets:  

\begin{pyin}
print(f"A prior string we had was: {s1}")
\end{pyin}
\begin{pyout}
A prior string we had was: hi
\end{pyout}

Of course, as scientists one of the most common uses of variables is going to be numbers, and so it is worth a separate discussion of how to format them. 




% \subsection{Inserting the values of variables into strings dynamically with f-strings}

% While \lstinline{str()} will convert, say, a floating point number to a string representing that floating point number, we often want to embed the resulting string into a sentence for the sake of clarity and readability.  There are many ways to do this, but our favorite is with ``f-strings."

% An f-string is denoted by including the character \lstinline{f} at the beginning of a string:

% \begin{pyin}
% f"This is an f-string"
% \end{pyin}
% While they are otherwise like any other string, f-strings have a special notation that allows you to inject the value of a variable into them:

% \begin{pyin}
% concentration = 1e-3 # M
% \end{pyin}
% \begin{pyin}
% f"The value of the variable concentration is {concentration}"
% \end{pyin}
% \begin{pyout}
% 'The value of the variable concentration is 0.001'
% \end{pyout}
% Inserting a variable name between \{\} symbols tells Python to look up the current value of that variable and replace everything between and including the \{\} with the string representation of that variable (implicitly by applying the \{\} function to it).  You can do this as many times as you want in a given string.

\subsection{Using format codes to control the display of values in f-strings}

You will probably want to exercise more control over the appearance of the data inserted into f-strings.  f-strings provide a means to do this by inserting a : after the variable name in the \{\} symbols, and after the colon, a special format code:

\begin{pyin}
f"The value of the variable concentration is {concentration:.6f}"
\end{pyin}
\begin{pyout}
'The value of the variable concentration is 0.001000'
\end{pyout}
Here we requested that the value of concentration be written out to 6 decimal places.  An overview of common format codes is given below:

\begin{table}[h!]
    \caption{Format Codes for Python f-strings}
    \label{tab:Text:FormatCodes}
    \centering
    \begin{tabular}{|>{\raggedright}p{4cm}|>{\raggedright\arraybackslash}p{10cm}|}
        \hline
        \textbf{Format Code} & \textbf{Description} \\
        \hline
        \multicolumn{2}{|c|}{\textbf{Strings}} \\
        \hline
        \{\} & Default format for strings. \\
        \hline
        \{:<n>\} & Left-align the string within a field of width \textit{n}. \\
        \hline
        \{:>n\} & Right-align the string within a field of width \textit{n}. \\
        \hline
        \{:\^n\} & Center-align the string within a field of width \textit{n}. \\
        \hline
        \multicolumn{2}{|c|}{\textbf{Integers}} \\
        \hline
        \{d\} & Decimal integer. \\
        \hline
        \{:b\} & Binary format. \\
        \hline
        \{:X\} & Hexadecimal format (uppercase). \\
        \hline
        \multicolumn{2}{|c|}{\textbf{Decimal Numbers}} \\
        \hline
        \{f\} & Fixed-point number. \\
        \hline
        \{.nf\} & Fixed-point number with \textit{n} digits after the decimal point. \\
        \hline
        \{:,.nf\} & Fixed-point number with \textit{n} digits after the decimal point and comma as a thousand separator. \\
        \hline
        \multicolumn{2}{|c|}{\textbf{Scientific Notation}} \\
        \hline
        \{e\} & \cite{e_notation} (lowercase). \\
        \hline
        \{E\} & e notation (uppercase). \\
        \hline
        \{.ne\} & e notation with \textit{n} digits after the decimal point. \\
        \hline
    \end{tabular}

    \label{tab:format_codes}
\end{table}

\subsection{Using \texttt{.format()} to insert values into strings dynamically}

An older and less readable way to insert variable values into strings is using the \lstinline{.format()} \cite{method}.  You will see this a lot in older Python code, as f-strings are relatively new.  There are a number of ways to use this method:

\begin{pyin}
"The value of the variable concentration is {}".format(concentration)
\end{pyin}
\begin{pyout}
'The value of the variable concentration is 0.001'
\end{pyout}

\begin{pyin}
"The value of the variable concentration is {0}".format(concentration)
\end{pyin}
\begin{pyout}
'The value of the variable concentration is 0.001'
\end{pyout}

\begin{pyin}
"The value of the variable concentration is {concentration}".format(concentration = concentration)
\end{pyin}
\begin{pyout}
'The value of the variable concentration is 0.001'
\end{pyout}

You can still use the format codes discussed above to format the text being inserted into the string:

\begin{pyin}
"The value of the variable concentration is {:.6f}".format(concentration)
\end{pyin}
\begin{pyout}
'The value of the variable concentration is 0.001000'
\end{pyout}

\subsection{Creating strings derived from other strings}
\setcounter{subsubsection}{-1}  % Start subsections from 0

Strings are immutable, and thus string modification methods must output new strings!  Also indexing operations are available with them.\footnote{For a more thorough discussion on mutability and indexing, consult the the Chapter on data types: Chapter \ref{DataTypes}}

\subsubsection{Concatenating two strings into one}
Concatenating is the process of joining two strings to make one.  The resulting string's length is the sum of the lengths of the two original strings.  There are many ways in Python to concatenate strings, but the simplest is just to use the addition operator (\lstinline{+}:

\begin{pyin}
str1 = "Hello"
\end{pyin}
\begin{pyin}
str2 = " World"
\end{pyin}
\begin{pyin}
str3 = str1 + str2
\end{pyin}
\begin{pyin}
print(str3)
\end{pyin}
\begin{pyout}
Hello World
\end{pyout}

\subsubsection{Replacing a subset of a string with a different string using \texttt{.replace()}}

This method will look through a string (let's call it \lstinline{parent_string}) and replace all instances of a shorter string (\lstinline{child_string}) with a different string:

\begin{pyin}
parent_string = "This book is the best book!"
\end{pyin}
\begin{pyin}
parent_string.replace("book", "class")
\end{pyin}
\begin{pyout}
"This class is the best class!"
\end{pyout}

If you want to remove a substring entirely, simply use an empty string (\lstinline{""}) for the second argument.  If \lstinline{child_string} is not found, then this method has no action. Remember, strings are not mutable, and thus this method produces a completely new string.

\subsubsection{Changing case}

Python includes many standard \cite{methods} to change the case (i.e. upper/lowercase) of words in strings.  A few common ones are:
\begin{itemize}
    \item \lstinline{.upper()}: change all letters to upper case
    \item \lstinline{.lower()}: change all letters to lower case
    \item \lstinline{.capitalize()}: capitalize all words
    \item \lstinline{.swapcase()}: change capital letters to lower case and \textit{vice versa}
    \item \lstinline{.title()}: force capitalization to follow title case conventions
    \item \lstinline{.casefold()}: like \lstinline{.lower()} but useful for more characters than just the latin alphabet. 
\end{itemize}

\subsection{Taking apart a string to create multiple new strings}
\setcounter{subsubsection}{-1}  % Start subsections from 0

Often, we have a string that has multiple substrings in it that we would like to extract.  A simple example is breaking a sentence up into individual words.  One common scenario in chemistry is reading text from a file or website that has numerical data presented in natural sentence form, like from simulation software.  In this case, it is useful to break the string up into substrings and convert the relevant substrings into numerical data types. 

\subsubsection{Using a specific string to denote the divisions between substrings with \texttt{.split()}}

The example of splitting a string representing a sentence into individual words is a classic use case of \lstinline{.split()}.  This method searches a string for a given substring, and at each instance of that substring, divides the string into two pieces.  After reaching the end of the string, the pieces are returned as a list of fragments.  For the example of splitting a sentence into words, the natural substring is the space (\lstinline{' '}).
\begin{pyin}
parent_string = "This book is the best book!"
\end{pyin}
\begin{pyin}
parent_string.split(' ')
\end{pyin}
\begin{pyout}
['This', 'book', 'is', 'the', 'best', 'book!']
\end{pyout}

\subsubsection{.splitlines()}

Perhaps your string covers multiple lines by inclusion of a new line character (\lstinline{\n}).  You can return a list of one-line fragments of the string using \lstinline{.splitlines()}:

\begin{pyin}
multiline_string = "This book is the best book!\nBut there are other very good ones too!"
\end{pyin}
\begin{pyin}
multiline_string.splitlines()
\end{pyin}
\begin{pyout}
['This book is the best book!', 'But there are other very good ones too!']
\end{pyout}
This is particularly helpful when you get a large block of text all at once from a file or a website.

\subsection{Analyzing strings to determine some information about their text}
\setcounter{subsubsection}{-1}  % Start subsections from 0

Sometimes, you don't actually want to change a string, just to know some information about it.  There are many helpful methods to do this, a few of which are mentioned here.

\subsubsection{Finding the first instance of a substring with \texttt{.index()}}

This function searches the string for a substring and returns the index of the beginning of the substring:

\begin{pyin}
parent_string.index("book")
\end{pyin}
\begin{pyout}
5
\end{pyout}
If the substring is not found, then a \lstinline{ValueError} is raised.

\subsubsection{.find()}

\lstinline{.find()} is basically the same as \lstinline{.index()}, with the only difference being that in \lstinline{.find()}, if the substring is not found, then the value \lstinline{-1} is returned.  This can be useful if you want to straightforwardly handle not finding a string.  Of course, you could handle the error thrown by \lstinline{.index()}, but sometimes that would be more work.

\subsubsection{Checking if the string can be converted to a different type with \texttt{.is<type>()}}

It is useful to check to see if you can do a type conversion before you actually try to do it.  It is also useful to check in a comparison expression what the implied type of a string is (basically, ``does this string represent a number?")  There are a series of methods to do this:

\begin{pyin}
'18'.isnumeric()
\end{pyin}
\begin{pyout}
True
\end{pyout}
Note this evaluates to false if your number has a decimal point!

\subsubsection{StringA in StringB}

If you just want to know if a substring is in a string, you can use the \lstinline{in} \cite{keyword}.

\begin{pyin}
"book" in parent_string
\end{pyin}
\begin{pyout}
True
\end{pyout}

\end{document}

% From data types:


