\documentclass[../book.tex]{subfiles}
\graphicspath{{\subfix{.}}}
\begin{document}

\chapter{Automating Basic Image Processing}
\label{15Pillow}

\setcounter{section}{-1}  % Start subsections from 0

\input{prereqs/15Pillow_prereqs}

\newpage
\section{Problem}
\setcounter{subsection}{-1}  % Start subsections from 0

The day you realize that word has really gotten around about your coding skills is the day that a friend of a friend walks into your lab looking for your help.  He works for a professor over in the Materials Science department and, while his project involves some chemistry, they are often dealing with images, rather than spectra. 

``Ummm..." he starts, ``So Kim tells me that you have been learning coding.  I was wondering if you could help me out."

He goes on to explain that he has been taking some thermal images of polymeric materials heated up by shining light on them. While the images show what they want, the software on their thermal imaging camera does not provide the coloring options they want. He asks if you know how to recolor images in Python. 

``Not yet!" you say, "But why don't you show me what you want, and I can do a bit of reading online and get back to you."

So, he opens up his laptop and shows you a few images.

\begin{figure}[H]
    \centering
    \includegraphics[width = 2in]{./Figures/inb_visual.jpg}
    \includegraphics[width = 2in]{./Figures/inb_thermal.jpg}    
    \caption{The images that the thermal camera outputs.  On the left is the `visual' image---sort of a `normal' picture.  On the right is the thermal image.}
    \label{fig:Pillow:CameraImages}
\end{figure}


``Basically, we have a greyscale photo of our setup, and a greyscale thermal image during the experiment.  The camera takes both, and can overlay them.  But the issue is that we don't want just greyscale colors, and the color options the camera provides are not great. For instance, it can provide a colorscale based on a rainbow, but we don't to use that colorscale.  It would be great if we could use a colorscale based around reds, yellows, or oranges, since people typically associate those colors with heat. Also, it would be nice if we could crop the images to remove the colorscale from the image.  We want just the part that is hot to be shown." He looks at you, clearly a bit nervous to be taking up your time. "Ummm... and yeah, I can do all of this in Photoshop, but it takes so much time for each image.  I have \textit{hundreds} of these images, and I don't want to lose an entire weekend to this. I know its not your problem, but if you could help out, then I could pay you back in beer or pizza or something."

``Well, let me do a bit of googling and see what I come up with. Don't worry about it too much, I am interested in learning to process images.  I'll let you know if you owe me anything after I see how long this takes."

\section{Solution}
\setcounter{subsection}{-1}  % Start subsections from 0

Even though you don't quite know how to do this in Python, you can still come up with a general plan of what you will need to do.  
\begin{enumerate}
    \item Import the image into Python, so you can work with it.  

    \item Change the image data to be the colors you want. 

    \item Crop the image.

    \item Save the image.
\end{enumerate}

\begin{wrapfigure}{r}{0.5\textwidth}
    \begin{mdframed}
        \textbf{Where to next?}
        
        \noindent Here, we use Pillow to work with images, because it is both simple and fast.  If you find yourself wanting to accomplish things that Pillow cannot, we recommend moving to OpenCV next. This is a very powerful library, but does have a more complex interface, and can be slower than Pillow. Thus, for simple tasks, the authors of this book turn to Pillow first. 
    \end{mdframed}
\end{wrapfigure}
So, really you only need to figure out how to do these 4 steps, and you are done. 

With just a small amount of searching on the internet,\footnote{As AI chatbots like ChatGPT emerge, they will likely  be a much better choice for finding this information.} you find that there are several different libraries that are dedicated to working with images in Python.  These include Pillow, OpenCV and scikit-image. Of these, it seems that Pillow is the simplest to use, and should be adequate for the tasks you want to accomplish. Also, it seems that Pillow directly handles steps 1, 3, and 4 above. 

Before you dive in, you need to understand how images are structured, and also introduce a few features of the Pillow library. 


\subsection{Understanding how computers represent images}
\label{Understanding_how_computers_represent_images}

The very first thing you realize about images is that there are two general types: \cite{scalable_vector_graphic}s and \cite{bitmap}s. 

Vector graphics break apart images into components and then represent each component via an equation that is solved in order to draw the image. For instance, if we wanted to draw a smiley face, we could include the equation for a large circle, two small circles, and an arc. Then we just place these in the appropriate relative positions, and we have the image we want.  The advantage of this approach is that our image can be scaled to any size, and the equations are the same, so that our image always looks crisp and sharp. The disadvantage is that not all images can be readily broken apart into such simple components.  For instance, consider a photograph of a person.  There is simply too many different shapes, colors, etc, to easily represent this as a series of equations.

Bitmaps represent images by breaking them down into small bits, called pixels, and then specifying the color of each of these pixels in turn.  For instance, imagine we had only black and white.  We could represent this as a series of 0s (black) and 1s (white). This provides explicit directions on how to draw our image. Thus, if we wanted to draw a smiley face (Figure \ref{fig:Pillow:Bitmap}), we could represent a series of black and white pixels in the shape of this face.  Already, you can see a disadvantage of this approach: it is hard to represent curves well.  In real implementation, we typically use much larger densities of pixels, and grays to make a curve look like a curve. Additionally, if we resize an image after making it, we cannot know how to color any new pixels that might occur, and so we will always preserve the blocky nature of our image.  The advantage, however, is that we can represent very complex images in a straightforward manner. 


\begin{figure}[H]
    %\centering
    % Left side: the image
    \hspace{3em}
    \begin{minipage}[b]{0.2\textwidth}
        %\centering
\raisebox{5em}{\includegraphics[width=1in]{{ChapNarrative/15Pillow/Figures/bitmapSmiley.png}}}
    \end{minipage}%
    % Right side: the array in a framed box
    \begin{minipage}[b]{0.3\textwidth}
        %\centering
        \begin{mdframed}[innerleftmargin=1em, innerrightmargin=1em, innertopmargin=5pt, innerbottommargin=5pt,
        linecolor = white]
            \footnotesize
            \texttt{[}\\[-0.em]
            \texttt{[1,1,1,1,1,1,\textbf{0,0,0,0},1,1,1,1,1,1],}\\[-0.em]
            \texttt{[1,1,1,1,\textbf{0,0},1,1,1,1,\textbf{0,0},1,1,1,1],}\\[-0.em]
            \texttt{[1,1,1,\textbf{0},1,1,1,1,1,1,1,1,\textbf{0},1,1,1],}\\[-0.em]
            \texttt{[1,1,\textbf{0},1,1,1,1,1,1,1,1,1,1,\textbf{0},1,1],}\\[-0.em]
            \texttt{[1,\textbf{0},1,1,1,1,1,1,1,1,1,1,1,1,\textbf{0},1],}\\[-0.em]
            \texttt{[1,\textbf{0},1,1,\textbf{0,0},1,1,1,1,\textbf{0,0},1,1,\textbf{0},1],}\\[-0.em]
            \texttt{[\textbf{0},1,1,1,\textbf{0,0},1,1,1,1,\textbf{0,0},1,1,1,\textbf{0}],}\\[-0.em]
            \texttt{[\textbf{0},1,1,1,1,1,1,1,1,1,1,1,1,1,1,\textbf{0}],}\\[-0.em]
            \texttt{[\textbf{0},1,1,1,1,1,1,1,1,1,1,1,1,1,1,\textbf{0}],}\\[-0.em]
            \texttt{[\textbf{0},1,1,1,1,1,1,1,1,1,1,1,1,1,1,\textbf{0}],}\\[-0.em]
            \texttt{[1,\textbf{0},1,1,\textbf{0},1,1,1,1,1,1,\textbf{0},1,1,\textbf{0},1],}\\[-0.em]
            \texttt{[1,\textbf{0},1,1,1,\textbf{0},1,1,1,1,\textbf{0},1,1,1,\textbf{0},1],}\\[-0.em]
            \texttt{[1,1,\textbf{0},1,1,1,\textbf{0,0,0,0},1,1,1,\textbf{0},1,1],}\\[-0.em]
            \texttt{[1,1,1,\textbf{0},1,1,1,1,1,1,1,1,\textbf{0},1,1,1],}\\[-0.em]
            \texttt{[1,1,1,1,\textbf{0,0},1,1,1,1,\textbf{0,0},1,1,1,1],}\\[-0.em]
            \texttt{[1,1,1,1,1,1,\textbf{0,0,0,0},1,1,1,1,1,1]}\\[-0.em]
            \texttt{]}
        \end{mdframed}
    \end{minipage}
    \hspace{0.1\textwidth}
    \begin{minipage}[b]{0.3\textwidth}
        %\centering
        \begin{mdframed}[innerleftmargin=1em, innerrightmargin=1em, innertopmargin=5pt, innerbottommargin=5pt,
        linecolor = white]
            \footnotesize
            % \texttt{[}\\[-0.em]
            % \texttt{[6,1,4,0,6,1],}\\[-0.em]
            % \texttt{[4,1,2,0,4,1,2,0,4,1],}\\[-0.em]
            % \texttt{[3,1,1,0,8,1,1,0,3,1],}\\[-0.em]
            % \texttt{[2,1,1,0,8,1,1,0,2,1],}\\[-0.em]
            % \texttt{[1,1,1,0,12,0,1,0,1,1],}\\[-0.em]
            % \texttt{[1,1,1,0,2,1,2,0,4,1,2,0,2,1,1,0,1,1],}\\[-0.em]
            % \texttt{[1,0,3,1,2,0,4,1,2,0,3,1,1,0]],}\\[-0.em]
            % \texttt{[1,0,14,1,1,0],}\\[-0.em]
            % \texttt{[1,0,14,1,1,0],}\\[-0.em]
            % \texttt{[1,0,14,1,1,0],}\\[-0.em]
            % \texttt{[1,1,1,0,2,1,1,0,6,1,1,0,2,1,1,1],}\\[-0.em]
            % \texttt{[1,1,1,0,3,1,1,0,4,1,1,0,3,1,1,0,1,1],}\\[-0.em]
            % \texttt{[2,1,1,0,3,1,4,0,3,1,1,0,2,1],}\\[-0.em]
            % \texttt{[3,1,1,0,8,1,1,0,3,1],}\\[-0.em]
            % \texttt{[4,1,2,0-,4,1,2,0,4,1],}\\[-0.em]
            % \texttt{[6,1,4,0,6,1]}\\[-0.em]
            % \texttt{]}

            \texttt{[}\\[-0.em]
            \texttt{[6,4,6],}\\[-0.em]
            \texttt{[4,2,4,2,4],}\\[-0.em]
            \texttt{[3,1,8,1,3],}\\[-0.em]
            \texttt{[2,1,8,1,2],}\\[-0.em]
            \texttt{[1,1,b,1,1],}\\[-0.em]
            \texttt{[1,1,2,2,4,2,2,1],}\\[-0.em]
            \texttt{[0,1,3,2,4,2,3,1]],}\\[-0.em]
            \texttt{[0,1,d,1],}\\[-0.em]
            \texttt{[0,1,d,1],}\\[-0.em]
            \texttt{[0,1,d,1],}\\[-0.em]
            \texttt{[1,1,2,1,6,1,2,1],}\\[-0.em]
            \texttt{[1,1,3,1,4,1,3,1,1,],}\\[-0.em]
            \texttt{[2,1,3,4,3,1,2],}\\[-0.em]
            \texttt{[3,1,8,1,3],}\\[-0.em]
            \texttt{[4,2,4,2,4],}\\[-0.em]
            \texttt{[6,4,6]}\\[-0.em]
            \texttt{]}
        \end{mdframed}
\end{minipage}
    \caption{A black and white bitmap image on the left, with a representation of this using arrays of 1 and 0s in the middle, and a the same data under a simple compression scheme on the right. To some extent, all bitmap images employ similar encoding.}
    \label{fig:Pillow:Bitmap}
\end{figure}


Pillow is designed to work with bitmaps.\footnote{If you want to work with vector graphics in Python, consider using svgwrite (for creating and modifying svg files) or cairosvg (for converting from svg to bitmap formats).} We are using Pillow in this chapter, because we are working with images taken with a camera, which are almost exclusively saved as bitmaps. This is important to realize, because knowing the type of image we have dictates the way we wish to work with it.  In particular, since bitmaps are just a series of pixels, we can represent the information in these pixels as a 2D array of numbers---in other words, Numpy arrays can be useful to use. 

Understanding that you are working with a bitmap that will ultimately be represented as a series of arrays is just the first step.  Next is to decide what bitmap format to use.  There are a large number of options, but the four most common are: tiff, bmp, png, and jpg (or jpeg). Of course, this begs another question: if bitmaps are just lists of numbers specifying the color of pixels, why do we have so many different formats. 

The short answer is that most bitmaps are not simple lists of pixel colors, because this can be quite wasteful of memory. Consider our bitmap of our smiley face (Figure \ref{fig:Pillow:Bitmap}), and imagine we were reading this row by row.  There are several places where we have multiple black spots that follow one another.  The top starts with 6 white pixels followed by 4 black ones.  One way to represent this is 1,1,1,1,1,1,0,0,0,0.  But another way might be [6,1,4,0], Where the first number tells us how many times to write the second number. Or you save even more room by knowing that there are only two options for color (0 and 1) so you could just list the numbers of each number, also starting with the number of 1s you need.  In this case, we would write this sequence as 6,4.  Looking at the image, we can see that there are instances where there are 12 or 14 numbers in a row, so using \cite{hexadecimal}, you can replace these with b, and d, respectively. For our smiley face, if we apply this final scheme, we get the result shown on the right of Figure \ref{fig:Pillow:Bitmap}. This is known as encoding.  When chosen properly, we can use a smaller amount of data to represent the same information---a concept known as compression. 

Our above encoding led to lossless compression---we produce the exact same image.  However, there are ways to encode that might lead to loss. For instance, what if we took a $3\times3$ grid and represented it by a single number: the dominant color?  That would lead to a smaller image, but one that is not accurate. We call this compression `lossy.'  

The different image formats use different compression encodings, and can have different amount of information loss.  tiff images are lossless, but are very large as  a result.  jpg images are small, but often lose too much information---especially in a scientific context.\footnote{Back when the internet was young, it was also very slow, and so passing graphical data could take a long time.  Thus, there was a premium on compression and people were more willing to accept loss of data.  This is why jpg files often look 'old.' } png files provide reasonable preservation of data, while making smallish files.  Thus, we will prefer png files here, though we note that tiff is nice to use when you really want to preserve everything, and this has long been the standard for serious graphic designers. However, no matter the format you use, we can always end up representing an image as a series of numbers.




\subsection{Importing and manipulating images using the Pillow library}
\label{Importing_and_manipulating_images_using_the_Pillow_library}

There are many modules in the Pillow library, but we are going to be able to accomplish everything using the \lstinline{Image} module. In particular, it provides five functions that we will use. 

\begin{itemize}
    \item \lstinline{Image.open()} accepts a \ul{positional argument} that is the path to an image file.  It returns an image object that can be manipulated in Python.  Thus, this function is the way that we gain access to the image. Once we have access to this image, we can do many things to it, like crop it, combine it with other images, or even get information on each pixel and convert that to Numpy arrays for further manipulation. 

    \item \lstinline{Image.fromarray()}  accepts a positional argument that is an array of numbers, and then converts these numbers into a Image object that can be further manipulated in Pillow. Combined with the \lstinline{Image.open()} function, we can see a convenient way to recolor images.  We can open the image object, get the pixel data (which is just color information) as a Numpy array, change the values of the entries in the Numpy array, and then convert this array into a image object.  This will effectively recolor our image. 

    \item \lstinline{Image.show()} is used to export an image object. The \lstinline{Image.show()} \cite{method} simply displays the image using the default output. It is a simple way to directly see the image you have created. Also, since it will open using a standard image viewer on your computer, you can save it from there.  If you wish to save directly, there is an \lstinline{Image.write()} function, though we will not use it below. 

    \item \lstinline{Image.blend()} is used to combine information from two different images.  It accepts two positional arguments (the two image objects to be combined) as well as several keyword arguments.  Here, we will use just one: \lstinline{alpha} which controls how opaque the second image is. 

    \item \lstinline{crop()}, which is a method of an image object.  This allows us to control how much we wish to remove from the left, top, right, and bottom parts of an image. We can use this to remove the parts that are not the focus of our image. 
\end{itemize}






\bigskip
And with this information, you realize you have all the information you need to recolor an image in Python! 



\section{Code}
\setcounter{subsection}{-1}  % Start subsections from 0

\begin{figure}[H]
    \centering
    \includegraphics[]{ChapNarrative/15Pillow/Figures/FlowChartCh16.pdf}
    \caption{Flowchart for code that reads in the content of bitmap images, recolors the image, and then saves the recolored images.}
    \label{fig:Pillow:ActivityDiagram}
\end{figure}

\lstinputlisting[language = Python, lastline = 56]{../../Distribution/Code by Chapter/Ch16 - Automating Basic Image Processing/ThermalRecoloring.py}

You run this code, select the thermal file `inb\_thermal.png' in your Thermal Imaging folder, select teh file `inb\_visual.png' in the same folder, and the following image appears on your screen: 

\begin{figure}[H]
    \centering
    \includegraphics[width = 2 in]{ChapNarrative/15Pillow/Figures/inb_recolored.png}
    \caption{Results of running the code in this chapter.}
    \label{fig:Pillow:Recolored}
\end{figure}


You see that you can easily build the capabilities to select several different files and perform the same action on all of them.  So, at this point, the problem is solved. 

You show this solution to your friend, who says, ``Wow, that is amazing! I really wish I knew how to do this."

``Tell you what," you say, hinking of the favor \first did for you, ``I am free over lunch, so if you wanted to take an hour, I could give you a quick introduction to coding and get you to the point that we can start solving real problems for you."

\subsection{Line-by-line explanation}

\begin{footnotesize}
\begin{itemize}
    \item [7--9:] Libraries we will need for this code. 

    \item [12:] We are going to need to build a new color scale, held in a list. So we create an empty list that will contain specifications for the colors we wish to map the greyscale onto when re-coloring.

    \item [13--19:] We are going to use three nested \citett{for} \cite{loops} to build our color scale. Each entry in our color scale has three values, for red, green, and blue.  The plan is to start at black [0, 0, 0], then go through to pure red [255, 0, 0], then through to pure yellow [255, 255, 0], and then to white [255, 255, 255]. This will give the impression of generally increasing the brightness uniformly. 

    \item [13:] We start with a for loop that will iterate over the numbers 0 through 2.  This will let us understand which index in our color specification we are currently increasing. 

    \begin{itemize}
        \item [14:] We now immediately start the next loop and iterate through the numbers 0--255. These are the numbers we are increasing.

        \begin{itemize}
            \item [15:] We start with black [0, 0, 0].

            \item [16 \& 17:] We start the final loop. Here we iterate from 0 to the index \textit{prior} to the one that we are currently increasing. Within this loop, we simply set all the values we are not currently increasing to their maximum (255).  The reason is that we have already increased them to their maximum. 

            \item [18:] We then change the value for the index we are increasing to the value we are currently on. 

            \item [19:] We then append this color to our list of colors.   Once we complete all these loops, we will have a list of $3\times256$ items long, which specifies smoothly changing colors from red, through yellow, to white. 
        \end{itemize}
    \end{itemize}

    \item [22:] We prompt the user with a popup message that tells them we want them to identify a thermal image (not a visual one).
    
    \item [23:] We then use \lstinline{quickOpenFilename()} to let the user point to the file we will use.  We use the \lstinline{filetypes} keyword to specify that the filename \textit{must} end in '\_thermal.jpg' which will ensure they do not open a visual image---provided the user is following the naming convention we expect. We then use the path object returned by this function and open it and pass the resulting Image object to a variable. 

    \item [24:] We convert the Image object into a Numpy array.  This will create a 2D array that contains a number of interior arrays equal to the number of pixels tall the image is, and each interior array is as long as the number of pixels wide the image is. 

    \item [27:] We create a 2D Numpy array that is exactly the same size as the one we made in line 24, however, we set all the values initially to zero. We assign this array to a variable. This is the array that we will recolor. 

    \item [30:] We use the \lstinline{shape()} method of Numpy arrays to get the numbers of inner arrays, and the number of elements in each inner array of the 2D array. This corresponds to the number of pixels tall and the number of pixels wide the image will be. We have to use a \cite{slice} from 0 to 2, because each pixel is represented by an element of the inner arrays, which is itself a 3-item array (for rgb values).  Thus, it is only the first two shape values that we want (just the number of inner arrays and the number of elements in them). 

    \item [33--36:] We start the process of recoloring the image by iterating over numbers associated with the width of the image. 
    \begin{itemize}
        \item [34:] We then start another for loop where we iterate over numbers associated with the height of the image. 
        \begin{itemize}
            \item [35:] Using the indices, we check the color for a pixel in the original image.  The greyscale colors are specified by a list [r, g, b], but in which r=g=b, so we only need one of these values.  We take the first one. 

            \item [36:] We then specify the color for the new thermal image at the same pixel  We set the color equal to the value in our \lstinline{scale_bar} list.  We simply pull the value at the index associated with $3\times$ the greyscale value.  The reason is that the \lstinline{scale_bar} list is three times longer than the larger than a list associated with greyscale colors.  Thus, we are simply re-scaling the greyscale onto our color scale. Once we complete the list, we will have a Numpy array that carries all the colors we want for our new thermal image. 
        \end{itemize}
    \end{itemize}

    \item [39:] We use the \lstinline{fromarray()} function from the \lstinline{Image} module of Pillow to create a image object from the 2D Numpy array we just constructed above. 

    \item [42:] We now prompt the user to select the visual image that we wish to combine with the recolored thermal image. 

    \item [43:] We then use the same approach as on line 23 to get the path to the visual image, this time restricting the files so that they must end in '\_visual.jpg', and then pass the path to a function that opens the image and pass the Image object to a variable. 

    \item [44:] We now overlay the two images, using the \lstinline{blend} function from the \lstinline{Image} module of Pillow. This accepts two positional arguments, which are the images to be blended, and we supply a keyword argument that specifies how opaque the second image should be. Here we set it to 30\% opaque (70\% transparent). The result is as if the second image was laid on top of the first, at that opacity. 

    \item [45--48:] We need to crop our image, using the \lstinline{crop} method of the image object. This accepts 4 positional arguments, that give the amount (in pixels) to crop the left, top, right, and bottom sides, respectively.  It can be hard to know exactly how much to crop any given image in pixels, so we specify the fraction of the width and height we wish to crop each side, which we think is easier to estimate when looking at an image.

    \item [51:] Finally, we show the image. This should bring it up in a new window on your computer. After it is shown, we can save it, or do anything else with it. 
\end{itemize}
\end{footnotesize}


\hrule
\input{Exercises/Chapter 16 Pillow}
\hrule

\end{document}
