\documentclass[../../book.tex]{subfiles}
\graphicspath{{\subfix{.}}}
\begin{document}

\chapter{Useful, popular, and alternative libraries}
\label{Libraries}
\setcounter{section}{-1}  % Start subsections from 0


One of the nice things about Python is that there are a large number of libraries that can be used to extend the base Python.  This means that there are number of ready-made solutions to both common and niche problems.  For instance, we introduced lmfit as a library to easily accomplish fitting lines to data, while for those who need to work with old NMR data, the package NMRglue provides the means to convert between Varian and Brucker formats. And, as we pointed out, you can even write your own libraries to create solutions to your particular problems. 

The flipside of this situation is that there are can be multiple solutions to the same problem, and so a Python user has to decide which of these solutions to use. For instance, when thinking about plotting data, one can choose between Matplotlib, Plotly, Bokeh, Seaborn, etc.  And these are just the popular choices!

The point of this chapter is to briefly introduce a few libraries that we think are work considering, and also to mention some of the popular libraries that we do not find ourselves using. This is not to say that we think you need to adopt our opinions, but just to provide some information to help orient yourselves. We present them in rough grouping by the type of task they enable. 



\section{Handling scientific data}
\setcounter{subsection}{-1}  % Start subsections from 0

\subsection{Numpy}
This is the primary library we use to interface with data in our book. The main advantage of numpy is its vectorization of data. Though other libraries also do this (see below), they also add other ideas about how data should be used and how it should be structured. Thus, while they add capabilities, they also are a bit less flexible.  In our opinion, we think that the flexibility of Numpy is a great advantage, and this is why we use it exclusively. 

\subsection{Pandas}

If you search on how to use Pandas online, or if you use tools like ChatGPT, it will almost certainly suggest using Pandas.  Pandas is built on Numpy arrays, but adds additional capabilities. Perhaps the most readily apparent is that it structures data into 'data frames,' which are essentially Numpy arrays that are grouped together.  If you envision the data frame as a spreadsheet (much like what you see in Microsoft Excel), then the Numpy arrays are the columns of this sheet. 

This arrangement of data is familiar and comforting to many, but it can also be restrictive. Pandas wants all columns to be the same length, which can be inconvenient for much of the data a chemist will encounter. This is the primary reason we do not use it in this book.  However, it is a reasonable library to use, if you wish to explore it on your own. 


\subsection{Polars}

Polars is sort of a successor to Pandas. Pandas was invented to handle 'big data' coming from databases. However, the reading in of this data can be quite slow, and so Polars attempts to solve this (and other issues).  It is a more modern library and so you might find less examples online, but it is certainly an emerging library and one to keep an eye on, if you are going to work with data. In this book, we don't use it for the same reasons given for Pandas. 


\subsection{scipy}

Scipy is a very large library, containing 18 modules, most with their own sub-modules.  Here, we will consider the 9 modules that we think you might be most interested in. 


\begin{itemize}
    \item \textbf{scipy.constants:} 
    This provides access to physical and mathematical constants. Using them has two major advantages: you don't need to remember them and you will not make an error in typing in your own. The disadvantage is that you are stuck using hte units that scipy has chosen, which may not be the units you are used to, so you will still need to convert them. However, we believe the advantages outweight the disadvantages. It is also worth noting that a few constants (like pi) are also within numpy.  Scipy provides a much more exhaustive list.

    \item \textbf{scipy.fft:}
    This provides access to many different fast Fourier transfer functions (and their inverse transforms), ranging from 1-dimensional to N-dimensional. This can be useful for processing interferograms or creating noise filters. 


    \item \textbf{scipy.integrate:}
    This includes routines for integration (single, double, etc) as well as solvers for ordinary differential equations (ODE). Thus, you will find this module useful when you are attempting tasks such as integrating signals in a spectrum, or when you are solving for complex chemical kinetics. 

    \item \textbf{scipy.interpolate:} 
    This provides a very large number of routines for interpolating data---that is providing guesses for dependent variables, at values of independent variables that were not actually measured/collected. This can be useful when trying to directly compare, add, subtract, etc. data that was collected at different values of independent variables. However, there are two things to be aware of: (1) the values provided are only guesses, and the quality of these guesses are dependent on the data and interpolation routine used and (ii) many of the interpolation routines provide smoothing of data as a result. Nevertheless, there are times that you will wish to use interpolation. 

    %\item \textbf{scipy.io:} Input and output, including file formats.

    \item \textbf{scipy.linalg:} Linear algebra routines.
    Many mathematical operations are most efficiently handled using linear algebra operations, such as vector multiplication and matrix multiplication. If you find yourself needing to handle such operations, then scipy provides routines for them, and many more.  While Numpy also has linear algebra routines, scipy contains all of those, adds more, and is often times faster.  So we recomend starting with scipy, when you need to do linear algebra. 

    \item \textbf{scipy.ndimage:} N-dimensional image processing.
    In chapter \ref{15Pillow} of this book, we used the Pillow library in order to process and change a 2D image. While the Pillow library is useful for such manipulations, it requires the image can be represented as a 2D array of values (i.e., colors on an x-y grid). However, there are several types of data where images have much more information per pixel than a simple color.  For instance, a Raman map of a material has a Raman spectrum associated with each pixel.  Thus, each pixel has more dimensions than a simple color value.  We call such images n-dimensional. This scipy module provides capabilities for processing such images. 
    

    \item \textbf{scipy.optimize:} A large number of problems in science come down to optimizations. For instance, when fitting equations to data, as handled in Chapters \ref{04EqFit}, \ref{05CompFit}, and \ref{06AnalFit}, we find the best fit through a least squares routine, where we optimize parameter values of the equations such that we find the smallest possible sum of squares of residuals between the equation and the data. This module provides the fundamental routines needed for such work, as well as many others.  Indeed, the lmfit module that we recommend using to fit equations to data is just a \ul{wrapper} for some of the functions in the scipy.optimize module.  

    
    
    \item \textbf{scipy.signal:} 
    This module contains functions that are designed for smoothing, filtering, and sampling data.  Additionally, it has functions for fast Fourier transforms and convolutions. We don't anticipate that many readers will wish to use such behaviors, but if you do need them, this is the first place to turn to. 

    \item \textbf{scipy.stats:}
    This module contains many functionalities that we think are best found in other modules.  For instance, scipy has a random number generator, but you can also use the one from numpy, since you are most likely importing numpy anyways.  Also, this module has information on probablility distribution functions, that one could use to fit data---but lmfit also has this information in a more convienent means for fitting. 
    
    However, there are some functionalities that are worth using.  For instance, scipy.stats provides built in T-tests and KS-tests for significance.  Thus, for those wishing to test significance, this is a nice way to do so. 
    
\end{itemize}
 






\subsection{sympy}
This is a symbolic math library, meaning that it provides the means to write out equations, and then solve for variables in those equations (i.e., you are using symbols in your math). This can be a very convenient means to carry forward results from one area of coding into a mathematical expression.  For instance, you could use lmfit in order to solve fit an equation to data, and extract a rate constant from the fit.  You could then take that rate constant and plug it into a series of kinetic equations written in sympy, in order to solve for the concentration at a given time, for a different reaction, assuming the same rate constant. 

\subsection{lmfit}
This is the fitting library that we recommend using, and its use was described in detail in Chapters \ref{04EqFit}, \ref{05CompFit}, and \ref{06AnalFit}. 

\subsection{symfit}
Symfit is another option for fitting data. It tries to combine the power of lmfit, with the simplicity of symbolic math.  Thus, one can write out equations that you wish to fit in a symbolic manner, and then execute the fit using commands that look a lot like those found in lmfit.  We think this is a library that is very much worth looking into, however, it is somewhat new and still being developed.  The more robust knowledge base present for lmfit is why we current recommend it over symfit. 

\section{Plotting scientific data}
\setcounter{subsection}{-1}  % Start subsections from 0

\subsection{Plotly}

Since this book exclusively use Plotly, we will not describe the package in detail.  However, it is worth re-iterating the aspects that we like: it is object oriented, it creates a persistent object, the out-of-the-box interactivity is nice, and it can produce static images, like pngs and svgs.

Some frustrations are that it can be had to format text exactly as you might like, and it does rely on external libraries to produce the static svg and pngs. 

\subsection{Matplotlib}
This is, by far, the most common library for plotting in Python.  There are many things to recommend it: it is fast, possess the ability to make a wide range of 2D and 3D plots, there a large set of examples and tutorials on Matplotlib, and it is simpler to get working with Python.  However, there are also a few disadvantages that have kept us for recommending matplotlib. First, we think the interactivity is not as nice as Plotly presents. Second, the matplotlib figure object cannot be accessed once it is shown.  Whereas, one can create a Plotly figure, show it, add to it, and then show the updated plot, this cannot be done in Matplotlib. Finally, the Plotly figure object is easier to understand and access.  If desired, one can directly access and change settings for a figure within the dictionary figure object in Plotly.  This is not really possible for Matplotlib figure objects. For these reasons, we have choosen Plotly over Matplotlib. 


\subsection{Seaborn}
Seaborn is another popular plotting library, but it is built upon Matplotlib, and so the same disadvantages of Matplotlib are also present in Seaborn. 

\subsection{Bokeh}
This is an emerging plotting library that offers quicker and even better interactivity than Plotly.  However, a major disadvantage is that it is harder to get the static plots that a practicing scientist will need for publications and reports.  Thus, for those that anticipate publishing their data visualizations, we believe Plotly still offers the best marriage of making good static plots and offering a strong interactive experience. 




\section{Chemistry}
\setcounter{subsection}{-1}  % Start subsections from 0

Since this book is written for the practicing chemist, we have some recommendations that will make many of the tasks that are specific to chemistry simple to accomplish in Python.


\subsection{rdkit}
Introduced in Chapter \ref{Sec:N:03DrawMol:rdkit}, this library can process SMILES strings for chemicals and produce drawn structures. However, rdkit also has some basic chemiformatics information and routines built in.  So, if you are ever attempting cheminformatics, this is a library to consider. 

\subsection{mendeleev}
As the name implies, this library provides a convenient means to access information from the periodic table in Python. For instance, imagine you wanted to know the atomic mass of oxygen, you could use:

\begin{pyin}
from mendeleev import element
\end{pyin}
\begin{pyin}
element('Oxygen').atomic_weight
\end{pyin}
And there are a host of other properties that are availible, such as molecular volume, electron configuration, boiling point, hardness, etc. 

\subsection{chempy}
In many ways, chempy might feel like rdkit meets mendeleev.  Similar to rdkit, chempy contains functions that provide useful analysis---but is focused on things like Arrhenius behaviors, and other chemical reaction phenomena.  Similar to mendeleev, chempy has a store of information, but on collections of molecules, rather than atoms.  Thus, you can get information like the density or viscosity of molecular liquids. 

\subsection{pycroscopy}
While general packages exist for processing images, there are number of manipulations that are common to microscopy images, and these are brought together in this package. For instance, this provides built-in functions that will take microscopy images and give you a histogram of the pixel values inside it.  




\subsection{qmsolve}
For experimental chemists that are working more on the physical chemistry side of this, this library provides functions for solving and visualizing the solutions to the Schrodinger equation for systems of interest. This provides this functionality for both time-independent and time-dependent treatments, and so can be used to help understand the behavior of systems in both ground and non-equilibrium states. 


\subsection{NMRglue}
This library provides the ability to process raw data from NMR spectrometers.  This can be done for modern instruments (i.e., from Bruker) and legacy instruments (i.e., from Varian, etc.). Thus, for old data, this might be the only viable way to work it up. However, even for modern data, the ability to import raw data, process it, and present it all programatically within Python could have significant value.  For instance, for people that takes hundreds of NMRs each week, such a use case could easily save many hours per week. 


\subsection{Computational crystallography toolbox}
This library provides a number of tools for processing and simulating light scattering experiments, and building and representing crystal structures. We note that the documentation is rather obtuse and perhaps hard to parse.  But for those with a need and some willingness to parse the library (perhaps with the aid of AI tools like ChatGPT-4), this library could also save significant time.  




\section{Computer interface}
\setcounter{subsection}{-1}  % Start subsections from 0

A common theme of coding is that you will need to interface with the computer---entering commands, accessing files, etc. There are a number of packages that are designed to facilitate this. 


\subsection{pathlib}
This is the library that we recommend using to navigate file systems on computers. The advantages are that it is built-in to Python, it is able to easily parse file structure information, and it can handle moving easily between different operating systems. Thus, this provides an accessible, simple, and robust means to handle file structures. This was first introduced in Chapter \ref{01OnePlot}, and used throughout this book.

\subsection{tk}
This is the built-in GUI module for Python. We introduced it in Chapter \ref{02ManyPlots}, as a means to pop up a file-selection GUI interface. However, there is much more that it can do.  Essentially, tk provides the means to create native-looking GUI apps. Thus, anytime you use a Python app that has a native-looking GUI, there is a reasonable chance it was built with tk. Though we do not provide instruction beyond file-selection, there are a number of tutorials online on how to use tk

\subsection{sysimplegui}
This module provides another way to create a GUI interface for your code. An advantage of this, is that it is rather straightforward to create extensive GUIs.  However, it is not a built-in Python library, and it can prove challenging to create complex interfaces.  Furthermore, the future of pysimplegui is not clear to us, as it may be leaving active development.  Thus, we believe it is best to use tk. 

\subsection{streamlit}
This is one way that you can create a stand-along GUI application.  Essentially, you create a Python code, then you use Streamlit to provide a GUI interface for it, and then you host this on a website, using a server to do so. Someone can then access your application and run it from that server.  If you are not comfortable hosting your own server, then the Streamlit community provides free hosting.  For this reason, this is probably the simplest way to get a stand-alone Python GUI-based application up and running. However, we do note that the current implementation of Streamlit requires that the entire Python code run for each button press or input in the GUI.  For this reason, applications that have extensive code can be a bit slow. This is planned to change in the future, but for now this is a limitation worth considering. 

\subsection{dash}
Dash is a bit like Streamlit, but native to Plotly. Essentially, if you have a program that is all about plotting and manipulating data and you are already going to use Plotly, then you can use Dash to provide the GUI interface for it. It has the advantage over Streamlit, that the entire program does not need to run for each input---just part of the program can be updated.  However, Dash apps do need to be hosted by a server and there is not a free hosting service for this.  Thus, deploying the app can be harder. 

\subsection{pyodide}
Pyiodide is a web-assembly means of creating stand-along apps.  What this means is that a web browser reads a Python script and constructs an instance of Python to run it, all on the user's local machine. In other words, there is no need to host the app.  Additionally, because it is a web browser that is running the code, it is straightforward to only invoke the aspects of Python that you need to run for any given input---thus, the program can be relatively fast. The disadvantage is that you need to know a bit about html and javascript. 





\section{File read/write}
\setcounter{subsection}{-1}  % Start subsections from 0

When reading and writing files, there are going to be times you need to handle specific formats, and there are a number of libraries that are availible for a wide-range of file- and data-types. 

\subsection{docx}
We introduced this library in Chapter \ref{08Report}, and it is the library we recommend for working with Microsoft Word files. In this book, we showed how to write titles, paragraphs, and tables, as well as format text.  However, this is more than can be done, just as handling sectioning, table of contents, etc. While we don't think that you will want to write large Word Documents using Python, it is possible. 

\subsection{openpyxl}
We introduced this library in Chapter \ref{09Excel}, and it is the library we recommend for working with Microsoft Excel files.  We only showed a small fraction of what is possible, using this library and we do think it is possible to work entirely within Python, and then generate Excel Files are a means of archiving and sharing data. This may not be the absolute best practice, but is one that will likely serve many researchers well---especially if you need to interface with people that do not know coding, but use Excel instead. 

\subsection{pillow}
We introduced this library in Chapter \ref{15Pillow} as a recommended library for working with images. This is a nice library for relatively simple image manipulations, such as combining images, resizing images, cropping images, recoloring images, and changing bitmap file formats. For more complex image analysis we think opencv (see next) is more capable. 

\subsection{opencv}
An alternative to the pillow library for working with images. We think the syntax and usage is a bit more complex, and the manipulations can be slower than in pillow, however, there are many more capabilities present in opencv.  For instance, opencv can be used to identify features in an image, fit shapes to these features or encode metadata. 

\subsection{regular expressions}
Though not used in this book, regular expressions is a powerful way to parse text. In this book all the text parsing could be accomplished using the \lstinline{split()} method of strings, but this requires knowing the pattern you are splitting on. This is fine when you only need to recognize a pattern like "','".  However, what if you wanted to find a number that was positive, and had any number of digits before a decimal place, and only 4 digits afterwards?  You can specify a complex pattern like this in regular expressions. 

\subsection{requests}
If you want to pull information from a website, the way this is done is to request the information from the websites server. The requests library manages this process. Thus, if you are going to be trying to pull data from the web, this is library you will need to use. 

\subsection{json}
A common data format that will be returned from requests to a web server is the json format.  This is a format that is a bit like a dictionary, in that it contains key/value pairs, but the structure and syntax is slightly different the json library provides methods for parsing this data structure, and even for converting from json to Python libraries, or the other way around. 

\subsection{beautifulsoup}
Another way to get information from the internet is to go to a page, and then get the HTML code for that page. This code can be hard to parse for humans, and this library can do this parsing for you---making it easier to find and extract the information you are after. This process of acquiring the HTML and getting the information from it is known as `web scraping.'








\end{document}