\documentclass[../book.tex]{subfiles}
\graphicspath{{\subfix{.}}}
\begin{document}

\chapter{Thinking about Your Next Steps}
\label{16Beyond}

% prereqs here
\setcounter{subsection}{-1}  % Start subsections from 0

\newpage
It is a bittersweet moment in the lab. \First has successfully defended their thesis, which you are very happy about. They are also moving on to their dream job, which you are ecstatic about. But this also means that they will no longer be a desk away, and so asking them questions about Python any time you like is about to come to an end. 

On the walk back from a celebratory lunch, \first makes sure to find you. ``Don't worry too much, you have have already learned so much about programming!  I even heard you were solving other people's problems now."

You agree, but you also feel there is still so much to learn and you aren't even sure what might be worth learning next. You already know how to read data, plot data, fit data, simulate data, record data, and work with common file formats.  You know how to integrate, smooth, and interpolate data, as well as process images. Thinking through your normal workflow in lab, it isn't clear what else you might like to learn. 

``Oh, I can appreciate that! But just keep your eyes open.  Every time you find yourself doing a task on a computer that you wish \textit{you} didn't have to do, think about if you could automate it with Python.  Every time you think of something you wish you could do with your data, but don't know how, look online for other people doing that thing using Python. And if the solutions you are finding aren't quite clear immediately, that's ok.  Think back to learning Python for the first time.  You were confused, but then were able to figure things out. Just keep doing that.  And we can always stay in touch!  If I find something I think you might like to know about, I can give you a call or sned you an email.''

The next Monday, you come into lab, and \first is gone, but there is an email from them in your inbox.  It reads:

``Hey! I am just getting settled in, but on the drive to my new place, I thought of a few things you might like to consider learning about.  Here they are!''

You smile, thinking about how fortunate you were to have \first as a friend and mentor, and start reading the email...

\section{Managing research projects}

\subsection{Implementing a reasonable data management plan}
\label{Implementing_a_reasonable_data_management_plan}
This is not strictly a Python problem, but it is worth thinking about, because it will impact your Python workflows.  Data management refers to how you will store your data, organize it, and ensure that you know where it came from and how it has been modified. 

One way to do this is to establish a consistent file name convention, which might specify the date data was taken, the instrument that recorded it, and other useful information (e.g., temperature). The advantage of this approach is that the name is linked to the file.  The disadvantage is that you end up with very long file names, which can be hard to read and work with.

If your data is held in .\cite{csv} files or Excel files, you can add in \cite{headers} to these files that will explain the nature of the data. This is nice because you can use shorter file names.  But a disadvantage is that it is likely that not all the data you collect will come in as csv or Excel files, so you will need to convert your data to a format where you can add in arbitrary headers.

Yet another option is to keep data in folders, and then include a \cite{plain_text} file in the folder that explains all the data. This kind of documentation can work with any sort of data file. However, it is relatively easy to separate the file with the descriptions from the data files, and you have to remember to update the description when you add or change data. 

Finally, you can use software that is designed to help maintain data in a manner where you know what it is, and even if it was changed. Two relatively popular means to do so right now is to use GitHub or create a database. These are discussed next, in turn.

%github
\subsection{Managing coding projects with GitHub}
\label{Managing_coding_projects_with_github}
If you change a the data in a file, what do you do about it? The best practice is to make a note of the changes, so you can back track, if need be. However, as you work with lots of data, or perform many different transformations, this can become cumbersome.  One way that people default to is to make multiple copies of the same file, giving it successively longer and longer names to keep track of the changes.  Anyone how has received a file that ends in '...final final FINAL4\_final' has seen the results of such practices.  This is perhaps the \textit{worst} way to document changes. A much better practice is to keep a single file, but keep track of the changes you have made to it. Perhaps by growing a longer and longer header...

There is a better approach though: Git. The most popular and accessible implementation of this is GitHub: an online repository that can be used to store and share code and data, and also manages \cite{version_control}.  That is, each time you make a change, GitHub asks you to describe what you did, and then remembers what was changed. This means that you, and anyone else that you give access too, has a complete record of the history of your code. This is a wonderful way to manage code and data---especially when working across multiple devices or with multiple people. 

GitHub is just one implementation of Git, and you can create your own, but GitHub has a relatively easy interface to learn, and there are lots of tutorials out there on how to use it. See the book's website (\website) for links to resources on learning Git. 

% Databases
\subsection{Organizing your data in databases}
\label{Organizing_your_data_in_databases}

Even if you are a pretty organized person, it can be easy to end up with so much data that you lose track of it. And if you wanted to keep track of data from other people in the lab, this can be quite cumbersome.  

There is a tool invented for managing large amounts of information: the database.  You can store \cite{objects} in this database with information about who took it, what project it was for, what instrument it was taken on, experimental conditions, etc. and all of this can be searchable. Thus, this is a nice way to organize data.  

Python provides many ways to build, maintain, and interact with these databases, so this may be of interest.  Some of the more popular libraries to explore are: sql-lite and duck-db.  It will take some thought to consider how you want to set up a database, but you might find it worth your time. 




\subsection{Ensuring your data meet FAIR data standards}
\label{Ensuring_your_data_meet_FAIR_data_standards}
FAIR data standards are a set of principles designed to ensure that scientific data is \ul{F}indable, \ul{A}ccessible, \ul{I}nteroperable, and \ul{R}eusable. Developed by a broad community, FAIR standards are guidelines to help researchers manage data effectively, enabling collaboration, reproducibility, and long-term usability by other researchers. Since you are now in control of the manner of production and the format of your data, you have an opportunity to increase its impact by following these guidelines.  Briefly, \textit{findability} ensures that data is well-described with metadata and assigned unique identifiers. \textit{Accessibility} promotes open sharing while respecting security and legal constraints. \textit{Interoperability} ensures compatibility across different software and formats through standardized vocabularies and data structures. \textit{Reusability} mandates clear licensing, provenance tracking, and adherence to community standards, allowing data to be repurposed for new research. You can find out more by visiting the FAIR website, \href{https://www.go-fair.org/}{https://www.go-fair.org/}.

\section{Incorporating advanced analysis}

\subsection{Implementing more advanced data analysis techniques}
\label{Implementing_more_advanced_data_analysis_techniques}
We have touched on some of the more standard approaches to data analysis and manipulation in our book---things like integration, peak finding, fitting, etc.  But there is so much more that could be done. Once you have gathered a significant amount of data, you can search for patterns and meaning in it using approaches like ANOVA (analysis of variance), principle component analysis, etc. Python has many libraries to help accomplish these tasks. There are too many approaches to list here, but a good approach to starting might be to read a basic college-level statistics book, and then search for how to do the analysis presented there in Python. Or you might consider reading a book on this topic, such as: \textit{Practical Statistics for Data Scientists: 50+ Essential Concepts Using R and Python,} by by Peter Bruce, Andrew Bruce, and Peter Gedeck. Certainly you now have sufficient understanding of Python to read this book, and ones like it.

% your own libraries 
\subsection{Making your own libraries}
\label{Making_your_own_libraries}
Rather than having to add a local folder paths to your \acite{PythonPATH} or use the functionality from the codechembook libarary to incorporate external code, you can take code you wish to re-use, package it up, and then add it to places like conda-forge or PyPi.  If you do this, then it becomes installable exactly the way you have been installing other \acite{libraries}, which means that it is easier for you to access in other scripts and other people can use it to.  If you have encountered a problem that benefited from making a code solution, maybe other people will find it useful as well. That is, in part, how the codechembook library started.

While it might seem daunting to create your own libraries, they are really just .py files with code, and then a few other \cite{plain_text} files that specify details on how Python should use the library.  If you have gone through this book, you certainly have the ability to learn how to make your own library. A good resource to get started is: \href{https://realpython.com/pypi-publish-python-package/}{https://realpython.com/pypi-publish-python-package/}

%machine learning
\subsection{Incorporating machine learning in your workflow}
\label{Incorporating_machine_learning_in_your_workflow}
A huge amount of the emerging machine learning software uses Python as the interface for giving commands. Thus, if you want to learn how to use machine learning, you are now in a position to start using these interfaces. The landscape is evolving so quickly that it is not worth trying to provide a specific material, but some web searches should get you going in the correct direction. We will try to keep a list of resources we like at the book's website: \website

% APIS
\subsection{Interfacing with hardware and software through vendor-provided APIs}
\label{Interfacing_with_hardware_and_software_through_vendor-provided_APIs}
API is an acronym for \ul{A}pplication-\ul{P}rogrammer \ul{I}nterface.  They are libraries, methods, etc that allow for a programmer to interface with other software.  In this book, we used APIs for external software like Microsoft Excel.  But there are many other API libraries out there, so if you have software you want to use, and are wondering if you can use it via Python, search the web! You can find APIs for many scientific instruments and software, e.g. for making PDF files and interacting with other spreadsheet software like Google Sheets or Microsoft Powerpoint. 

You will also find that Python is so widely used that many other programs are written in it, or natively provide APIs for the program.  For instance, one of the best 3D rendering programs, Blender, allows interfacing to it using Python.  Thus, if you learn this API, you can programmatically assemble a scene for creating very pretty drawings of your molecules, all from Python. 


% web scaping$
\subsection{Collecting data from the internet \textit{via} scraping}
\label{Collecting_data_from_the_internet_via_scraping}
There is a huge amount of information compiled into pages and databases on the web. There is pricing on chemicals, authors of papers you like, stock market information, etc.  This can be obtained by `visiting' the website using Python and pulling the information back into your code to be parsed.  This is known as web scraping, and Python provides several libraries for such things. There are also always multiple tutorials being written on this subject on the web, so you can also just search for this topic, and find the most up-to-date approaches. At the time of this writing, looking for tutorials on using the library `beautiful soup' would provide nice resources. 


\subsection{Implementing ChemInformatics workflows}
\label{Implementing_ChemInformatics_workflows}
One area not addressed in this book was ChemInformatics. But there are many libraries for such practices. See the excellent ACS InFocus book \textit{Python for Chemists} for details on applications of Python to ChemInformatics problems. These libraries provide tools for processing data about molecules and chemical reactions, and this book provides a great place to get started. 

\section{Expanding your coding toolset}

% GPT
\subsection{Taking advantage of large language mode based assistants like ChatGPT}
\label{Taking_advantage_of_large_language_model_based_assistants_like_ChatGPT}
Speaking of machine learning, there are also tools that can help you code and learn to write code. Currently, ChatGPT is the most well-known, but others exist, like Gemini, Microsoft (or GitHub) Copilot, Anthropic Claude, etc. Most of these models do an admirable job producing code, though they are not yet reliable or capable of producing complex solutions without supervision so you really do need to have basic coding skills to efficiently use these tools. Fortunately for you, you now have these skills.  When using AI coding tools, don't forget to not only ask for code examples, but to ask questions about how these code examples work.  Used this way, these tools have an amazing potential to hasten your learning and speed up code development. 

\subsection{Understanding the strengths and weaknesses of other IDEs}
\label{Understanding_the_strengths_and_weaknesses_of_other_IDEs}
Though this book focused on the Spyder IDE, which think is likely the best choice for basic chemical workflows, there are large number of other IDEs out there, each offering some advantages.  Probably the best-known is VScode, which is a free IDE maintained by Microsoft.  This IDE can be used for many other languages, meaning that if you are doing coding in multiple languages, this is a one-stop shop.  Additionally, it can connect to and control external computers and clusters using SSH (secure shell) and interface natively with GitHub (with Spyder, you will use external software to achieve the same results), so that you can have a seamless interaction with resources off your personal computer. Another popular choice is Jupyter Labs, which provides a notebook-style interface, which can be nice when documenting how you work through a problem.  Google colab offers an all-online IDE that enables real-time collaboration between you and any research partners. An emerging IDE that the authors of this look like is Positron, which is sort of a mix between Spyder and VScode, and Marimo notebooks (see next section). It is worth emphasizing that the Positron IDE is still in the early stages of its development, but it is already a nice option.



\subsection{Creating a record of data workup in Python notebooks}
\label{Creating_a_record_of_data_workup_in_Python_notebooks}

Speaking of notebooks, it is worth talking about this a bit more. The idea behind a notebook is that you can document a workflow for working up data. The notebook provides `cells'(areas that you can write and run code) separated by regions where the outputs of running the code is displayed.  Thus, you could write code that creates a plot of data and show this plot.  Then the next cell could manipulate this data somehow---maybe normalizing it, and you can output that plot.  In this manner you have a log of what you have done. In many Python notebooks, you can also place other things in the cells---such as Markdown text (which provides a means by which to create fancy text output).  For instance, you can render nice looking chemical and mathematical equations using Markdown. All together, these can be a great resource for keeping track of how you have analyzed your data. 

At the time of writing this book, Jupyter notebooks was by far the most popular implementation of notebooks for Python. However, there are two very large caveats to keep in mind when you consider using Jupyter notebooks:
\begin{enumerate}
    \item Jupyter notebooks are not simple Python files.  Thus, you cannot really read this file easily from another Python program.  This means that if you are writing code in a Jupyter notebook that you would like to use as a module, it will be  challenging to do so.

    \item Jupyter notebooks also have a serious problem whereby cells can become `stale.'  That is, you can run one cell and change data, and then another cell can display the state of the data prior to you changing it.  This means that you can have multiple outputs that show different values for the same object!  This is a serious problem, because when you look at your notebook you cannot be assured that you seeing the current state of all your data. 
\end{enumerate}
For the reasons above, we don't use Jupyter notebooks much. However, there is an emerging solution: Marimo notebooks.  This notebook addresses both of the shortcomings of Jupyter notebooks highlighted above. The Marimo notebook is just a Python file, so you can access code written in it as if it were a module. Additionally, Marimo keeps track of the current state of your variables, and if you change one, it will run \textit{all} cells that depend on that variable---the result being that your cells do not become `stale' and you are assured that the notebook you are looking at is an accurate reflection of the state of your data. Marimo is still being developed, but already it is the preferred notebook for the authors of this book. 

\section{Sharing your research products}

\subsection{Simplifying the use of your code using graphical user interfaces or webassembly}
\label{Simplifying_the_use_of_your_code_using_graphical_user_interfaces_or_webassembly}
In this book, we made some scripts that solved problems for chemical workflows.  But nearly all of the interfaces with the scripts were via the command line.  The only GUI interface we had was for identifying and saving files. The codechembook library also features a number of simple dialog boxes to give and receive information from users. However, writing full-on GUI interfaces for programs is not hard to do.  We include some tutorials on our website (\website). In this way, you can create professional looking and customized GUI interfaces for your scripts. 

Another option is to use web-assembly, which creates a Python environment inside a web browser, allowing you to interact with your scripts using the familiar text boxes, buttons, etc from websites. One benefit of the Marimo notebook mentioned above is that it includes this capability naturally. At the time of this writing, the notebooks included a button to generate the web assembly needed. Another nice thing about  Marimo notebooks is that they include ready-made interactivity, such as creation of slider bars or check boxes, etc., which can be included in the web assembled version of the notebook.  Thus, even though Marimo may not produce the absolute most tidy looking interactive programs, they produce very nice and powerful ones quickly.  We definitely recommend looking into them.



\subsection{Formatting scientific communications with Latex}
\label{Formatting_scientific_communications_with_Latex}

If you decide to get deeper into notebooks, you will start using the Markdown language. This provides the means to format text quickly... for instance creating itemized lists, check boxes, drawing mathematical formula, representing chemical formula well, etc. 

However, there are even more powerful means to generate beautifully formatted bits of text. Perhaps the most widely used tool in scientific writing is \LaTeX (Latex, pronounced lay-tech).  This is a typesetting program, built around a paradigm called "what you \textit{say} is what you get." The idea is to provide a series of instructions to the Latex interpreter, which will then render your text was you want. You just need to learn the \cite{syntax} for Latex. For instance, writing the formula $k = \nu_Ne^{-\frac{E_A}{k_BT}}$ is done using \lstinline|k = \nu_N e^{-\frac{E_A}{k_BT}}|.  If you have gone through this book, then you should be quite familiar with providing such instructions, and so you should feel confident learning the syntax for Latex. 

As you learn Latex, you will find that it can handle many of the frustrations of scientific writing. It provides a native means to add and place figures, add citations, and more.  You can organize large amounts of data across a project using Latex.  Indeed, the entirety of this book, including the table of contents, formulas, representations of code, figures, and the cross linked glossary was written entirely in Latex. 

If you want to start your Latex journey, then we think the best resource for this is to just start using Latex at www.overleaf.com.  At the time of this writing, you can sign up for a free account, which provide online access to a Latex interpreter and a very nice interface to the program. Overleaf handles most of the details of making Latex work, and it also has some nice tutorials and excellent documentation. So, don't be afraid to jump on in!



\section{Finding instruction not covered in this book}
\label{Finding_instruction_not_covered_in_this_book}
Finally, this book is, by no means, meant to be the end all of resources for learning Python. Instead, our goal was to get you solving chemical problems, while preparing you to be able to learn from more advanced resources. Here is a partial list of these advanced resources:

\noindent \ul{Chemistry-specific}
\begin{itemize}
    \item Christian Hill's \ul{Python for Chemists}, Cambridge University Press (Physical Chemistry Focus)
    \item Charles Weiss's \ul{Scientific Computing for Chemists with Python}, \textit{via} 
    
    \hyperlink{https://weisscharlesj.github.io/SciCompforChemists/}{https://weisscharlesj.github.io/SciCompforChemists/}
    \item Tanemura, Sierra-Costa, and Merz's \ul{Python for Chemists}, ACS InFocus (ChemInformatics Focus)
\end{itemize}
\ul{General Python}
\begin{itemize}
    \item Al Sweigert's \ul{Automate the Boring Stuff with Python}, No Starch Press
    \item RealPython---\href{https://realpython.com/}{https://realpython.com/}
\end{itemize}
\ul{Numerical methods}
\begin{itemize}
    \item \ul{Python Programming and Numerical Methods: A Guide for Engineers and Scientists} by Kong, Siauw, and Bayen, Academic Press, or \textit{via} 
    
    \href{https://pythonnumericalmethods.studentorg.berkeley.edu/notebooks/Index.html}{https://pythonnumericalmethods.studentorg.berkeley.edu/notebooks/Index.html}
    \item \ul{Numerical Methods: An Inquiry-Based Approach With Python} by Sullivan, \textit{via} 
    
    \href{https://numericalmethodssullivan.github.io/}{https://numericalmethodssullivan.github.io/}
\end{itemize}
The authors will try to keep more exhaustive and up-to-date list, in addition to descriptions of rapidly evolving Python capabilities, on the companion website for this book, \website.

\hrule
\input{Exercises/Chapter 17 Beyond the book}
\hrule

\end{document}
