This module introduces you to Python, making heavy use of this Software Carpentry lesson.
Our primary computing context will be the UC Berkeley DataHub. This provides us with a common platform on which we are assured to get the same behavior as each other, allowing us to use the command line/shell/terminal and Python, including Jupyter notebooks. In particular getting a common command line environment would be hard otherwise.
If you haven’t already used the shell/terminal and Python on your laptop, we recommend you use the DataHub so that we don’t have to troubleshoot problems on multiple laptops.
If you do use your laptop, you’ll need Python installed, for which we recommend using Miniforge. Another option is the Anaconda distribution of Python.
For the shell/terminal parts of this material, you can use the Terminal on your Mac, but not on Windows.
If you do use your own laptop, you’ll need to clone this GitHub repository. If you don’t know how to do that, just download this zip file and unzip it into your home directory on your computer.
I think it’s fair to say that most of us are still trying to understand how best to learn with the help of AI, without fooling ourselves into using AI without really learning/understanding.
You’re welcome to use an AI assistant (e.g., you can use GitHub Copilot for free with the GitHub Education program), but I strongly suggest you use it to assist and critique code you start to write on your own or ask it for tips rather than having it generate full code for you. That said, a regular internet search is likely to be just as effective for much of what we’ll be doing in the workshop.
If you have it generate entire blocks of code, you are unlikely to remember the syntax and ideas later, unlikely to understand well enough to debug/troubleshoot in the future, and likely to introduce errors from AI hallucinations.
When you do have AI generate code, particularly when learning or for important code, make sure you run, understand, and critique the result completely.
You may also want to look at this useful perspective from Software Carpentry about learning to code in the age of AI.
There are various options for how to save our code and in what context to run it.
For our purposes here, we’ll focus on having the code either in a Jupyter notebook or in a script (a code file in a simple text format).
Suppose we want to run this code (already seen in the previous session):
import numpy as np
import time
def run_linalg(n):
z = np.random.normal(0, 1, size=(n, n))
print(time.time())
x = z.T.dot(z) # x = z'z
print(time.time())
U = np.linalg.cholesky(x) # factorize as x = U'U
print(time.time())The file calc.py contains a variation on the code shown above and it can be run in the shell as
python calc.pyOr we can run the code in a notebook using calc.py as a module that provides the function of interest:
import calc
calc.run_linalg(4000)1785365903.4532278
1785365903.9953458
1785365904.5019023Having code in a script file will be convenient when creating packages, using Git/version control (but see nbdime if using notebooks), and running code in a batch mode (i.e., in the background rather than interactively).
One can also run Python (or IPython for some enhanced capabilities, including easier copy/pasting) from the terminal.
Finally running Python within VS Code or an IDE such as PyCharm is also common.
For various reasons, for most of the demos, I will run code in an IPython session in the terminal (or perhaps sometimes in VS Code).
We’ll use this Software Carpentry lesson for most of this material.
We’ll work through the Python Fundamentals section of the Software Carpentry workshop. The primary ideas covered are variables, data types, and basic function usage.
Using the official “The Python Standard Library” reference, figure out how to compute:
Is 1.0 of the integer type? What about 1? Is the result of 5/3 - 2/3 of the integer type? Is the mathematical value seen in Python an integer?
Here’s a numerical puzzle. Why does the last computation not work, when the others do? And, for those of you coming from R, which of these computations don’t work in R?
100000**10
100000.0**10
100000**100
100000.0**100We’ll work through the Analyzing Patient Data section of the Software Carpentry workshop. The primary ideas covered are libraries (aka packages), numpy arrays, and slicing.
Work on the questions about slicing in the section.
Note that ? and ?? only work in IPython (or a Jupyter notebook). For help in plain Python, use help(np.ndim).
What happens if you type np.ndim?? (i.e., use two question marks)? What additional do you see compared to np.ndim??
What does np.ndim() do? How does it execute under the hood? Consider why the following uses of ndim both work.
a = np.array([0, 1, 2])
a.ndim
np.ndim(a)Now explain why only one of these works.
a = [0, 1, 2]
a.ndim
np.ndim(a)Type np.array? in a Notebook or at the IPython prompt. Briefly skim the docstring. nparray allows you to construct numpy arrays.
Type np. followed by the <Tab> key in a Notebook or at the IPython prompt. Choose two or three of the completions and use ? to view their docstrings. In particular, pay attention to the examples provided near the end of the docstring and see whether you can figure out how you might use this functionality.
We’ll work through the Visualizing Tabular Data section of the Software Carpentry workshop. The primary ideas covered are making basic plots using the matplotlib package.
Sidenote: For the plots to show in a Jupyter Notebook, it seems that we need all the code for creating a given plot to be in a single cell.
Note that when we made the basic plot in the Software Carpentry material we were (indirectly) interacting with matplotlib objects present in the background, with matplotlib having the concept of the “active” plot (i.e., the current state).
When we made the multi-panel figure we were using an object-oriented interface and directly interacting with the figure (fig) and axes (axes1, axes2, axes3) objects.
Using the following code, read in the GapMinder data using Pandas (to be discussed later) and run the following code to make the variables easily available (you don’t need to know anything about Pandas).
Do this in a Jupyter notebook so that it’s easy to see the plots.
import numpy as np
import pandas as pd
dat = pd.read_csv('gapminder.csv')
lifeExp = np.array(dat.lifeExp)
gdpPercap = np.array(dat.gdpPercap)
year = np.array(dat.year)
## Hint: slicing using an array of booleans
## gdpPercap[year > 2010]lifeExp vs gdpPerCap for 2007.We’ll work through the Storing Multiple Values in Lists section of the Software Carpentry workshop. The primary ideas covered are creating, extracting from, and manipulating lists.
Create a list of numbers, called x1. Reverse the order of the items in the list using slicing. Now reverse the order of the items using a list method. How does using the method differ from slicing? Hint: you can type x1. followed by the <Tab> key in a Notebook or at the IPython prompt to find the various methods that can be applied to a list.
x1. followed by the <Tab> key in a Notebook or at the IPython prompt to find the various methods that can be applied to a list.)Answer the question related to Overloading in the Software Carpentry section.
What does the following tell you about copying and use of memory in lists in Python?
a = [1, 3, 5]
b = a
id(a)
id(b)
# this should confirm what you might suspect
a[1] = 5We’ll work through the Repeating Actions with Loops section of the Software Carpentry workshop. The primary ideas covered relate to using for loops to automate operations.
In addition to the explicit for looping shown in the unit, Python also has a construct called list comprehension to do something similar. Note that some people really dislike the list comprehension syntax as being hard to read.
vals = [0, 5, 7]
vals_times2 = [val*2 for val in vals]
vals_times2[0, 10, 14]Finally, we can use map to iterate as well.
vals = [0, 5, 7]
result = map(lambda x: x**2, vals) # a lazy iterator
list(result) # actual result[0, 25, 49][1, 2, 3] + 3 returns. Try to explain what happened and why.range function may be helpful as might the enumerate function.[1, 2, 3] to be a numpy array and then add three using + 3.We’ll work through the Making Choices section of the Software Carpentry workshop. The primary ideas covered are using conditionals (if-then-else statements) and boolean (logical) operations.
Answer this question from the Software Carpentry section.
Answer this question from the Software Carpentry section.
We’ll work through the Creating Functions section of the Software Carpentry workshop. The primary ideas covered are defining, using, testing, and debugging functions.
We’ll skip over testing and documentation as we’ll cover those in depth later this week (and definitely NOT because they are unimportant).
Exercises
Write a function that will count vowels in a character string (i.e., using code from the exercise of the previous section). Allow the user to specify that “y” should count as a vowel.
Answer the question related to local vs. global variables in the Software Carpentry section.
We’ll work through the Errors and Exceptions section of the Software Carpentry workshop. The primary ideas covered relate to understanding error messages and tracebacks. In the Computational Tools and Practices session, we’ll talk about writing your own code that handles errors.
Answer this question from the Software Carpentry section.
Answer these questions from Sofware Carpentry:
Tuples are immutable sequences of (zero or more) objects. Functions in Python often return tuples.
x = 1; y = 'foo'
xy = (x, y)
type(xy)
xy = x,y
type(xy)
xy
xy[1]
xy[1] = 3 immutable!Cell In[4], line 11 xy[1] = 3 immutable! ^ SyntaxError: invalid syntax
What’s weird about this? What are the types involved?
z = x,y
a,b = x,yCreate the following: x=5 and y=99. Now swap their values using a single line of code. (For R users, how would you do this in R?)
What happens when you multiply a tuple by a number?
reverse method for tuples?Dictionaries are mutable, unordered collections of key-value pairs. They’re a very important data structure in Python so I’m surprised they weren’t discussed in the Software Carpentry sections.
students = {"Francis Bacon": ['A', 'B+', 'A-'],
"Marie Curie": ['A-', 'A-'],
"Aristotle": 'and now for something completely different'}
students
students.keys()
students.values()
students["Marie Curie"]
students["Marie Curie"][1]'A-'students.students.clear( students.items( students.setdefault(
students.copy( students.keys( students.update(
students.fromkeys( students.pop( students.values(
students.get( students.popitem() How can you add an item to a dictionary?
How do you combine two dictionaries into a single dictionary?
Standard lists in Python are not amenable to mathematical manipulation, unlike standard vectors in R. Instead we generally work with numpy arrays. These arrays can be of various dimensions (i.e., vectors, matrices, multi-dimensional arrays).
import numpy as np
z = [0, 1, 2]
y = np.array(z)
y*3
y.dtype
x = np.array([[1, 2], [3, 4]], dtype=np.float64)
x*x
x.dot(x)
x.T
np.linalg.svd(x)
e = np.linalg.eig(x)
e[0] # first eigenvalue (not the largest in this case...)
e[1][:, 0] # corresponding eigenvectorarray([-0.82456484, 0.56576746])All of the elements of the array must be of the same type.
There are a variety of numpy functions that allow us to do standard mathematical/statistical manipulations.
Here we’ll use some of those functions in addition to some syntax for subsetting and vectorized calculations.
np.linspace(0, 1, 5)
np.random.seed(0)
x = np.random.normal(size=10)
pos = x > 0
y = x[pos]
x[[1, 3, 4]]
x[pos] = 0
# Vectorized boolean arithmetic
x = np.random.normal(size=10)
x[(x > 1) | (x < -0.5)]
np.cos(x)array([0.98964365, 0.11625932, 0.72412071, 0.99260672, 0.90309941,
0.94484532, 0.07664202, 0.97902875, 0.95139326, 0.65690057])One can’t use and and or in vectorized (e.g., numpy) code. Instead we need & and |.
scipy has even more numerical routines, including working with distributions and additional linear algebra.
import scipy.stats as st
st.norm.cdf(1.96, 0, 1)
st.norm.cdf(1.96, 0.5, 2)
st.norm(0.5, 2).cdf(1.96)Pandas provides a Python implementation of R’s dataframe capabilities. Let’s see some example code.
import pandas as pd
dat = pd.read_csv('gapminder.csv')
dat.head()
dat.columns
dat['year']
dat.year
dat[0:5]
dat.sort_values(['year', 'country'])
dat.loc[0:5, ['year', 'country']] # R-style indexing
dat[dat.year == 1952]
ndat = dat[['pop','lifeExp','gdpPercap']]
ndat.apply(lambda col: col.max() - col.min())pop 1.318623e+09
lifeExp 5.900400e+01
gdpPercap 1.132820e+05
dtype: float64Now let’s see the sort of split-apply-combine functionality that is popular in dplyr and related R packages.
dat2007 = dat[dat.year == 2007].copy()
dat2007.groupby('continent', as_index=False)[['lifeExp','gdpPercap']].mean()
def stdize(vals):
return((vals - vals.mean()) / vals.std())
dat2007['lifeExpZ'] = dat2007.groupby('continent')['lifeExp'].transform(stdize)