Ipython (DONE)¶

Numpy basics (DONE)¶

Pandas¶

high level data structures for data analysis
built on top of numpy
Specs
- Labeled axes supporting automatic or explicit data alignement
- Time series functionalities
- Support data as both time series and non-time series
- Handling of missing data
- Merge operation like in SQL dbs

# from pandas import Series, DataFrame

import pandas as pd
import pandas_datareader.data as web

import numpy as np

print(pd.__version__)

import pprint

0.24.1

Introduction to pandas Data Structures¶

Series¶

obj = pd.Series([4, 7, -5, 3])

obj

# An index is assigned automatically.

0    4
1    7
2   -5
3    3
dtype: int64

# Print numpy underlying data struct.
obj.values

array([ 4, 7, -5, 3])

# Index is a special pandas object.
obj.index

RangeIndex(start=0, stop=4, step=1)

# Assign index explicitly.
obj2 = pd.Series([4, 7, -5, 3], index="d b a c".split())

obj2

d    4
b    7
a   -5
c    3
dtype: int64

# Access by index.
obj2["a"]

-5

obj2["d"] = 6

obj2

d    6
b    7
a   -5
c    3
dtype: int64

# Select slice.
obj2[["c", "a", "d"]]

c    3
a   -5
d    6
dtype: int64

# numpy operations preserve index-value link.
obj2

d    6
b    7
a   -5
c    3
dtype: int64

obj2[obj2 > 0]

d    6
b    7
c    3
dtype: int64

obj2 * 2

d    12
b    14
a   -10
c     6
dtype: int64

print("b" in obj2)
print("e" in obj2)

True
False

# Build from dict.
sdata = {"Ohio": 35000, "Texas": 71000, "Oregon": 16000, "Utah": 5000}

obj3 = pd.Series(sdata)

print(obj3)

Ohio      35000
Texas     71000
Oregon    16000
Utah       5000
dtype: int64

# Build from dict with explicit index.

states = ["California", "Ohio", "Oregon", "Texas"]
# Note that since data for California doesn't exist.
obj4 = pd.Series(sdata, index=states)

print(obj4)

California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
dtype: float64

obj4.isnull()

California     True
Ohio          False
Oregon        False
Texas         False
dtype: bool

# Data are aligned automatically, merging indices using "outer".
print("obj3=\n", obj3)
print("\nobj4=\n", obj4)

obj3 + obj4

obj3=
 Ohio      35000
Texas     71000
Oregon    16000
Utah       5000
dtype: int64

obj4=
 California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
dtype: float64

California         NaN
Ohio           70000.0
Oregon         32000.0
Texas         142000.0
Utah               NaN
dtype: float64

print(obj4)

# Set name of the series.
obj4.name = "population"
print("\n", obj4)

# Set name of the index.
obj4.index.name = "state"
print("\n", obj4)

California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
dtype: float64

 California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
Name: population, dtype: float64

 state
California        NaN
Ohio          35000.0
Oregon        16000.0
Texas         71000.0
Name: population, dtype: float64

Data frame¶

Tabular data structure with ordered collection of columns and indexes.
- Index and columns are rather symmetric
- One can represent higher dimensional data using multi-indexing
Df can be built from:
- 2d ndarray
- dict of arrays / lists / tuples
- dict of Series / dicts
- list of Series / dicts
- list of lists / tuples (like the 2d ndarray)
- another DataFrame

# Build from dict of arrays.
data = {
    "state": ["Ohio", "Ohio", "Ohio", "Nevada", "Nevada"],
    "year": [2000, 2001, 2002, 2001, 2002],
    "pop": [1.5, 1.7, 3.6, 2.4, 2.9],
}
frame = pd.DataFrame(data)

print(frame)

    state  year  pop
0    Ohio  2000  1.5
1    Ohio  2001  1.7
2    Ohio  2002  3.6
3  Nevada  2001  2.4
4  Nevada  2002  2.9

# Change order of columns and set index.
frame2 = pd.DataFrame(
    data,
    columns="year state pop test".split(),
    index="one two three four five".split(),
)

print(frame2)

       year   state  pop test
one    2000    Ohio  1.5  NaN
two    2001    Ohio  1.7  NaN
three  2002    Ohio  3.6  NaN
four   2001  Nevada  2.4  NaN
five   2002  Nevada  2.9  NaN

# Access row, which is returned as series.
print(frame2.loc["three"])

year     2002
state    Ohio
pop       3.6
test      NaN
Name: three, dtype: object

# Assigning a column that doesn't exist creates a new column, broadcasting the value.
frame2["debt"] = 16.5

print(frame2)

       year   state  pop test  debt
one    2000    Ohio  1.5  NaN  16.5
two    2001    Ohio  1.7  NaN  16.5
three  2002    Ohio  3.6  NaN  16.5
four   2001  Nevada  2.4  NaN  16.5
five   2002  Nevada  2.9  NaN  16.5

# Assign a range.
frame2["debt"] = np.arange(5)

print(frame2)

       year   state  pop test  debt
one    2000    Ohio  1.5  NaN     0
two    2001    Ohio  1.7  NaN     1
three  2002    Ohio  3.6  NaN     2
four   2001  Nevada  2.4  NaN     3
five   2002  Nevada  2.9  NaN     4

# Assign a series.
val = pd.Series([-1.2, -1.5, -1.7], index=["two", "four", "five"])

frame2["debt"] = val

print(frame2)

       year   state  pop test  debt
one    2000    Ohio  1.5  NaN   NaN
two    2001    Ohio  1.7  NaN  -1.2
three  2002    Ohio  3.6  NaN   NaN
four   2001  Nevada  2.4  NaN  -1.5
five   2002  Nevada  2.9  NaN  -1.7

# One can use the dot notation besides the index notation.
# print frame2["debt"]
print(frame2.debt)

one      NaN
two     -1.2
three    NaN
four    -1.5
five    -1.7
Name: debt, dtype: float64

frame2["is_Ohio"] = frame2["state"] == "Ohio"

print(frame2)

       year   state  pop test  debt  is_Ohio
one    2000    Ohio  1.5  NaN   NaN     True
two    2001    Ohio  1.7  NaN  -1.2     True
three  2002    Ohio  3.6  NaN   NaN     True
four   2001  Nevada  2.4  NaN  -1.5    False
five   2002  Nevada  2.9  NaN  -1.7    False

# To delete a column.
del frame2["is_Ohio"]

print(frame2)

       year   state  pop test  debt
one    2000    Ohio  1.5  NaN   NaN
two    2001    Ohio  1.7  NaN  -1.2
three  2002    Ohio  3.6  NaN   NaN
four   2001  Nevada  2.4  NaN  -1.5
five   2002  Nevada  2.9  NaN  -1.7

# WARNING: this is a view not a copy.
frame2.columns

Index(['year', 'state', 'pop', 'test', 'debt'], dtype='object')

# Build a df from a dictionary of dicts.
# Rows are unioned and sorted.
pop = {
    "Nevada": {2001: 2.4, 2002: 2.9},
    "Ohio": {2000: 1.5, 2001: 1.7, 2002: 3.6},
}
print("pop=", pprint.pformat(pop))

frame3 = pd.DataFrame(pop)
print(frame3)

pop= {'Nevada': {2001: 2.4, 2002: 2.9}, 'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}
      Nevada  Ohio
2000     NaN   1.5
2001     2.4   1.7
2002     2.9   3.6

# Build a df from a dictionary of rows.
frame3 = pd.DataFrame(pop).T

display(frame3)

#?pd.Series

# Build series from dict.
Ohio_srs = pd.Series({2000: 1.5, 2001: 1.7, 2002: 3.6}, name="Ohio")
print(Ohio_srs)

Nevada_srs = pd.Series({2001: 2.4, 2002: 2.9}, name="Nevada")
print(Nevada_srs)

2000    1.5
2001    1.7
2002    3.6
Name: Ohio, dtype: float64
2001    2.4
2002    2.9
Name: Nevada, dtype: float64

# Build df from a dict of Series.
frame4 = pd.DataFrame(
    {
        "Ohio": Ohio_srs,
        "Nevada": Nevada_srs,
    }
)

display(frame4)

Index objects¶

Index objects hold
- axis labels and
- metadata (e.g., axis name)
Indices are specialized for containing different values
- Index: axis labels for nparray of Python objects
- Int64Index: index for int values
- MultiIndex: hierarchical index representing multiple levels of indexing
  - Similar to an array of tuples
- DatetimeIndex: stores ns timestamps (np.datetime64)
- PeriodIndex: stores period data, i.e., timespans

obj = pd.Series(list(range(3)), index="a b c".split())

index = obj.index

index

Index(['a', 'b', 'c'], dtype='object')

# Index objects are immutable.
# raise
#   TypeError: Index does not support mutable operations

# index[1] = 'd'

index = pd.Index(np.arange(3))

obj2 = pd.Series([1.5, -2.5, 0], index=index)

obj2.index

Int64Index([0, 1, 2], dtype='int64')

# There is also a in function

print(2 in obj2.index)
print(4 in obj2.index)

True
False

index1 = pd.Index(np.arange(3))
print("index1=", index1)
index2 = pd.Index(np.arange(5, 2, -1))
print("index2=", index2)

print("intersection=", index1.intersection(index2))

print("is_monotonic=", index1.is_monotonic, index2.is_monotonic)
print("is_unique=", index1.is_unique, index2.is_unique)

index1= Int64Index([0, 1, 2], dtype='int64')
index2= Int64Index([5, 4, 3], dtype='int64')
intersection= Int64Index([], dtype='int64')
is_monotonic= True False
is_unique= True True

Essential functionality¶

Reindexing¶

reindexing = create a new object with data conformed to a new index

obj = pd.Series([4.5, 7.2, -5.3, 3.6], index="d b a c".split())

obj

d    4.5
b    7.2
a   -5.3
c    3.6
dtype: float64

# Re-arrange data according to new index, introducing missing values.
obj2 = obj.reindex("a b c d e".split())

obj2

a   -5.3
b    7.2
c    3.6
d    4.5
e    NaN
dtype: float64

obj2 = obj.reindex("a b c d e".split(), fill_value=0)

obj2

a   -5.3
b    7.2
c    3.6
d    4.5
e    0.0
dtype: float64

obj3 = pd.Series("b p y".split(), index=[0, 2, 4])
print(obj3)

obj4 = obj3.reindex(list(range(6)), method="ffill")

print(obj4)

0    b
2    p
4    y
dtype: object
0    b
1    b
2    p
3    p
4    y
5    y
dtype: object

Dropping entries¶

obj = pd.Series(np.arange(5.0), index="a b c d e".split())
print(obj)

# Drop element by label.
new_obj = obj.drop("c")
print(new_obj)

new_obj = obj.drop("c d".split())
print(new_obj)

a    0.0
b    1.0
c    2.0
d    3.0
e    4.0
dtype: float64
a    0.0
b    1.0
d    3.0
e    4.0
dtype: float64
a    0.0
b    1.0
e    4.0
dtype: float64

Indexing, selection, filtering¶

obj[...] works as NumPy array indexing, but you can use also index values instead of integers

Series¶

obj = pd.Series(np.arange(4.0), index="a b c d".split())
print(obj)

a    0.0
b    1.0
c    2.0
d    3.0
dtype: float64

# Index by label or index.
print(obj["b"])
print(obj[1])

1.0
1.0

# Slicing with int indices or boolean mask.
print(obj[[1, 3]])

print(obj[obj < 2])

b    1.0
d    3.0
dtype: float64
a    0.0
b    1.0
dtype: float64

# In slicing note that the endpoint is inclusive.
print(obj["b":"c"])

b    1.0
c    2.0
dtype: float64

obj2 = obj.copy()
obj2["b":"c"] = 5

obj2

a    0.0
b    5.0
c    5.0
d    3.0
dtype: float64

DataFrame¶

data = pd.DataFrame(
    np.arange(16).reshape((4, 4)),
    index=["Ohio", "Colorado", "Utah", "New York"],
    columns=["one", "two", "three", "four"],
)

print(data)

          one  two  three  four
Ohio        0    1      2     3
Colorado    4    5      6     7
Utah        8    9     10    11
New York   12   13     14    15

# Select one column.
data["two"]

Ohio         1
Colorado     5
Utah         9
New York    13
Name: two, dtype: int64

# Select two columns.
data[["three", "one"]]

# Note that this int or boolean indexing selects rows.
data[:2]

data[data["three"] > 5]

# Select with a boolean "mask" df.
mask = data < 5
print(mask)

data[mask]

            one    two  three   four
Ohio       True   True   True   True
Colorado   True  False  False  False
Utah      False  False  False  False
New York  False  False  False  False

data

# To select using label indexing.
data.loc["Colorado", ["two", "three"]]

two      5
three    6
Name: Colorado, dtype: int64

# This is deprecated.
data.ix[["Colorado", "Utah"], [3, 0, 1]]

/Users/saggese/.conda/envs/pymc/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: 
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated

data.iloc[2]

one       8
two       9
three    10
four     11
Name: Utah, dtype: int64

data.loc[:"Utah", "two"]

Ohio        1
Colorado    5
Utah        9
Name: two, dtype: int64

obj[val]:
- select column by value
- select rows by bool array / data frame
obj.loc[val], obj.iloc[val]
- select row by label / index
obj.loc[:, val], obj.iloc[:, val]
- select column by label / index
obj.loc[val1, val2]
- select element

Arithmetic and data alignment¶

When doing arithmetic between objects with different indexes, the union of the index is peformed

# Operations between DataFrame and Series imply broadcasting.

df = pd.DataFrame(
    np.arange(12).reshape(4, 3),
    columns=list("bde"),
    index="Utah Ohio Texas Oregon".split(),
)

df

series = df.iloc[0]

series

b    0
d    1
e    2
Name: Utah, dtype: int64

# Broadcasting is by rows.
df - series

Function application and mapping.¶

np.random.seed(42)

frame = pd.DataFrame(
    np.random.randn(4, 3),
    columns=list("bde"),
    index=["Utah", "Ohio", "Texas", "Oregon"],
)

frame

# NumpPy functions work element-wise.
np.abs(frame)

apply()¶

# A function returning a single value.
f = lambda x: x.max() - x.min()

# DataFrame.apply() uses the row / column labels to build the resulting
# Series.
frame.apply(f)

b    1.082499
d    1.230852
e    1.117163
dtype: float64

frame.apply(f, axis=1)

Utah      0.785953
Ohio      1.757183
Texas     2.048687
Oregon    1.008290
dtype: float64

# Using a function that returns a Series of two values.
# DataFrame.apply() uses the row / column labels to build the resulting
# DataFrame.
def f(x):
    return pd.Series([x.min(), x.max()], index="min max".split())


print(frame.apply(f))

print(frame.apply(f, axis=1))

            b         d         e
min  0.496714 -0.463418 -0.469474
max  1.579213  0.767435  0.647689
             min       max
Utah   -0.138264  0.647689
Ohio   -0.234153  1.523030
Texas  -0.469474  1.579213
Oregon -0.465730  0.542560

applymap()¶

# Element-wise transformation.
format_ = lambda x: "%.2f" % x

print(frame.applymap(format_))

           b      d      e
Utah    0.50  -0.14   0.65
Ohio    1.52  -0.23  -0.23
Texas   1.58   0.77  -0.47
Oregon  0.54  -0.46  -0.47

Sorting and ranking¶

Series¶

obj = pd.Series(list(range(4)), index="d a b c".split())

obj

d    0
a    1
b    2
c    3
dtype: int64

# Sort by index.
obj.sort_index()

a    1
b    2
c    3
d    0
dtype: int64

obj.sort_values()

d    0
a    1
b    2
c    3
dtype: int64

DataFrame¶

frame = pd.DataFrame(
    np.arange(8).reshape((2, 4)),
    index=["three", "one"],
    columns=["d", "a", "b", "c"],
)

frame

# Sort rows.
display(frame.sort_index())

# Sort columns.
display(frame.sort_index(axis=1))

frame = pd.DataFrame({"b": [4, 7, -3, 2], "a": [0, 1, 0, 1]})

frame

# Sort by multiple columns.
frame.sort_values(by=["a", "b"])

# There are multiple ways of breaking ties.
frame.rank(axis=1)

Axis indexes with duplicate values.¶

obj = pd.Series(list(range(5)), index=["a", "a", "b", "b", "c"])

obj

a    0
a    1
b    2
b    3
c    4
dtype: int64

# Verify that it has multiple indices.
obj.index.is_unique

False

# Indexing returns different objects (Series or scalar), depending
# if the values are unique or not.
obj["a"]

a    0
a    1
dtype: int64

obj["c"]

4

Summarizing and descriptive statistics¶

pd objects have math and stat methods
- summary statistics
- handle missing data better than np

df = pd.DataFrame(
    [[1.4, np.nan], [7.1, -4.5], [np.nan, np.nan], [0.75, -1.3]],
    index="a b c d".split(),
    columns="one two".split(),
)

df

df.sum()

one    9.25
two   -5.80
dtype: float64

# NAs are skipped automatically.

# df.sum(axis=1)
df.sum(axis="columns")

a    1.40
b    2.60
c    0.00
d   -0.55
dtype: float64

# skipna gives a np behavior.
df.sum(axis=1, skipna=False)

a     NaN
b    2.60
c     NaN
d   -0.55
dtype: float64

df.max()

one    7.1
two   -1.3
dtype: float64

# df.argmax()

# Max value per column.
df.idxmax()

one    b
two    d
dtype: object

# Accumulate along columns.
df.cumsum()

print(df.describe())

            one       two
count  3.000000  2.000000
mean   3.083333 -2.900000
std    3.493685  2.262742
min    0.750000 -4.500000
25%    1.075000 -3.700000
50%    1.400000 -2.900000
75%    4.250000 -2.100000
max    7.100000 -1.300000

Correlation and covariance.¶

all_data = {
    ticker: web.get_data_yahoo(ticker)
    for ticker in ["AAPL", "IBM", "MSFT", "GOOG"]
}

print(list(all_data.keys()))

print(all_data["GOOG"].head())

['AAPL', 'IBM', 'MSFT', 'GOOG']
                  High         Low        Open       Close      Volume  \
Date                                                                     
2010-01-04  312.721039  310.103088  311.449310  311.349976   3937800.0   
2010-01-05  311.891449  308.761810  311.563568  309.978882   6048500.0   
2010-01-06  310.907837  301.220856  310.907837  302.164703   8009000.0   
2010-01-07  303.029083  294.410156  302.731018  295.130463  12912000.0   
2010-01-08  299.675903  292.651581  294.087250  299.064880   9509900.0   

             Adj Close  
Date                    
2010-01-04  311.349976  
2010-01-05  309.978882  
2010-01-06  302.164703  
2010-01-07  295.130463  
2010-01-08  299.064880

price = pd.DataFrame(
    {ticker: data["Adj Close"] for ticker, data in list(all_data.items())}
)
print(price.head())

volume = pd.DataFrame(
    {ticker: data["Volume"] for ticker, data in list(all_data.items())}
)
print(volume.head())

                 AAPL         IBM       MSFT        GOOG
Date                                                    
2010-01-04  20.386072  100.478867  24.615801  311.349976
2010-01-05  20.421322   99.265091  24.623755  309.978882
2010-01-06  20.096491   98.620255  24.472631  302.164703
2010-01-07  20.059338   98.278870  24.218124  295.130463
2010-01-08  20.192701   99.265091  24.385145  299.064880
                   AAPL        IBM        MSFT        GOOG
Date                                                      
2010-01-04  123432400.0  6155300.0  38409100.0   3937800.0
2010-01-05  150476200.0  6841400.0  49749600.0   6048500.0
2010-01-06  138040000.0  5605300.0  58182400.0   8009000.0
2010-01-07  119282800.0  5840600.0  50559700.0  12912000.0
2010-01-08  111902700.0  4197200.0  51197400.0   9509900.0

returns = price.pct_change()

print(returns.tail())

                AAPL       IBM      MSFT      GOOG
Date                                              
2019-03-20  0.008739 -0.006335 -0.001105  0.020953
2019-03-21  0.036830  0.013180  0.022975  0.006185
2019-03-22 -0.020708 -0.014070 -0.026368 -0.021144
2019-03-25 -0.012091 -0.001936  0.005211 -0.010369
2019-03-26 -0.010332  0.007472  0.002125 -0.007024

returns["MSFT"].corr(returns["IBM"])

0.48799896201872567

returns["MSFT"].cov(returns["IBM"])

8.728888902136086e-05

returns.corr()

returns.cov()

returns.corrwith(returns.IBM)

AAPL    0.373555
IBM     1.000000
MSFT    0.487999
GOOG    0.408626
dtype: float64

Unique values, counts, and membership¶

srs = pd.Series("c a d a a b b c c".split())
srs

0    c
1    a
2    d
3    a
4    a
5    b
6    b
7    c
8    c
dtype: object

srs.unique()

array(['c', 'a', 'd', 'b'], dtype=object)

pd.value_counts(srs.values, sort=False)

c    3
a    3
b    2
d    1
dtype: int64

# isin() vectorized set membership applied to each element of the series.
srs.isin(["b", "c"])

0     True
1    False
2    False
3    False
4    False
5     True
6     True
7     True
8     True
dtype: bool

# Index.get_indexer() allows to establish mapping between them.

to_match = pd.Series("c a b b c a".split())
unique_vals = pd.Series("c b a".split())

pd.Index(unique_vals).get_indexer(to_match)

array([0, 2, 1, 1, 0, 2])

Ipython (DONE)

Table of Contents¶

Ipython (DONE)¶

Numpy basics (DONE)¶

Pandas¶

Introduction to pandas Data Structures¶

Series¶

Data frame¶

Index objects¶

Essential functionality¶

Reindexing¶

Dropping entries¶

Indexing, selection, filtering¶

Series¶

DataFrame¶

Arithmetic and data alignment¶

Function application and mapping.¶

apply()¶

applymap()¶

Sorting and ranking¶

Series¶

DataFrame¶

Axis indexes with duplicate values.¶

Summarizing and descriptive statistics¶

Correlation and covariance.¶

Unique values, counts, and membership¶