{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KUdg15uf30TU"
   },
   "source": [
    "# Summary Statistics\n",
    "\n",
    "**Further Reading**: §1.2 in Navidi (2015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iCqLvLrU30TW",
    "tags": []
   },
   "source": [
    "## Learning Objectives\n",
    "\n",
    "After studying this notebook and your notes, completing the activities, asking questions in class, you should be able to:\n",
    "* Compute descriptive statistics for data using Pandas (Python package)\n",
    "* Interpret the elements of a covariance matrix\n",
    "* Explain why some data distributions have a very different median and mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two Types of Data: Numerical and Categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As engineers, you will encounter both **numerical** (quantitative) and **categorical** (qualitative) data.\n",
    "\n",
    "Unfortunately, Example 1.8 from the textbook was not included in the data files on the publisher's website. But do not fear! We can recreate the table from a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Specimen  Torque Failure Location\n",
      "0         1     165             Weld\n",
      "1         2     237             Beam\n",
      "2         3     222             Beam\n",
      "3         4     255             Beam\n",
      "4         5     194             Weld\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Specimen</th>\n",
       "      <th>Torque</th>\n",
       "      <th>Failure Location</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>165</td>\n",
       "      <td>Weld</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>237</td>\n",
       "      <td>Beam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>222</td>\n",
       "      <td>Beam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>255</td>\n",
       "      <td>Beam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>194</td>\n",
       "      <td>Weld</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Specimen  Torque Failure Location\n",
       "0         1     165             Weld\n",
       "1         2     237             Beam\n",
       "2         3     222             Beam\n",
       "3         4     255             Beam\n",
       "4         5     194             Weld"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Store all of the data from Example 1.8 in a dictionary\n",
    "# Notice the keys are the column names\n",
    "my_dict = {\"Specimen\":[1, 2, 3, 4, 5], \"Torque\":[165, 237, 222, 255, 194],\"Failure Location\":['Weld','Beam','Beam','Beam','Weld']}\n",
    "\n",
    "# Convert the dictionary into a Pandas dataframe\n",
    "my_df = pd.DataFrame(my_dict)\n",
    "\n",
    "# Print\n",
    "print(my_df)\n",
    "\n",
    "# Look at the first five entries\n",
    "my_df.head()\n",
    "\n",
    "# Profit???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well that was easy.\n",
    "\n",
    "In this example, we see that *Torque* is a numerical variable and *Failure Location* is a categorical variable.\n",
    "\n",
    "We will now learn about statistics to summarize key characteristics of samples. Let's start by focusing on numerical variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Mean\n",
    "\n",
    "The sample mean is the average of the sample.\n",
    "\n",
    "Let $X_1$, $X_2$, ..., $X_n$ be the sample. The **sample mean** is\n",
    "\n",
    "$$\\bar{X} = \\frac{1}{n} \\sum_{i=1}^{n} X_i$$\n",
    "\n",
    "Statisticians have many quirks, which make them the butts of many jokes. I would like to draw your attention to two:\n",
    "1. A capital variable, such as $X_i$, is often a **random variable**. More on this next class.\n",
    "2. Statisticians like to give variables decorations. Here $\\bar{~}$ (bar) means average. Later in the semester we'll see that $\\hat{~}$ (hat) means estimate.\n",
    "\n",
    "It is really easy to calculate the sample mean with pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "low = pd.read_csv('https://raw.githubusercontent.com/ndcbe/data-and-computing/main/notebooks/data/table1-1.csv')\n",
    "high = pd.read_csv('https://raw.githubusercontent.com/ndcbe/data-and-computing/main/notebooks/data/table1-2.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PM    3.714565\n",
       "dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That output looks strange."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(low.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting. The command `low.mean()` does not return a floating point number. It instead returns a variable that is type `pandas.core.series.Series`. What if I just want the sample mean in a floating point number?\n",
    "\n",
    "In pandas, we can access a column using its name. Recall here are the first five elements in the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PM</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     PM\n",
       "0  1.50\n",
       "1  0.87\n",
       "2  1.12\n",
       "3  1.25\n",
       "4  3.46"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data set only has one column. But if we want to extract that column, we write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      1.50\n",
       "1      0.87\n",
       "2      1.12\n",
       "3      1.25\n",
       "4      3.46\n",
       "       ... \n",
       "133    4.63\n",
       "134    2.80\n",
       "135    2.16\n",
       "136    2.97\n",
       "137    3.90\n",
       "Name: PM, Length: 138, dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.PM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And to get the numeric mean, we simply write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.714565217391306"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.PM.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition seealso\"> \n",
    "    <p class=\"title\"><b>Optional Home Activity</b></p>\n",
    " Calculate the sample mean for PM in the high altitude data set. Store your answer (a float) in the variable <tt>ans_14b_1</tt>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6.596451612903223\n"
     ]
    }
   ],
   "source": [
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "14b-1",
     "locked": true,
     "points": "0.2",
     "solution": false
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Removed autograder test. You may delete this cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Variance and Standard Deviation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While mean reflects the average of a sample, variance and standard deviation measure the spread."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $X_1$, ..., $X_n$ be a sample. The **sample variance** is\n",
    "\n",
    "$$s^2 = \\frac{1}{n-1} \\sum_{i}^{n} (X_i - \\bar{X})^2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note\"> \n",
    "<p class=\"title\"><b>Class Activity</b></p>\n",
    " If the measured quantitfy $X$ is velocity with units m/s, then what are the units of variance $s^2$?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Activity Answer**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's dig into the idea that variance measures the spread of the data set. Consider a synthetic (i.e., made up, artificial) data set with two columns, A and B:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A  B\n",
       "0   0  4\n",
       "1   0  4\n",
       "2   5  5\n",
       "3  10  6\n",
       "4  10  6"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data = pd.DataFrame({\"A\":[0, 0, 5, 10, 10], \"B\":[4, 4, 5, 6, 6]})\n",
    "my_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both columns have the same mean (average):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    5.0\n",
       "B    5.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition seealso\"> \n",
    "        <p class=\"title\"><b>Optional Home Activity</b></p>\n",
    " Please take a minute to verify (on paper or in your head) that both data sets do in fact have a mean of 5.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool, pandas just calculated the means for both columns in our synthetic data set.\n",
    "\n",
    "What about the variance of both columns? The variance formula,\n",
    "\n",
    "$$s^2 = \\frac{1}{n-1} \\sum_{i}^{n} (X_i - \\bar{X})^2$$\n",
    "\n",
    "sums the squared difference between each datum (a.k.a., data point) and the mean. Thus we expect a data set with a larger spread to have a larger variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    25.0\n",
       "B     1.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.var()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this is in fact what we see with the variance calculation. Column A, with range 0 to 10, has a much larger variance than column B, which has a range of 4 to 6.\n",
    "\n",
    "Often we prefer to work with **sample standard derivation**, which is the square root of **sample variance**:\n",
    "\n",
    "$$s = \\sqrt{ \\frac{1}{n-1} \\sum_{i}^{n} (X_i - \\bar{X})^2}$$\n",
    "\n",
    "Notice that $s$ has the same units as $X$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    5.0\n",
       "B    1.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But why do the standard deviation and variance formulas divide by $n-1$ and not $n$? Short answer: often, we do not know the population standard deviation. (Recall, the sample was drawn from a population.) So we use $s$ as an estimate for the population standard deviation. This uses one degree of freedom, so we divide by $n-1$ instead of $n$. The $-1$ is because we want to estimate one parameter (population standard deviation). We will revist this idea a few times during the semester. A common exercise in a graduate level statistics course is to prove that dividing by $n-1$ makes $s^2$ an unbiased estimate of population variance. Still curious? Check out this video: https://www.youtube.com/watch?v=D1hgiAla3KI\n",
    "\n",
    "If we really wanted to, we could change the degree of freedom from the default 1 to any number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "variance\n",
      " A    25.0\n",
      "B     1.0\n",
      "dtype: float64 \n",
      "\n",
      "standard deviation\n",
      " A    5.0\n",
      "B    1.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# divide by n - 1 in variance and standard derviation formulae. This is the default\n",
    "print(\"variance\\n\",my_data.var(ddof=1),\"\\n\")\n",
    "print(\"standard deviation\\n\",my_data.std(ddof=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "variance\n",
      " A    20.0\n",
      "B     0.8\n",
      "dtype: float64 \n",
      "\n",
      "standard deviation\n",
      " A    4.472136\n",
      "B    0.894427\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# divide by n in variance and standard derviation formulae.\n",
    "print(\"variance\\n\",my_data.var(ddof=0),\"\\n\")\n",
    "print(\"standard deviation\\n\",my_data.std(ddof=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "variance\n",
      " A    33.333333\n",
      "B     1.333333\n",
      "dtype: float64 \n",
      "\n",
      "standard deviation\n",
      " A    5.773503\n",
      "B    1.154701\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# divide by n - 2 in variance and standard derviation formulae.\n",
    "print(\"variance\\n\",my_data.var(ddof=2),\"\\n\")\n",
    "print(\"standard deviation\\n\",my_data.std(ddof=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like with the mean, we can easily extract a floating point number from pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data[\"A\"].std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.A.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition seealso\"> \n",
    "    <p class=\"title\"><b>Optional Home Activity</b></p>\n",
    " Calculate the standard deviation (using $n-1$) for the particulate matter example. Store the results in the dictionary <tt>ans_14b_2</tt> with keys \"low\" and \"high\". Hint: You'll need to calculate the standard deviation as a float and then save the answers into the dictionary.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'low': 2.5580401712274687, 'high': 4.518997820413171}\n"
     ]
    }
   ],
   "source": [
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "14b-2",
     "locked": true,
     "points": "0.2",
     "solution": false
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Removed autograder test. You may delete this cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition seealso\"> \n",
    "<p class=\"title\"><b>Optional Home Activity</b></p>\n",
    " Show the following formulae are equivalent to the definitions for sample variance and sample standard deviation given above. <b>This is excellent practice for the next exam.</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$s^2 = \\frac{1}{n-1} \\left( \\sum_{i=1}^{n} X_i^2 - n \\bar{X} \\right)$$\n",
    "\n",
    "$$s = \\sqrt{\\frac{1}{n-1} \\left( \\sum_{i=1}^{n} X_i^2 - n \\bar{X} \\right)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Median"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample median also measures the center of a data set. To compute the median, we order the data from smallest to largest and find the middle. For a data set with an even number of objects, we average the two middle datum.\n",
    "\n",
    "As you likely expect, pandas computes the median:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    5.0\n",
       "B    5.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition seealso\"> \n",
    "    <p class=\"title\"><b>Optional Home Activity</b></p>\n",
    " Calculate the median for the particulate matter example. Store the results in the dictionary <tt>ans_14b_3</tt> with keys \"low\" and \"high\". Hint: You'll need to calculate the median as a float and then save the answers into the dictionary.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'low': 3.18, 'high': 5.75}\n"
     ]
    }
   ],
   "source": [
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "14b-3",
     "locked": true,
     "points": "0.2",
     "solution": false
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Removed autograder test. You may delete this cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Mode and Range"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **sample mode** is the most common element in a sample.\n",
    "\n",
    "Let's review samples A and B:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A  B\n",
       "0   0  4\n",
       "1   0  4\n",
       "2   5  5\n",
       "3  10  6\n",
       "4  10  6"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall, these two data sets have the same mean and median but different variances. Let's compute the mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A  B\n",
       "0   0  4\n",
       "1  10  6"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.mode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting. In data set A, both there are two 0 elements and two 10 elements. Thus 0 and 10 are tied for the mode, and pandas returns two values. Likewise, 4 and 6 are tied for the mode in data set B.\n",
    "\n",
    "Let's look at the particulate matter example together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.11\n",
       "1    1.63\n",
       "Name: PM, dtype: float64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.PM.mode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This suggests that both 1.11 and 1.63 are tied for the mode. Notice that `.mode()` returns a dictionary-like structure. We can access the first mode with the index (key) 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.11"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.PM.mode()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the second mode with index (key) 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.63"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.PM.mode()[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can investigate further using the `.value_counts()` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.63    3\n",
       "1.11    3\n",
       "2.67    2\n",
       "2.96    2\n",
       "5.30    2\n",
       "       ..\n",
       "0.55    1\n",
       "3.67    1\n",
       "1.14    1\n",
       "1.37    1\n",
       "3.90    1\n",
       "Name: PM, Length: 124, dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.PM.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us the number of times each value repeats in the data set, sorted from most to least common."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition seealso\"> \n",
    "    <p class=\"title\"><b>Optional Home Activity</b></p>\n",
    " Calculate the mode for the high elevation particulate matter example. Store the results in the float <tt>ans_14b_4</tt>. Hint: You'll need to either extract the float from the pandas output with key 0 or manually enter your answer.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "14b-4",
     "locked": false,
     "points": "0.2",
     "solution": false
    },
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6.32\n"
     ]
    }
   ],
   "source": [
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "14b-4",
     "locked": true,
     "points": "0.2",
     "solution": false
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Removed autograder test. You may delete this cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **sample range** is the different between the smallest and largest values in a sample. Pandas allows us to easily calculate these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A  B\n",
       "0   0  4\n",
       "1   0  4\n",
       "2   5  5\n",
       "3  10  6\n",
       "4  10  6"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect the data set\n",
    "my_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    0\n",
       "B    4\n",
       "dtype: int64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# identify smallest values\n",
    "my_data.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    10\n",
       "B     6\n",
       "dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# identify largest values\n",
    "my_data.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quartiles and Percentiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The median is the 50%-ile of the data set. Half of the elements are above and half are below. But we can generalize this to any numeric value between 0% and 100%. You'll notice that the pandas documentation says **quantile** instead of **percentile**. These terms mean the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.18"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate 50% quantile, a.k.a, 50%-ile\n",
    "low.PM.quantile(0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.18"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verify this is the median\n",
    "low.PM.median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quartiles** divide the data set into quarters (four pieces). Quartiles are specific percentiles:\n",
    "\n",
    "| Quartiles | Percentile / Quantile |\n",
    "| - | - |\n",
    "| 1st | 25% |\n",
    "| 2nd | 50% |\n",
    "| 3rd | 75% |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the low altitude particulate matter example again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "138"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Determine the number of entries\n",
    "low.PM.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It contains 138 datum. So what happens if we want to compute the 52.1%-ile?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas will give us a number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.34754"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low.PM.quantile(0.521)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the answer ends in ...000004. *What is going on?*\n",
    "\n",
    "Because there are only 138 elements, the quantiles increment by 0.724...%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.007246376811594203"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/138"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is doing interpolation under the hood because there is not a datum exactly at the 52.1%-ile. The ...000004 ending is an artifact of inexact floating point arithmetic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition seealso\"> \n",
    "    <p class=\"title\"><b>Optional Home Activity</b></p>\n",
    " Calculate the 80%-ile for the high elevation particulate matter example. Store the results in the float <tt>ans_14b_5</tt>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8.822000000000001\n"
     ]
    }
   ],
   "source": [
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "14b-5",
     "locked": true,
     "points": "0.1",
     "solution": false
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Removed autograder test. You may delete this cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multivariate Data Example: Exam 1 Scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file `Exam-1-scores.csv` contains anonymized numeric scores from Exam 1 this semester. Let's open the file with Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Open the file\n",
    "exam1_full = pd.read_csv(\"https://raw.githubusercontent.com/ndcbe/data-and-computing/main/notebooks/data/Exam_1_scores.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Total Score   1-A   1-B  2-A-1  2-A-2  2-B  2-C-1  2-C-2  2-C-3  3-A  \\\n",
      "0         60.00  4.00  4.00   2.00   2.00  3.0   2.00   2.00    3.0  3.0   \n",
      "1         59.75  4.00  4.00   2.00   2.00  3.0   2.00   2.00    3.0  3.0   \n",
      "2         59.75  4.00  4.00   2.00   2.00  3.0   2.00   2.00    3.0  3.0   \n",
      "3         59.75  4.00  4.00   2.00   2.00  3.0   2.00   2.00    3.0  3.0   \n",
      "4         59.25  4.00  4.00   2.00   2.00  3.0   2.00   2.00    3.0  3.0   \n",
      "..          ...   ...   ...    ...    ...  ...    ...    ...    ...  ...   \n",
      "61        42.25  2.25  1.00   2.00   2.00  3.0   2.00   2.00    1.0  3.0   \n",
      "62        39.00  2.25  1.00   1.25   1.25  3.0   1.25   0.75    1.0  3.0   \n",
      "63        22.00  2.25  0.25   0.25   0.25  0.0   2.00   2.00    1.0  3.0   \n",
      "64        21.75  4.00  0.00   0.25   0.25  0.0   1.50   2.00    3.0  3.0   \n",
      "65        15.50  0.25  1.00   0.25   0.25  0.0   0.50   0.75    0.0  3.0   \n",
      "\n",
      "    3-B-1  3-B-2  3-B-3  4-A-1  4-A-2   4-B  4-C-1  4-C-2   5-A   5-B  \n",
      "0    4.00   1.00   2.00    2.0   2.00  3.00    4.0   3.00  7.00  7.00  \n",
      "1    4.00   1.00   2.00    2.0   1.75  3.00    4.0   3.00  7.00  7.00  \n",
      "2    4.00   1.00   2.00    2.0   1.75  3.00    4.0   3.00  7.00  7.00  \n",
      "3    4.00   1.00   2.00    2.0   1.75  3.00    4.0   3.00  7.00  7.00  \n",
      "4    4.00   1.00   2.00    2.0   1.50  3.00    4.0   3.00  6.75  7.00  \n",
      "..    ...    ...    ...    ...    ...   ...    ...    ...   ...   ...  \n",
      "61   4.00   1.00   2.00    2.0   1.75  3.00    1.5   0.00  7.00  1.75  \n",
      "62   4.00   1.00   2.00    1.0   0.00  0.25    4.0   2.75  5.50  3.75  \n",
      "63   0.75   1.00   1.00    2.0   1.25  1.00    1.5   0.00  1.00  1.50  \n",
      "64   0.75   1.00   0.00    0.0   1.00  0.00    0.0   0.00  5.00  0.00  \n",
      "65   0.25   0.25   0.25    2.0   1.25  3.00    1.0   0.00  0.00  1.50  \n",
      "\n",
      "[66 rows x 20 columns]\n"
     ]
    }
   ],
   "source": [
    "# Print entire dataframe\n",
    "print(exam1_full)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see 65 rows (students) and 20 columns. A perfect score was 60 points. Let's loop over the column names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Score\n",
      "1-A\n",
      "1-B\n",
      "2-A-1\n",
      "2-A-2\n",
      "2-B\n",
      "2-C-1\n",
      "2-C-2\n",
      "2-C-3\n",
      "3-A\n",
      "3-B-1\n",
      "3-B-2\n",
      "3-B-3\n",
      "4-A-1\n",
      "4-A-2\n",
      "4-B\n",
      "4-C-1\n",
      "4-C-2\n",
      "5-A\n",
      "5-B\n"
     ]
    }
   ],
   "source": [
    "for c in exam1_full.columns:\n",
    "    print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a new Pandas dataframe with only the problem totals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# create empty dictionary\n",
    "new_data = {}\n",
    "new_data['Total'] = exam1_full['Total Score']\n",
    "new_data['P1'] = exam1_full['1-A'] + exam1_full['1-B']\n",
    "new_data['P2'] = exam1_full['2-A-1'] + exam1_full['2-A-2'] + exam1_full['2-B']\n",
    "new_data['P2'] += exam1_full['2-C-1'] + exam1_full['2-C-2'] + exam1_full['2-C-3']\n",
    "new_data['P3'] = exam1_full['3-A'] + exam1_full['3-B-1'] + exam1_full['3-B-2'] + exam1_full['3-B-3']\n",
    "new_data['P4'] = exam1_full['4-A-1'] + exam1_full['4-A-2'] + exam1_full['4-B']\n",
    "new_data['P4'] += exam1_full['4-B'] + exam1_full['4-C-1'] + exam1_full['4-C-2']\n",
    "new_data['P5'] = exam1_full['5-A'] + exam1_full['5-B']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "exam1 = pd.DataFrame(new_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note\"> \n",
    "<p class=\"title\"><b>Class Activity</b></p>\n",
    " Print the first and last five elements of the data set.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 22736,
     "status": "ok",
     "timestamp": 1551970243340,
     "user": {
      "displayName": "Alexander Dowling",
      "photoUrl": "https://lh3.googleusercontent.com/-LChdQ2m5OQE/AAAAAAAAAAI/AAAAAAAAAA0/JeXJe4vQJ7M/s64/photo.jpg",
      "userId": "00988067626794866502"
     },
     "user_tz": 300
    },
    "id": "xF9R5s_o30Tf",
    "outputId": "c3c6077a-3147-465f-dbd7-3f4b50cee4b2",
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>P1</th>\n",
       "      <th>P2</th>\n",
       "      <th>P3</th>\n",
       "      <th>P4</th>\n",
       "      <th>P5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60.00</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>17.00</td>\n",
       "      <td>14.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59.75</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>16.75</td>\n",
       "      <td>14.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>59.75</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>16.75</td>\n",
       "      <td>14.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>59.75</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>16.75</td>\n",
       "      <td>14.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59.25</td>\n",
       "      <td>8.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>16.50</td>\n",
       "      <td>13.75</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Total   P1    P2    P3     P4     P5\n",
       "0  60.00  8.0  14.0  10.0  17.00  14.00\n",
       "1  59.75  8.0  14.0  10.0  16.75  14.00\n",
       "2  59.75  8.0  14.0  10.0  16.75  14.00\n",
       "3  59.75  8.0  14.0  10.0  16.75  14.00\n",
       "4  59.25  8.0  14.0  10.0  16.50  13.75"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print top (\"head\") of data frame\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>P1</th>\n",
       "      <th>P2</th>\n",
       "      <th>P3</th>\n",
       "      <th>P4</th>\n",
       "      <th>P5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>42.25</td>\n",
       "      <td>3.25</td>\n",
       "      <td>12.00</td>\n",
       "      <td>10.00</td>\n",
       "      <td>11.25</td>\n",
       "      <td>8.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>39.00</td>\n",
       "      <td>3.25</td>\n",
       "      <td>8.50</td>\n",
       "      <td>10.00</td>\n",
       "      <td>8.25</td>\n",
       "      <td>9.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>22.00</td>\n",
       "      <td>2.50</td>\n",
       "      <td>5.50</td>\n",
       "      <td>5.75</td>\n",
       "      <td>6.75</td>\n",
       "      <td>2.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>21.75</td>\n",
       "      <td>4.00</td>\n",
       "      <td>7.00</td>\n",
       "      <td>4.75</td>\n",
       "      <td>1.00</td>\n",
       "      <td>5.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>15.50</td>\n",
       "      <td>1.25</td>\n",
       "      <td>1.75</td>\n",
       "      <td>3.75</td>\n",
       "      <td>10.25</td>\n",
       "      <td>1.50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Total    P1     P2     P3     P4    P5\n",
       "61  42.25  3.25  12.00  10.00  11.25  8.75\n",
       "62  39.00  3.25   8.50  10.00   8.25  9.25\n",
       "63  22.00  2.50   5.50   5.75   6.75  2.50\n",
       "64  21.75  4.00   7.00   4.75   1.00  5.00\n",
       "65  15.50  1.25   1.75   3.75  10.25  1.50"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print bottom (\"tail\") of data frame\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note\"> \n",
    "<p class=\"title\"><b>Class Activity</b></p>\n",
    " Compute the mean, median, mode, standard deviation, and quartiles using pandas.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Total    51.261364\n",
       "P1        5.734848\n",
       "P2       12.875000\n",
       "P3        9.435606\n",
       "P4       14.924242\n",
       "P5       11.083333\n",
       "dtype: float64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mean\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Total    52.50\n",
       "P1        5.00\n",
       "P2       14.00\n",
       "P3       10.00\n",
       "P4       16.50\n",
       "P5       12.25\n",
       "dtype: float64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# median\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>P1</th>\n",
       "      <th>P2</th>\n",
       "      <th>P3</th>\n",
       "      <th>P4</th>\n",
       "      <th>P5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>58.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>12.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Total   P1    P2    P3    P4    P5\n",
       "0   58.5  5.0  14.0  10.0  17.0  12.5"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mode\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Total    8.487823\n",
       "P1       1.839146\n",
       "P2       2.156051\n",
       "P3       1.281008\n",
       "P4       3.065932\n",
       "P5       3.025956\n",
       "dtype: float64"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# standard deviation\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total    49.3125\n",
      "P1        5.0000\n",
      "P2       12.3125\n",
      "P3        9.5000\n",
      "P4       13.5625\n",
      "P5        9.9375\n",
      "Name: 0.25, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# 25%-ile\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total    52.50\n",
      "P1        5.00\n",
      "P2       14.00\n",
      "P3       10.00\n",
      "P4       16.50\n",
      "P5       12.25\n",
      "Name: 0.5, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# 50%-ile\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total    56.5000\n",
      "P1        8.0000\n",
      "P2       14.0000\n",
      "P3       10.0000\n",
      "P4       17.0000\n",
      "P5       12.9375\n",
      "Name: 0.75, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# 75%-ile\n",
    "# Add your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas offers a single function to compute these summary statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>P1</th>\n",
       "      <th>P2</th>\n",
       "      <th>P3</th>\n",
       "      <th>P4</th>\n",
       "      <th>P5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>66.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>66.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>51.261364</td>\n",
       "      <td>5.734848</td>\n",
       "      <td>12.875000</td>\n",
       "      <td>9.435606</td>\n",
       "      <td>14.924242</td>\n",
       "      <td>11.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.487823</td>\n",
       "      <td>1.839146</td>\n",
       "      <td>2.156051</td>\n",
       "      <td>1.281008</td>\n",
       "      <td>3.065932</td>\n",
       "      <td>3.025956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>15.500000</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>1.750000</td>\n",
       "      <td>3.750000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>49.312500</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>12.312500</td>\n",
       "      <td>9.500000</td>\n",
       "      <td>13.562500</td>\n",
       "      <td>9.937500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>52.500000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>16.500000</td>\n",
       "      <td>12.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>56.500000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>12.937500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>60.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>14.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Total         P1         P2         P3         P4         P5\n",
       "count  66.000000  66.000000  66.000000  66.000000  66.000000  66.000000\n",
       "mean   51.261364   5.734848  12.875000   9.435606  14.924242  11.083333\n",
       "std     8.487823   1.839146   2.156051   1.281008   3.065932   3.025956\n",
       "min    15.500000   1.250000   1.750000   3.750000   1.000000   1.500000\n",
       "25%    49.312500   5.000000  12.312500   9.500000  13.562500   9.937500\n",
       "50%    52.500000   5.000000  14.000000  10.000000  16.500000  12.250000\n",
       "75%    56.500000   8.000000  14.000000  10.000000  17.000000  12.937500\n",
       "max    60.000000   8.000000  14.000000  10.000000  17.000000  14.000000"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# all in one line\n",
    "exam1.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Covariance\n",
    "\n",
    "Standard devation (and variance) measure the average squared distance from each element of a dataset and the mean. Standard deviation is for a single dimension.\n",
    "\n",
    "Often, we want to know to the extent two variables are related. Let $X_1$, $X_2$, ..., $X_n$ and $Y_1$, $Y_2$, ..., $Y_n$ be the sample. Each pair ($X_i$, $Y_i$) corresponds to one experiment. For example, $X$ could be the effluent temperature and $Y$ could be the conversion for an adiabitic reactor. The experiment was repeated for $n$ trials.\n",
    "\n",
    "\n",
    "The **sample covariance** if a generalization of variance from one to two dimensions:\n",
    "\n",
    "$$s_{X,Y}^2 = \\frac{1}{n-1} \\sum_{i}^{n} (X_i - \\bar{X})(Y_i - \\bar{Y})$$\n",
    "\n",
    "$\\bar{X}$ and $\\bar{Y}$ are the sample means for variables $X$ and $Y$.\n",
    "\n",
    "If both $X_i$ and $Y_i$ tend to move together, i.e., both are either above ($Y_i > \\bar{Y}$ when $X_{i} > \\bar{X}$) or below ($Y_i < \\bar{Y}$ when $X_{i} < \\bar{X}$) their sample means, then $s_{X,Y}^2 >> 0$.\n",
    "\n",
    "If they move in oppositive directions, i.e., $Y_i > \\bar{Y}$ when $X_{i} < \\bar{X}$ and $Y_i > \\bar{Y}$ when $X_{i} < \\bar{X}$, then $s_{X,Y}^2 << 0$.\n",
    "\n",
    "If there is not a strong trend, then $s_{X,Y}^2 \\approx 0$.\n",
    "\n",
    "We can quickly compute covariance with Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>P1</th>\n",
       "      <th>P2</th>\n",
       "      <th>P3</th>\n",
       "      <th>P4</th>\n",
       "      <th>P5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>72.043138</td>\n",
       "      <td>10.143444</td>\n",
       "      <td>16.003365</td>\n",
       "      <td>7.843051</td>\n",
       "      <td>19.822028</td>\n",
       "      <td>20.955769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P1</th>\n",
       "      <td>10.143444</td>\n",
       "      <td>3.382459</td>\n",
       "      <td>1.928846</td>\n",
       "      <td>0.750932</td>\n",
       "      <td>2.007488</td>\n",
       "      <td>2.357051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P2</th>\n",
       "      <td>16.003365</td>\n",
       "      <td>1.928846</td>\n",
       "      <td>4.648558</td>\n",
       "      <td>1.763942</td>\n",
       "      <td>3.881731</td>\n",
       "      <td>4.382692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P3</th>\n",
       "      <td>7.843051</td>\n",
       "      <td>0.750932</td>\n",
       "      <td>1.763942</td>\n",
       "      <td>1.640982</td>\n",
       "      <td>2.086393</td>\n",
       "      <td>1.952564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P4</th>\n",
       "      <td>19.822028</td>\n",
       "      <td>2.007488</td>\n",
       "      <td>3.881731</td>\n",
       "      <td>2.086393</td>\n",
       "      <td>9.399942</td>\n",
       "      <td>3.725641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P5</th>\n",
       "      <td>20.955769</td>\n",
       "      <td>2.357051</td>\n",
       "      <td>4.382692</td>\n",
       "      <td>1.952564</td>\n",
       "      <td>3.725641</td>\n",
       "      <td>9.156410</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Total         P1         P2        P3         P4         P5\n",
       "Total  72.043138  10.143444  16.003365  7.843051  19.822028  20.955769\n",
       "P1     10.143444   3.382459   1.928846  0.750932   2.007488   2.357051\n",
       "P2     16.003365   1.928846   4.648558  1.763942   3.881731   4.382692\n",
       "P3      7.843051   0.750932   1.763942  1.640982   2.086393   1.952564\n",
       "P4     19.822028   2.007488   3.881731  2.086393   9.399942   3.725641\n",
       "P5     20.955769   2.357051   4.382692  1.952564   3.725641   9.156410"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exam1.cov()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note\"> \n",
    "<p class=\"title\"><b>Class Activity</b></p>\n",
    " What are the units of sample covariance?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Home Activity Answer:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, it is more convinent to interpret **sample correlation**, which is sample covariance scaled by the standard deviation of each variable.\n",
    "\n",
    "$$\n",
    "r_{X,Y} = \\frac{s_{X,Y}}{s_X \\cdot s_Y}\n",
    "$$\n",
    "\n",
    "Recall, $s_{X,Y}$ is the covariance. $s_{X}$ and $s_{Y}$ are the standard deviations for variable $X$ and $Y$.\n",
    "\n",
    "By construction, correlation is bounded between -1 and 1. During the remainded of the semester, we will see why the following rules hold:\n",
    "\n",
    "| Correlation | Interpretation |\n",
    "| :---: | :---: |\n",
    "| <div style=\"width:290px\"></div> | <div style=\"width:290px\"></div> |\n",
    "| $r_{X,Y} = -1.0$ | **Perfect negative correlation.** Samples $X_i$ and $Y_i$ lie exactly on a straight line with a negative slope. |\n",
    "| $-1.0 < r_{X,Y} < 0$ | **Negative correlation.** If we fit a line to the data $X_i$ and $Y_i$, the slope would be negative.\n",
    "| $r_{X,Y} = 0$ | **No correlation.** If we fit a line to the data $X_i$ and $Y_i$, the slope would be zero. |\n",
    "| $0 < r_{X,Y} < 1$ | **Positive correlation.** If we fit a line to the data $X_i$ and $Y_i$, the slope would be positive.\n",
    "| $r_{X,Y} = -1.0$ | **Perfect positive correlation.** Samples $X_i$ and $Y_i$ lie exactly on a straight line with a positive slope. |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note\"> \n",
    "<p class=\"title\"><b>Class Activity</b></p>\n",
    " What are the units of sample correlation?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Activity Answer:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the correlation for the exam1 data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>P1</th>\n",
       "      <th>P2</th>\n",
       "      <th>P3</th>\n",
       "      <th>P4</th>\n",
       "      <th>P5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.649790</td>\n",
       "      <td>0.874492</td>\n",
       "      <td>0.721335</td>\n",
       "      <td>0.761709</td>\n",
       "      <td>0.815915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P1</th>\n",
       "      <td>0.649790</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.486432</td>\n",
       "      <td>0.318737</td>\n",
       "      <td>0.356020</td>\n",
       "      <td>0.423536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P2</th>\n",
       "      <td>0.874492</td>\n",
       "      <td>0.486432</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.638665</td>\n",
       "      <td>0.587224</td>\n",
       "      <td>0.671768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P3</th>\n",
       "      <td>0.721335</td>\n",
       "      <td>0.318737</td>\n",
       "      <td>0.638665</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.531229</td>\n",
       "      <td>0.503722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P4</th>\n",
       "      <td>0.761709</td>\n",
       "      <td>0.356020</td>\n",
       "      <td>0.587224</td>\n",
       "      <td>0.531229</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.401583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P5</th>\n",
       "      <td>0.815915</td>\n",
       "      <td>0.423536</td>\n",
       "      <td>0.671768</td>\n",
       "      <td>0.503722</td>\n",
       "      <td>0.401583</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Total        P1        P2        P3        P4        P5\n",
       "Total  1.000000  0.649790  0.874492  0.721335  0.761709  0.815915\n",
       "P1     0.649790  1.000000  0.486432  0.318737  0.356020  0.423536\n",
       "P2     0.874492  0.486432  1.000000  0.638665  0.587224  0.671768\n",
       "P3     0.721335  0.318737  0.638665  1.000000  0.531229  0.503722\n",
       "P4     0.761709  0.356020  0.587224  0.531229  1.000000  0.401583\n",
       "P5     0.815915  0.423536  0.671768  0.503722  0.401583  1.000000"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exam1.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note\"> \n",
    "<p class=\"title\"><b>Class Activity</b></p>\n",
    " Interpret correlation for the exam data. Why is the diagonal exactly 1.0. What would a negative correlation mean?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we first normalize the scores by the number of available points?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Make a copy of the DataFrame\n",
    "exam1_norm = exam1.copy()\n",
    "\n",
    "# Divide by the total number of available points\n",
    "exam1_norm[\"P1\"] = exam1_norm[\"P1\"] / 8\n",
    "exam1_norm[\"P2\"] = exam1_norm[\"P2\"] / 14\n",
    "exam1_norm[\"P3\"] = exam1_norm[\"P3\"] / 10\n",
    "exam1_norm[\"P4\"] = exam1_norm[\"P4\"] / 17\n",
    "exam1_norm[\"P5\"] = exam1_norm[\"P5\"] / 14\n",
    "exam1_norm[\"Total\"] = exam1_norm[\"Total\"] / 60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>P1</th>\n",
       "      <th>P2</th>\n",
       "      <th>P3</th>\n",
       "      <th>P4</th>\n",
       "      <th>P5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>0.020012</td>\n",
       "      <td>0.021132</td>\n",
       "      <td>0.019052</td>\n",
       "      <td>0.013072</td>\n",
       "      <td>0.019433</td>\n",
       "      <td>0.024947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P1</th>\n",
       "      <td>0.021132</td>\n",
       "      <td>0.052851</td>\n",
       "      <td>0.017222</td>\n",
       "      <td>0.009387</td>\n",
       "      <td>0.014761</td>\n",
       "      <td>0.021045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P2</th>\n",
       "      <td>0.019052</td>\n",
       "      <td>0.017222</td>\n",
       "      <td>0.023717</td>\n",
       "      <td>0.012600</td>\n",
       "      <td>0.016310</td>\n",
       "      <td>0.022361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P3</th>\n",
       "      <td>0.013072</td>\n",
       "      <td>0.009387</td>\n",
       "      <td>0.012600</td>\n",
       "      <td>0.016410</td>\n",
       "      <td>0.012273</td>\n",
       "      <td>0.013947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P4</th>\n",
       "      <td>0.019433</td>\n",
       "      <td>0.014761</td>\n",
       "      <td>0.016310</td>\n",
       "      <td>0.012273</td>\n",
       "      <td>0.032526</td>\n",
       "      <td>0.015654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P5</th>\n",
       "      <td>0.024947</td>\n",
       "      <td>0.021045</td>\n",
       "      <td>0.022361</td>\n",
       "      <td>0.013947</td>\n",
       "      <td>0.015654</td>\n",
       "      <td>0.046716</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Total        P1        P2        P3        P4        P5\n",
       "Total  0.020012  0.021132  0.019052  0.013072  0.019433  0.024947\n",
       "P1     0.021132  0.052851  0.017222  0.009387  0.014761  0.021045\n",
       "P2     0.019052  0.017222  0.023717  0.012600  0.016310  0.022361\n",
       "P3     0.013072  0.009387  0.012600  0.016410  0.012273  0.013947\n",
       "P4     0.019433  0.014761  0.016310  0.012273  0.032526  0.015654\n",
       "P5     0.024947  0.021045  0.022361  0.013947  0.015654  0.046716"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute covariance matrix\n",
    "exam1_norm.cov()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total</th>\n",
       "      <th>P1</th>\n",
       "      <th>P2</th>\n",
       "      <th>P3</th>\n",
       "      <th>P4</th>\n",
       "      <th>P5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.649790</td>\n",
       "      <td>0.874492</td>\n",
       "      <td>0.721335</td>\n",
       "      <td>0.761709</td>\n",
       "      <td>0.815915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P1</th>\n",
       "      <td>0.649790</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.486432</td>\n",
       "      <td>0.318737</td>\n",
       "      <td>0.356020</td>\n",
       "      <td>0.423536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P2</th>\n",
       "      <td>0.874492</td>\n",
       "      <td>0.486432</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.638665</td>\n",
       "      <td>0.587224</td>\n",
       "      <td>0.671768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P3</th>\n",
       "      <td>0.721335</td>\n",
       "      <td>0.318737</td>\n",
       "      <td>0.638665</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.531229</td>\n",
       "      <td>0.503722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P4</th>\n",
       "      <td>0.761709</td>\n",
       "      <td>0.356020</td>\n",
       "      <td>0.587224</td>\n",
       "      <td>0.531229</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.401583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P5</th>\n",
       "      <td>0.815915</td>\n",
       "      <td>0.423536</td>\n",
       "      <td>0.671768</td>\n",
       "      <td>0.503722</td>\n",
       "      <td>0.401583</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Total        P1        P2        P3        P4        P5\n",
       "Total  1.000000  0.649790  0.874492  0.721335  0.761709  0.815915\n",
       "P1     0.649790  1.000000  0.486432  0.318737  0.356020  0.423536\n",
       "P2     0.874492  0.486432  1.000000  0.638665  0.587224  0.671768\n",
       "P3     0.721335  0.318737  0.638665  1.000000  0.531229  0.503722\n",
       "P4     0.761709  0.356020  0.587224  0.531229  1.000000  0.401583\n",
       "P5     0.815915  0.423536  0.671768  0.503722  0.401583  1.000000"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute correlation matrix\n",
    "exam1_norm.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note\"> \n",
    "<p class=\"title\"><b>Class Activity</b></p>\n",
    " Why is the correlation matrix not changed by scaling?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary Statistics for Categorical Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Categorical variables are qualititative. Recall our example from earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Specimen</th>\n",
       "      <th>Torque</th>\n",
       "      <th>Failure Location</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>165</td>\n",
       "      <td>Weld</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>237</td>\n",
       "      <td>Beam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>222</td>\n",
       "      <td>Beam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>255</td>\n",
       "      <td>Beam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>194</td>\n",
       "      <td>Weld</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Specimen  Torque Failure Location\n",
       "0         1     165             Weld\n",
       "1         2     237             Beam\n",
       "2         3     222             Beam\n",
       "3         4     255             Beam\n",
       "4         5     194             Weld"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here failure location was a categorical variable. We cannot compute the median, standard deviation, or range for a categorical variable. Instead, we often compute **frequencies** by counting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Beam    3\n",
       "Weld    2\n",
       "Name: Failure Location, dtype: int64"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_df['Failure Location'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can instead compute the **sample proportions** by normalizing (dividing by) the total number of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Beam    0.6\n",
       "Weld    0.4\n",
       "Name: Failure Location, dtype: float64"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_df['Failure Location'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary Statistics and Population Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each sample statistic we have discussed (e.g., mean, median, standard deviation, covariance) has a counterpart for the population. We will call numerical summaries of samples **statistics** and numerical summaries of populations **parameters**. A central idea in data analysis is to use statistics to estimate/infer parameters. Often, we cannot directly measure a population. So instead we sample."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "L14-Descriptive-Statistics.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}