From 43a585a17d82019440f33c9ac6dfdf0c7f575c03 Mon Sep 17 00:00:00 2001 From: Bryan Kwong Date: Thu, 23 Apr 2020 23:32:38 -0700 Subject: [PATCH 1/2] Added section stated in the branch --- .../7_2_Visualizing_Higher_Dimensions.ipynb | 731 ++++++++++++++++++ 10-Textual-Data/10.3 Distance Metrics.ipynb | 660 ++++++++++++++++ .../10.4 Visual Analysis of Text.ipynb | 505 ++++++++++++ 3 files changed, 1896 insertions(+) create mode 100644 07-Unsupervised-Learning/7_2_Visualizing_Higher_Dimensions.ipynb create mode 100644 10-Textual-Data/10.3 Distance Metrics.ipynb create mode 100644 10-Textual-Data/10.4 Visual Analysis of Text.ipynb diff --git a/07-Unsupervised-Learning/7_2_Visualizing_Higher_Dimensions.ipynb b/07-Unsupervised-Learning/7_2_Visualizing_Higher_Dimensions.ipynb new file mode 100644 index 0000000..c2cb755 --- /dev/null +++ b/07-Unsupervised-Learning/7_2_Visualizing_Higher_Dimensions.ipynb @@ -0,0 +1,731 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "name": "7.2 Visualizing Higher Dimensions.ipynb", + "provenance": [], + "collapsed_sections": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "XTPLBCWKzfWS", + "colab_type": "text" + }, + "source": [ + "# 7.2 Visualizing Higher Dimensions\n", + "\n", + "By this point, we hope we've convinced you how important it is to visualize your data. While summary statistics are helpful, it doesn't provide us with a good grasp of what the entire dataset looks like. In two dimension, we can use a 2-D scatter plot. In three dimension, we can use a 3-D scatter plot. But what if we have more than three dimensions? This chapter talks about how we can visualize data that is beyond 3 dimensions." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f1mm_fCWm9dO", + "colab_type": "text" + }, + "source": [ + "## Goal of Visualizing Higher Dimensions\n", + "\n", + ">\"I am a Tralfamadorian, seeing all time as you might see a stretch of the Rocky Mountains. All time is all time. It does not change. It does not lend itself to warnings or explanations. It simply is.\" \n", + ">\n", + ">-Kurt Vonnegate in \"Slaughterhour-Five\"\n", + "\n", + "We unfortunately are not Tralfamadorians, instead we are three dimensional beings who can't visually see a fourth dimension like its a location on the Rocky Mountains. However, this doesn't mean that the fourth dimension is meaingless to us. We can derive a lot of understand from understanding the higher dimensions. The problem is, we can't it and thus we can't plot it. \n", + "\n", + "Fortunately, very clever mathematicians throughout history has invented techniques to allow us to simulate what the higher dimension would look like. The rest of the section will discuss these techniques." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XR-JkcQuwozv", + "colab_type": "text" + }, + "source": [ + "## Using Size and Color\n", + "\n", + "This is more of a review from previous sections but one way to visualize more dimensions is by using the size and color attributes of your scatter plots. This is rather intuitive. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "kgChDoxMwyKa", + "colab_type": "code", + "outputId": "cc578404-1e67-476a-f736-e070bc9c233e", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 368 + } + }, + "source": [ + "import pandas as pd\n", + "from scipy import stats\n", + "from sklearn.linear_model import LinearRegression\n", + "from sklearn.preprocessing import LabelEncoder, StandardScaler\n", + "import altair as alt\n", + "\n", + "df_bordeaux = pd.read_csv(\"http://dlsun.github.io/pods/data/bordeaux.csv\")\n", + "\n", + "alt.Chart(df_bordeaux).mark_circle().encode(\n", + " alt.X('age',\n", + " scale=alt.Scale(zero=False)\n", + " ),\n", + " alt.Y('sep',\n", + " scale=alt.Scale(zero=False)\n", + " ),\n", + " color=\"summer\",\n", + " size=\"win\"\n", + ")" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "alt.Chart(...)" + ], + "text/html": [ + "\n", + "
\n", + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 30 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gxV83MG_y2LJ", + "colab_type": "text" + }, + "source": [ + "I am sure you can see the limitations of this method: you can only go up for 4 dimensions (5 if you use a 3-D scatter plot). This is still worth mentioning as sometimes, this may be all you need. \n", + "\n", + "For higher dimensions, we should consider either feature selection or feature reduction. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tlWDTyDsr5Zv", + "colab_type": "text" + }, + "source": [ + "## Feature Selection \n", + "\n", + "If we want to compress 10 dimensions worth of data into 2 dimensions, we're bound to lose some detail during that compression. We can measure how much detail we kept at the end with the explained variance ratio which gives the percentage of variance/detail we kept after the compression. The higher the ratio, the more variance and detail we kept. \n", + "\n", + "One way very simple, almost trivial, way to only visualize higher dimensional data is to only plot the two dimensions that explains the most variation in the data. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "g9kxLPD30dxa", + "colab_type": "code", + "outputId": "c1aedb62-573b-406c-cfa6-844ab4054122", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 136 + } + }, + "source": [ + "sclr = StandardScaler()\n", + "df_bordeaux = pd.DataFrame(sclr.fit_transform(df_bordeaux.dropna()), columns=df_bordeaux.columns)\n", + "\n", + "X = df_bordeaux.drop(\"price\", axis=1)\n", + "y = df_bordeaux[\"price\"]\n", + "\n", + "reg = LinearRegression()\n", + "reg.fit(X, y)\n", + "\n", + "scores = pd.Series(dtype=float, name=\"R^2 Values\")\n", + "for column in X.columns: \n", + " reg = LinearRegression()\n", + " reg.fit(X[[column]], y)\n", + " scores[column] = reg.score(X[[column]], y)\n", + "\n", + "scores.sort_values(ascending=False)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "summer 0.343538\n", + "sep 0.323582\n", + "age 0.206936\n", + "year 0.206936\n", + "har 0.199621\n", + "win 0.053456\n", + "Name: R^2 Values, dtype: float64" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 31 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qHprVgDLxKyE", + "colab_type": "text" + }, + "source": [ + "As we can see above, average summer temperature **summer** and average september temperature **sep** are the two variables that explain the most variance in the quality of the wine **price**. Thus, if we want to get the best representation of the dataset with only two dimensions, we can make a scatterplot of **summer** vs **sep**. However, even with the two variables that explain the most variation, we can only capture 33% of the variation of the original data. The other 66% is lost to the other features we chose to ignore. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "FcUCeYGOx2nP", + "colab_type": "code", + "outputId": "16dc19dd-f2ac-44b7-8ac6-0236cac283d4", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 330 + } + }, + "source": [ + "explained_var = X[[\"summer\", \"sep\"]].var(axis=0).sum() / X.var(axis=0).sum() \n", + "\n", + "print(\"% Variance Explained:\", explained_var, end=\"\\n\\n\")\n", + "df_bordeaux.plot.scatter(x=\"summer\", y=\"sep\")" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "% Variance Explained: 0.3333333333333333\n", + "\n" + ], + "name": "stdout" + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 32 + }, + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAUNklEQVR4nO3dfYxcV33G8efZeLu26rQYrwvEDpiQqLwaJ2zTgCmF8NKQVqZgKFDRgqByU4RU1Ao7CLX0TY1spP5RUVqshAKCBigmtVsCIWAoL2rSrINf8kJCQEmzbkrM1glZsLfrzK9/zF083njt2ezcOefO+X6kVWbvzO793ZvxPHvOPedcR4QAAOUZSl0AACANAgAACkUAAEChCAAAKBQBAACFWpK6gIUYHR2NtWvXpi4DABpl7969P4yIVXO3NyoA1q5dq/Hx8dRlAECj2L7vVNvpAgKAQhEAAFAoAgAACkUAAEChCAAAKBQBAABzTE5Na//9D2lyajp1KbVq1DBQAKjbrn2HtHXnAQ0PDWmm1dL2Teu0cf3q1GXVghYAAFQmp6a1decBHZtp6ZHp4zo209KWnQcGtiVAAABAZeLIUQ0PnfyxODw0pIkjRxNVVC8CAAAqa1Ys00yrddK2mVZLa1YsS1RRvQgAAKisXD6i7ZvWaenwkM4eWaKlw0PavmmdVi4fSV1aLbgIDAAdNq5frQ3nj2riyFGtWbFsYD/8JQIAAB5j5fKRgf7gn0UXEAAUigAAgEIRAABQKAIAAApFAABAoQgAACgUAQAAhSIAAKBQBAAAFIoAAIBCEQAAUCgCAAAKlSwAbJ9r+6u277B9u+0/TFULAJQo5WqgxyX9cUTcavtsSXtt3xgRdySsCQCKkawFEBEPRMSt1eNHJN0paTDvvAwAGcriGoDttZIulHTzKZ7bbHvc9vjhw4f7XRoADKzkAWB7uaSdkt4dET+a+3xE7IiIsYgYW7VqVf8LBIABlTQAbA+r/eH/yYj4XMpaAKA0KUcBWdI1ku6MiL9JVQcAlCplC2CDpN+RdKntfdXX5QnrAYCiJBsGGhHflORU+weA0iW/CAwASIMAAIBCEQAAUCgCAAAKRQAAQKEIAAAoFAEAAIUiAACgUAQAABSKAACAQhEAAFAoAgAACkUAAEChCACgzyanprX//oc0OTWduhQULtly0ECJdu07pK07D2h4aEgzrZa2b1qnjetXpy4LhaIFAPTJ5NS0tu48oGMzLT0yfVzHZlrasvMALQEkQwAAfTJx5KiGh07+Jzc8NKSJI0cTVYTSEQBAn6xZsUwzrdZJ22ZaLa1ZsSxRRSgdAQD0ycrlI9q+aZ2WDg/p7JElWjo8pO2b1mnl8pHUpaFQXAQG+mjj+tXacP6oJo4c1ZoVy/jwR1IEANBnK5eP8MGPLNAFBACFIgAAoFAEAADMUcpsba4BAECHkmZr0wIAgEpps7UJAACo9Gu2di5dTHQBAUClH7O1c+piogUAAJW6Z2vn1sVECwAAOtQ5W3u2i+mYTrQyZruYUkwOJAAAYI66ZmvntiAgXUAA0Ce5LQhICwAA+iinBQEJAADos1wWBKQLCAAKlTQAbH/E9oO2b0tZBwCUKHUL4KOSLktcAxool5mU/VDSsaK/kl4DiIiv216bsgY0T04zKetW0rGi/1K3AM7I9mbb47bHDx8+nLocJJbbTMo6lXSsSCP7AIiIHRExFhFjq1atSl0OEuvXYl05KOlYkUb2AQB0ym0mZZ1KOlakQQCgUXKbSVmnko4VaTgi0u3cvlbSSyWNSvqBpPdHxDXzvX5sbCzGx8f7VB1yNjk1ncVMyn4o6VhRD9t7I2Js7vbUo4DenHL/aK5cZlL2Q0nHiv6iCwgACkUAAEChCAAAKBQBAACFIgAAoFAEAFAwFporGzeEAQrFQnOgBQAUiIXmIBEAQJEGYaE5uq8Wjy4goEBNX2iO7qveoAUAFKjJC83RfdU7tACAQm1cv1obzh9t3EJzs91Xx3SiBTPbfdWUY8gFAQAUrIkLzTW9+yondAEBaJQmd1/lhhYAgMZpavdVbggAAI3UxO6r3NAFBACFIgAAoFAEAAAUigAAgEIRAABQKAIAAApFAABAoRYUALZ/zvbZdRUDAOifrgLA9i/ZPijpgKTbbO+3/YJ6S0PTsV47kLduZwJfI+mdEfENSbL9Ykn/KGldXYWh2VivHchft11Aj85++EtSRHxT0vF6SkLTsV470AzdtgD+3faHJV0rKSS9UdLXbF8kSRFxa031oYFYrx1ohm4D4PnVf98/Z/uFagfCpT2rCI03COu1T05NF7HSZCnHiVPrKgAi4mV1F4LBMbte+5Y51wCa8gFTyvWLUo4T83NEnPlF9pMk/bWkcyLi1bafLemFEXFN3QV2Ghsbi/Hx8X7uEovQxL8uJ6emtWHbHh2bOdGCWTo8pG9tvbQxx9CNUo4Tbbb3RsTY3O3dXgT+qKQbJJ1TfX+3pHf3pjQMqpXLR/T8c5/QqA+U2esXnWavX8ynicNdH89xPh5NPDcl6fYawGhEfMb2eyUpIo7bfrTGuoAkFnr9oqndKP24TtPUc1OSblsAP7a9Uu0LvrJ9iaSHa6sKSGQh95tt8nDXuu+r2+RzU5JuWwB/JGm3pGfY/pakVZJeX1tVQELd3m+26cNd67yvbtPPTSm6DYBnSHq1pHMlbZL0ywv4WaBxurnf7CAMd63rvrqDcG5K0G0X0J9ExI8krZD0MkkfkvT3i9257cts32X7HttXLvb3Af1UdzdKk3FumqHbYaDfjogLbV8l6WBE/NPstse9Y/sstUcTvVLShKRbJL05Iu6Y72cYBoocNXG4a79wbvIw3zDQbrtxDlVLQbxS0jbbI1r8vQQulnRPRHy/KvBTkl4jad4AAHJUVzfKIODc5K3bD/HfUnsewK9FxEOSnijpPYvc92pJ93d8P1FtO4ntzbbHbY8fPnx4kbsEAMzqdimIn0j6XMf3D0h6oK6i5ux7h6QdUrsLqB/7BIASpLwl5CG1RxXNWlNtA5ApZvYOlpRDOW+RdIHtp6v9wf8mSb+dsB4Ap8HM3sGTrAUQEcclvUvtawt3SvpMRNyeqh4A82Nm72BKOpkrIq6XdH3KGkrGED10K8eZvbx/F4/ZvIWiOY+FyG1mL+/f3kh5ERiJ0JzHQuU0s5f3b+/QAihQjs155K/OxeMWgvdv7xAABcqtOY/myGFmL+/f3qELqEA5NeeBheL92ztdLQaXCxaD6y1GUTQD/59OjfPSvcUuBocBlENzHqfHaJf58f5dPLqAgEwx2gV1IwCATM2Oduk0O9oF6AUCADiFHBY9Y7QL6sY1AGCOXPrdZ0e7bJlTC/3e6BUCAOjQ2e8+O9Foy84D2nD+aJIP3lwmX2EwEQBAhxxnmTLaBXXhGgDQYc2KZTo6c/ykbUdnjtPvjoFEAABz2D7t98CgIACADhNHjmrpkrNO2rZ0yVkMvcRAIgCADgy9REkIAKADC42hJIwCQteavPjWQmpf6NDLJp8XlI0AQFdymRz1eDye2rsdetnk8wLQBYQzavKiZHXW3uTzAkgEALrQ5EXJ6qy9yecFkAgAdKHJI2PqrL3J5wXNUtfihAQAzqjJI2PqrL3J5wXNsWvfIW3Ytkdvufpmbdi2R7v3HerZ7+aWkOhak0e71Fl7k88L8jY5Na0N2/bo2MyJlubS4SF9a+ulC3qvcUtILFqTFyWrs/Ymnxfkre7FCekCAoBM1X2diQAAgEzVfZ2JLiAAyFidNwUiAAAgc3VdZ6ILCAAKRQAAfVbXpB5goegCAvqIxeOQE1oAQJ+weBxyQwAAfcLicchNkgCw/Qbbt9tu2X7M9GRgELF4HHKTqgVwm6TXSfp6ov0DfcficchNkovAEXGnJNlOsXsgmTon9QALlf0oINubJW2WpKc+9amJqwEWj8XjkIvaAsD2lyU9+RRPvS8idnX7eyJih6QdUns56B6VBwDFqy0AIuIVdf1uAMDiMQwUAAqVahjoa21PSHqhpM/bviFFHQBQslSjgK6TdF2KfQMA2ugCAoBCEQAAUCgCAAAKRQAAQKEIAAAoFAEAAIUiAACgUAQAABSKAACAQhEAAFAoAgAACkUAAEChCAAAKBQBAACFIgAAoFAEAAAUigAAgEIRAABQKAIAAApFAABAoQiAzE1OTWv//Q9pcmo6dSkABsyS1AVgfrv2HdLWnQc0PDSkmVZL2zet08b1q1OXBWBA0ALI1OTUtLbuPKBjMy09Mn1cx2Za2rLzAC0BAD1DAGRq4shRDQ+d/L9neGhIE0eOJqoIwKAhADK1ZsUyzbRaJ22babW0ZsWyRBUBGDQEQKZWLh/R9k3rtHR4SGePLNHS4SFt37ROK5ePpC4NwIAo4iLw5NS0Jo4c1ZoVyxr1Abpx/WptOH+0kbUDyN/AB0DTR9KsXD7CBz+AWgx0FxAjaQBgfgMdAIykAYD5DXQAMJIGAOY30AHASBoAmN/AXwRmJA0AnNrAB4DESBoAOJWB7gICAMyPAACAQiUJANsfsP0d2wdsX2f7CSnqANBc3Ctj8VK1AG6U9NyIWCfpbknvTVQHgAbate+QNmzbo7dcfbM2bNuj3fsOpS6pkZIEQER8KSKOV9/eJGlNijoANA8z/Hsnh2sAb5f0hfmetL3Z9rjt8cOHD/exLAA5YoZ/79Q2DNT2lyU9+RRPvS8idlWveZ+k45I+Od/viYgdknZI0tjYWNRQKoAGYYZ/79QWABHxitM9b/ttkn5D0ssjgg92AF2ZneG/Zc4qv8z1WbgkE8FsXyZpi6RfjYifpKgBQHMxw783Us0E/qCkEUk32pakmyLiikS1AGggZvgvXpIAiIjzU+wXAHBCDqOAAAAJEAAAUCgCAAAKRQAAQKHcpCH4th+RdFfqOuYYlfTD1EWcQo515ViTlGddOdYk5VlXjjVJedX1tIhYNXdj024Ic1dEjKUuopPt8dxqkvKsK8eapDzryrEmKc+6cqxJyreuTnQBAUChCAAAKFTTAmBH6gJOIceapDzryrEmKc+6cqxJyrOuHGuS8q3rpxp1ERgA0DtNawEAAHqEAACAQmUdAN3ePN72vbYP2t5nezyTmi6zfZfte2xfWWdN1f7eYPt22y3b8w496/O56ramfp+rJ9q+0fZ3q/+umOd1j1bnaZ/t3TXVctpjtz1i+9PV8zfbXltHHQus6W22D3ecm9/rQ00fsf2g7dvmed62/7aq+YDti+quqcu6Xmr74Y5z9af9qKtrEZHtl6RXSVpSPd4mads8r7tX0mguNUk6S9L3JJ0n6Wck7Zf07JrrepakX5T0NUljp3ldP8/VGWtKdK62S7qyenzlad5XUzXXccZjl/ROSf9QPX6TpE9nUNPbJH2wH++hjn2+RNJFkm6b5/nL1b61rCVdIunmTOp6qaR/6+e5WshX1i2AyPDm8V3WdLGkeyLi+xHxf5I+Jek1Ndd1Z0RkNUu6y5r6fq6q3/+x6vHHJP1mzfubTzfH3lnrZyW93NVNNBLW1HcR8XVJ/3ual7xG0sej7SZJT7D9lAzqylrWATDH6W4eH5K+ZHuv7c0Z1LRa0v0d309U23KQ6lzNJ8W5elJEPFA9/h9JT5rndUttj9u+yXYdIdHNsf/0NdUfHg9LWllDLQupSZI2VV0tn7V9bo31dCvnf3MvtL3f9hdsPyd1MZ2SLwXRo5vHvzgiDtn+BbXvMvadKplT1tRz3dTVhb6fqxROV1fnNxERtucbC/206lydJ2mP7YMR8b1e19pA/yrp2oiYtv37ardQLk1cU65uVft9NGX7ckn/IumCxDX9VPIAiB7cPD4iDlX/fdD2dWo3Yx/3h1oPajokqfOvojXVtkU5U11d/o6+nqsu9P1c2f6B7adExANVN8GD8/yO2XP1fdtfk3Sh2v3jvdLNsc++ZsL2Ekk/L2myhzUsuKaI6Nz/1WpfU0mtlvfRYkXEjzoeX2/7Q7ZHIyKLReKy7gLyiZvHb4x5bh5v+2dtnz37WO2LtKe8It+vmiTdIukC20+3/TNqX7yrZRTJQvT7XHUpxbnaLemt1eO3SnpMS8X2Ctsj1eNRSRsk3dHjOro59s5aXy9pz3x/CPWrpjl96xsl3VljPd3aLel3q9FAl0h6uKObLxnbT569ZmP7YrU/c+sM8IVJfRX6dF+S7lG7X29f9TU7GuIcSddXj89Te6TCfkm3q931kLSm6vvLJd2t9l+MtdZU7e+1avd7Tkv6gaQbMjhXZ6wp0blaKekrkr4r6cuSnlhtH5N0dfX4RZIOVufqoKR31FTLY45d0l+o/QeGJC2V9M/V++4/JZ3Xh/Nzppquqt4/+yV9VdIz+1DTtZIekDRTvafeIekKSVdUz1vS31U1H9RpRsL1ua53dZyrmyS9qB91dfvFUhAAUKisu4AAAPUhAACgUAQAABSKAACAQhEAAFAoAgAACkUAAJmpJjPxbxO1402GIlWzoj9fLdJ1m+03un2vhNHq+bFq+QfZ/jPbH7P9Ddv32X6d7e1u31fhi7aHq9fda/uqat33cdsX2b7B9vdsX9Gx7/fYvqVaTO3Pq21r3V6D/+Nqz87OYYE1DDgCAKW6TNJ/R8TzI+K5kr54htc/Q+0FzzZK+oSkr0bE8yQdlfTrHa/7r4hYL+kbkj6q9vINl0ia/aB/ldqLgV0sab2kF9h+SfWzF0j6UEQ8JyLuW/whAqdHAKBUByW90vY2278SEQ+f4fVfiIiZ6ufO0onAOChpbcfrdndsvzkiHomIw5Km3b573Kuqr2+rvVLkM3Vidcj7or2WPdAXyVcDBVKIiLvdvm3g5ZL+yvZX1F7ee/aPoqVzfmS6+rmW7Zk4sYZKSyf/O5ru2D7dsX32dZZ0VUR8uPOXu32rxx8v5piAhaIFgCLZPkfSTyLiE5I+oPZt/e6V9ILqJZtq2vUNkt5ue3lVx+rq3gxA39ECQKmeJ+kDtltqr+T4B5KWSbrG9l+qfR/jnouIL9l+lqT/qFYJnpL0FkmP1rE/4HRYDRQACkUXEAAUigAAgEIRAABQKAIAAApFAABAoQgAACgUAQAAhfp/rOG9WPsCRBEAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [], + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Y6gWdO1CINp-", + "colab_type": "text" + }, + "source": [ + "Additionally, if we look below, using only two features has hindered our predictive accuracy. This sucks! Fortunately, some very clever mathematicians came up with ways to get around this. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "l7re2E6KHUbs", + "colab_type": "code", + "outputId": "24c2f5c2-ff71-420e-874c-2bbdee6f9a8f", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + } + }, + "source": [ + "reg = LinearRegression()\n", + "\n", + "reg.fit(X, y)\n", + "print(\"All Features:\\t\\t\", reg.score(X, y))\n", + "reg.fit(X[[\"summer\", \"sep\"]], y)\n", + "print(\"With PCA Features:\\t\", reg.score(X[[\"summer\", \"sep\"]], y), end=\"\\n\\n\")" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "All Features:\t\t 0.7526018827767169\n", + "With PCA Features:\t 0.4633153344681292\n", + "\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tflTZWS688xW", + "colab_type": "text" + }, + "source": [ + "## Dimensionality Reduction\n", + "\n", + "With feature selection, we were only able to capture 33% of the original variance, which isn't great. To capture more variation while still remaining in two variables, have to utilize some clever math.\n", + "\n", + "These clever mathematical techniques are known as feature creation, where we try to create new variables that helps us visualize higher dimensional data. There are many different techniques for dimensionality reduction all of which attempts to accomplish different things. Let's start off with the simplest and most popular one: **Principle Component Analysis**." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cZ52OQlGAm6Q", + "colab_type": "text" + }, + "source": [ + "### Principle Component Analysis (PCA)\n", + "\n", + "Simply put, Principle Component Analysis create new features that maximizes variation. What PCA is **not** doing is feature selection, rather it is creating an entirely new arbitrary feature that is a combination of all the features. \n", + "\n", + "PCA involves some simple linear algebra, but SciKit-Learn has a PCA implementation. Note that all dimensionality reduction algorithms in SciKit-Learn is operated very similarity to machine learning algorithms you learned in the previous chapters. Create the object and then run `fit()` or `fit_transform()`." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "FESc4NoI8-3v", + "colab_type": "code", + "outputId": "1d1b00d1-65f5-40dd-ca5f-93f9ba84ef7c", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 255 + } + }, + "source": [ + "from sklearn.decomposition import PCA\n", + "\n", + "# We want to be able to plot this on a 2-D scatter plot so we choose 2 Dimensions\n", + "dimension_we_want = 2\n", + "\n", + "pca = PCA(n_components=dimension_we_want)\n", + "X_2d = pca.fit_transform(X) \n", + "X_2d = pd.DataFrame(X_2d, columns=[\"PCA1\", \"PCA2\"])\n", + "\n", + "print(\"\\n% Variance Explained:\", pca.explained_variance_ratio_.sum(), end=\"\\n\\n\")\n", + "X_2d.head()" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "stream", + "text": [ + "\n", + "% Variance Explained: 0.6462543462926849\n", + "\n" + ], + "name": "stdout" + }, + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
PCA1PCA2
02.620822-1.437859
12.1664460.732196
22.359265-0.067934
31.518218-0.792417
41.5199200.451721
\n", + "
" + ], + "text/plain": [ + " PCA1 PCA2\n", + "0 2.620822 -1.437859\n", + "1 2.166446 0.732196\n", + "2 2.359265 -0.067934\n", + "3 1.518218 -0.792417\n", + "4 1.519920 0.451721" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 36 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Gj87JXmmBdNu", + "colab_type": "code", + "outputId": "4574186e-9f94-4b91-9cf9-ba4a68b9f866", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 296 + } + }, + "source": [ + "X_2d.plot.scatter(x=\"PCA1\", y=\"PCA2\")" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 37 + }, + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAATe0lEQVR4nO3dcWzcZ33H8c/nEtfxcLZGjpWucUMqUtCqYIKwOjZv0lrYKIgFtVk1OsHEYIuQhkQlpoSqbGOaJtGwMaSViUWUwaSKqpvbBdFWNFVgUAQFB7mmaQrqGKyOGA1eSmOwXaf33R93Xm3X8Tm5+93zu3veL8mS73eX+30vd/597nme3/P8HBECAOSnkroAAEAaBAAAZIoAAIBMEQAAkCkCAAAytTF1ARdi69atsXPnztRlAEBHOX78+E8iYnDl9o4KgJ07d2p8fDx1GQDQUWz/cLXtdAEBQKYIAADIFAEAAJkiAAAgUwQAAGSKAADQEtMz83rs6Wc1PTOfuhSsU0edBgqgnI5MnNLBsUn1VCpaqFZ1aN+w9u7ZnrosNEALAEBTpmfmdXBsUnMLVZ2dP6e5haoOjE3SEugABACApkydmVVPZfmhpKdS0dSZ2UQVYb0IAABNGdrSp4Vqddm2hWpVQ1v6ElWE9SIAADRloL9Xh/YNa1NPRZt7N2pTT0WH9g1roL83dWlogEFgAE3bu2e7Rndt1dSZWQ1t6ePg3yEIAAAtMdDfy4G/w9AFBACZIgAAIFPJAsD2JtvftP2Y7RO2/ypVLQCQo5RjAPOSrouIGds9kh6x/WBEfCNhTQCQjWQBEBEhaaZ+s6f+E6nqAYDcJB0DsL3B9oSkZyQdjYhHV3nMftvjtsdPnz7d/iIBoEslDYCIeCEi9kgaknSN7d2rPOZwRIxExMjg4EuuaQwAuEilOAsoIp6V9CVJ16euBQBykfIsoEHbl9Z/75P025KeTFUPAOQm5VlAvyzps7Y3qBZE90TEFxLWAwBZSXkW0KSk16baPwDkrhRjAACA9iMAACBTBAAAZIoAAIBMEQAAkCkCAAAyRQAAQKYIAADIFAEAAJkiAAAgUwQAAGSKAACATBEAAJApAgAAMkUAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMhUsgCwfYXtL9l+wvYJ2+9PVQsA5Ghjwn2fk/SBiPi27c2Sjts+GhFPJKwJALKRrAUQET+KiG/Xfz8r6aSk7anqAYDclGIMwPZOSa+V9Ogq9+23PW57/PTp0+0uDQC6VvIAsN0vaUzSLRHx3Mr7I+JwRIxExMjg4GD7CwSALpU0AGz3qHbwvysi7k1ZCwDkJuVZQJZ0p6STEfGxVHUAQK5StgBGJb1T0nW2J+o/b0lYDwBkJdlpoBHxiCSn2j8A5C75IDAAIA0CAAAyRQAAQKYIAADIFAEAAJkiAIAVpmfm9djTz2p6Zj51KUChUq4GCpTOkYlTOjg2qZ5KRQvVqg7tG9bePaxRiO5ECwCom56Z18GxSc0tVHV2/pzmFqo6MDZJSwBdiwAA6qbOzKqnsvxPoqdS0dSZ2UQVAcUiAIC6oS19WqhWl21bqFY1tKUvUUVAsQgAoG6gv1eH9g1rU09Fm3s3alNPRYf2DWugvzd1aUAhGAQGlti7Z7tGd23V1JlZDW3p4+CPrkYAACsM9Pdy4EcW6AICgEwRAACQKQIAQCGYUV1+jAEAaDlmVHcGWgB1fFsBWoMZ1Z2DFoD4tgK00uKM6jm9OKlucUY1Z1eVS/YtAL6tdB9ac2kxo7pzZBUAqx0YWP+luxyZOKXR24/pHZ96VKO3H9PnJ06lLik7zKjuHNl0AZ2vm4dvK91jaWtusfvhwNikRndt5eDTZsyo7gxZtADW6ubh20r3oDVXLgP9vXrNFZfyt1RiWbQAGg1K8W2lO9CaAy5M0haA7U/bfsb240XuZz0HBr6tdD5ac8CFSd0C+IykOyT9S5E7WTwwHFgxBsCBofvQmgPWL2kARMRXbO9sx744MOSD1TyB9UndAmjI9n5J+yVpx44dTT0XBwYAeFHpzwKKiMMRMRIRI4ODg6nLAdqOiW0oSulbAEDOWKYERSp9CwDIFcuUoGipTwP9nKSvS3qV7Snb70lZD1AmTGxD0VKfBXRzyv0DZcbENhSNLiCgpJjYVl7dMjDPIDBQYsxfKZ9uGpgnAICSY/5KeXTbirN0AQHAOnXbwDwBAADr9LJLNmj+3AvLtnXywDwBsIZuGejBi3hPcbGOTJzSW+94RJWKJUm9G9zxA/OMAZxHNw30oIb3FBdrad//orB1//t+Q7u2bU5YWXPW1QKw3bPKtq2tL6cYF/qtjxmY3Yf3FM1Yre+/d0NFP3v+hfP8i86wZgDYvtb2lKQf2X5oxdLNDxVZWKtczEXCu22gB7ynaE63Tspr1AI4JOlNEbFV0mFJR22/vn6fC62sBS72W1+3vtk54z3tTGUZs+nWSXmNxgAuiYgTkhQR/2b7pKR7bR+UFIVX16RG1wI+H64g1n14TztP2cZsunFSXqMAWLB9WUT8jyRFxAnbb5D0BUmvKLy6JjXzra8b3+zc8Z52jrJOuOq2SXmNuoA+KGnb0g0RMSXptyR9pKCaWqbZZhsXiu8+vKedgTGb9lizBRARD5/nrs2Snm99Oa3Ht758TM/M8z53CcZs2mPd8wBsD0q6SdLNki6XdF9RRbVatzXb8FJl6y9GcxizaY81A8D2Zkk3SvoDSa+UdK+kKyNiqA21AetS1v5iNIfWe/EatQCekfRNSR+S9EhEhO0bii8LWL+LPdsL5UfrvViNBoFvldQr6R8l3Wq79Gf+ID/0FwMXZ80AiIiPR8TrJb2tvunfJV1u+6DtVxZeHbAO3TpJByiaIy5sPpft3aoNBP9+ROwqpKrzGBkZifHx8XbuEh2Es4CA1dk+HhEjK7c3GgTeJWlbRHxtcVtEPG77QUn/3Poy0clSH4DpLwYuTKNB4I+rNg6w0k8l/b2k3215RehInIYJdJ5Gg8DbIuI7KzfWt+0spCJ0HJZaBjpTowC4dI37OMUCkso5bb8sq0gCZdYoAMZt/8nKjbb/WNLxZndu+3rb37X9lO0PNvt8SKNsp2FezDUggBw1CoBbJP2R7S/b/rv6z39Ieo+k9zezY9sbJH1C0pslXS3pZttXN/OcSKNMp2HSHQWsX6PF4H4s6ddtXytpd33z/RFxrAX7vkbSUxHxfUmyfbdq8w2eaMFzo83KMm2fWcEoo9RnyJ1Po9NAN0l6r6Rdkr4j6c6IONeifW+X9PSS21OSfnWVGvZL2i9JO3bsaNGuUYQynIZZtu4ooMxnyDXqAvqspBHVDv5vlvS3hVe0QkQcjoiRiBgZHBxs9+7RYcrUHQWUvUuy0TyAqyPi1ZJk+07VFoZrlVOSrlhye6i+DWhKWbqjgLJ3STa8JOTiLxFxzm7pdeC/Jekq21eqduB/u2rLTgNNK0N3FFD2LslGXUCvsf1c/eespOHF320/18yO62MJ75P0RUknJd2zeAF6AOgGZe+SbHQW0IYidx4RD0h6oMh9AEBKZe6SXPclIQEAF6esXZKNuoAAAF2KAACATBEAAJApAgAAMkUAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMgUAQAAmSIAACBTBABWNT0zr8eeflbTM/OpS0HG+BwWiyuC4SWOTJzSwbFJ9VQqWqhWdWjfsPbu2Z66LGSGz2HxaAFgmemZeR0cm9TcQlVn589pbqGqA2OTfANDW/E5bA8CAMtMnZlVT2X5x6KnUtHUmdlEFSFHfA7bgwDAMkNb+rRQrS7btlCtamhLX6KKOgt91q3B57A9CAAsM9Dfq0P7hrWpp6LNvRu1qaeiQ/uGNdDfm7q00jsycUqjtx/TOz71qEZvP6bPT5xKXVLH4nPYHo6I9u/UvknShyX9iqRrImJ8Pf9uZGQkxsfX9VA0aXpmXlNnZjW0pY8/unWYnpnX6O3HNLfw4rfWTT0Vfe3gdfz/NYHPYWvYPh4RIyu3pzoL6HFJN0r6p0T7RwMD/b0d8QdXlgPEYp/1nF4MgMU+6074fyyrTvkcdqokARARJyXJdordo0uU6TRB+qzRiRgDQEdq1WmCrRq0pc8anaiwFoDthyVdtspdt0XEkQt4nv2S9kvSjh07WlQdOl0rulxa3YLYu2e7RndtLUWXFLAehQVARLyxRc9zWNJhqTYI3IrnROdrtstlaQtiMUQOjE1qdNfWpg7c9Fmjk9AFhI7UbJcLE42ARIPAtm+Q9A+SBiXdb3siIt6UohZ0rma6XBi0BRK1ACLivogYiojeiNjGwR8Xa6C/V6+54tIL7nZh0BZgNVBkjEFb5I4AQNYYtEXOGAQGgEwRAACQKQIAADJFAABApggAACi5oi40xFlAAFBiRa56SwsAAEqqVaveng8BAAAlVfSaVQQAAJRU0WtWEQAAUFJFr1nFIDAAlFiRa1YRAABQckWtWUUXELJU1HnVQCehBYDsFHledSebnplnaezMEADISlHXAu50hGKe6AJCVrgW8EsVPdkI5UUAICtcC/ilCMV8EQDICtcCfilCMV+MASA7XAt4ucVQPLBiDCD3/5ccEADIEtcCXo5QzBMBAEASoZgjxgAAIFNJAsD2R20/aXvS9n22L01RBwDkLFUL4Kik3RExLOl7km5NVAcAZCtJAETEQxFxrn7zG5KGUtQBADkrwxjAuyU9eL47be+3PW57/PTp020sqxxYtAxAUQo7C8j2w5IuW+Wu2yLiSP0xt0k6J+mu8z1PRByWdFiSRkZGooBSS4v1WQAUqbAAiIg3rnW/7XdJequkN0REVgf29WDRMgBFS3UW0PWSDkjaGxE/T1FD2bE+C4CipRoDuEPSZklHbU/Y/mSiOkqL9VkAFC3JTOCI2JViv52E9VkAFI2lIEqM9VkAFIkAKDnWZwFQlDLMAwAAJEAAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMgUAQAAmSIAACBTBACyMD0zr8eeflbTM/OpSwFKg+Wg0fWOTJzSwRUX1tm7Z3vqsoDkaAGgq03PzOvg2KTmFqo6O39OcwtVHRibpCUAiABAl5s6M6ueyvKPeU+loqkzs4kqAsqDAEBXG9rSp4Vqddm2hWpVQ1v6ElUElAcBgK420N+rQ/uGtamnos29G7Wpp6JD+4a5zCYgBoGRgb17tmt011ZNnZnV0JY+Dv5AHQGALAz093LgB1agCwgAMpUkAGz/te1J2xO2H7J9eYo6ACBnqVoAH42I4YjYI+kLkv4iUR0AkK0kARARzy25+TJJkaIOAMhZskFg238j6Q8l/VTStWs8br+k/ZK0Y8eO9hQHABlwRDFfvm0/LOmyVe66LSKOLHncrZI2RcRfruM5T0v6YeuqLNRWST9JXUQb8Xq7W06vtxtf68sjYnDlxsICYL1s75D0QETsTlpIi9kej4iR1HW0C6+3u+X0enN6ranOArpqyc23SXoyRR0AkLNUYwAfsf0qSVXVunTem6gOAMhWkgCIiH0p9ttmh1MX0Ga83u6W0+vN5rUmHwMAAKTBUhAAkCkCAAAyRQAUyPZHbT9ZX/foPtuXpq6pSLZvsn3CdtV2V55GZ/t629+1/ZTtD6aup2i2P237GduPp66laLavsP0l20/UP8fvT11T0QiAYh2VtDsihiV9T9Ktiesp2uOSbpT0ldSFFMH2BkmfkPRmSVdLutn21WmrKtxnJF2fuog2OSfpAxFxtaTXS/rTbn9/CYACRcRDEXGufvMbkoZS1lO0iDgZEd9NXUeBrpH0VER8PyKel3S3avNYulZEfEXS/6auox0i4kcR8e3672clnZS0PW1VxSIA2ufdkh5MXQSasl3S00tuT6nLDxC5sr1T0mslPZq2kmJxRbAmrWfNI9u3qda8vKudtRVhvWs8AZ3Kdr+kMUm3rFi5uOsQAE2KiDeudb/td0l6q6Q3RBdMumj0ervcKUlXLLk9VN+GLmG7R7WD/10RcW/qeopGF1CBbF8v6YCkvRHx89T1oGnfknSV7SttXyLp7ZI+n7gmtIhtS7pT0smI+FjqetqBACjWHZI2Szpav/zlJ1MXVCTbN9iekvRrku63/cXUNbVSfUD/fZK+qNoA4T0RcSJtVcWy/TlJX5f0KttTtt+TuqYCjUp6p6Tr6n+vE7bfkrqoIrEUBABkihYAAGSKAACATBEAAJApAgAAMkUAAECmCABgFbZfqJ8G+Ljtf7X9C/Xtl9m+2/Z/2j5u+wHbr1zy726xPWf7l5ZsG6ivMjlj+44UrwdYDQEArG42IvZExG5Jz0t6b32i0H2SvhwRr4iI16m2wuu2Jf/uZtUmjN24ZNucpD+X9GftKR1YHwIAaOyrknZJulbSQkT8/4S+iHgsIr4qSbZfIalf0odUC4LFx/wsIh5RLQiA0iAAgDXY3qja+v/fkbRb0vE1Hv521ZaI/qpqM2e3rfFYIDkCAFhdn+0JSeOS/lu1NWIauVnS3RFRVW1BsZsKrA9oGquBAqubjYg9SzfYPiHp91Z7sO1XS7pKtXWfJOkSSf+l2npQQCnRAgDW75ikXtv7FzfYHrb9m6p9+/9wROys/1wu6XLbL09VLNAIi8EBq7A9ExH9q2y/XNLHJb1OtUHdH0i6RbUVQt8SEU8ueezHJP04Im63/QNJv6hay+BZSb8TEU8U/TqAtRAAAJApuoAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMjU/wFur9UWE4oSkQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [], + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cT7Rv9lEDSji", + "colab_type": "text" + }, + "source": [ + "As we can see from above, the two new features (known as Principle Components) are not like any of our input features. Additionally, these two new components explains 64.6% of all the original variation. Let's see how well this performs in explaining our dataset." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "eBHVGhd-CJF8", + "colab_type": "code", + "colab": {} + }, + "source": [ + "reg = LinearRegression()\n", + "\n", + "reg.fit(X, y)\n", + "print(\"All Features:\\t\\t\", reg.score(X, y))\n", + "reg.fit(X_2d, y)\n", + "print(\"With PCA Features:\\t\", reg.score(X_2d, y), end=\"\\n\\n\")" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bsoN_a8XI4p2", + "colab_type": "text" + }, + "source": [ + "As expected, with a higher explained variance ratio, we perform better in predicting the quality of the wine. With Dimensionality Reduction, we were able to capture most of the variation in the dataset while still being able to view it in two dimensions. In the most basic of terms, PCA creates a variable projected along the axis of maximum variable.\n", + "\n", + "![](https://raw.githubusercontent.com/bfkwong/data/master/IMG_0187%202.jpg)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vwDiZRX3JYLN", + "colab_type": "text" + }, + "source": [ + "### Optional: Linear Algebra Behind PCA\n", + "\n", + "Principle Component Analysis chooses principle components along te **axis of greatest variance**. In Linear Algebra terms, the axis of greatest variance is the **Eigenvector with the largest Eigenvalue of the covariance matrix ($\\Sigma$)**\n", + "\n", + "The following are the steps in order to do PCA manually. \n", + "\n", + "1. Given data matrix $M$, generate the covariance matrix of $M$ denoted as $\\Sigma$\n", + "2. We then compute the Eigenvector and Eigenvalue of covariance matrix $\\Sigma$\n", + "3. Project the features to the Eigenvector with the largest Eigenvalue using the dot product (cross product for more than 1 dimensions)" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "J26dEAoAD0q7", + "colab_type": "code", + "colab": {} + }, + "source": [ + "import numpy as np\n", + "\n", + "# Step 1: Calculate covariance matrix \n", + "cov_mtrx = np.cov(X.T)\n", + "\n", + "# Step 2: Calculate Eigenvector and Eigenvalue\n", + "W,v = np.linalg.eig(cov_mtrx)\n", + "\n", + "# Step 3: Find the largest Eigenvalue and project our data onto the corresponding Eigenvector\n", + "idx_largest_eigenval = np.argmax(W)\n", + "eigenvec = v[:,idx_largest_eigenval]\n", + "\n", + "total = []\n", + "for row in X.index: \n", + " total.append(np.dot(X.loc[row], eigenvec))\n", + "\n", + "pd.DataFrame(pd.Series(total), columns=[\"PCA1\"]).head()" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_G7NWh_1nf_k", + "colab_type": "text" + }, + "source": [ + "One interesting thing you may see here is that the eigenvalue corresponds to the variance explained by its corresponding eigenvalue. The eigenvalue of PCA1 is 2.26 and the variance of PCA1 is also 2.26" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "9QCALn43iTYd", + "colab_type": "code", + "colab": {} + }, + "source": [ + "idx_largest_eigenval = np.argmax(W)\n", + "variance = pd.Series(total).var()\n", + "\n", + "print(\"Largest Eigenvalue:\\t\", W[idx_largest_eigenval])\n", + "print(\"Variance of Eigenvector:\", variance)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bmLOaYzvEeG7", + "colab_type": "text" + }, + "source": [ + "### Multidimensional Scaling (MDS)\n", + "\n", + "We now pay a visit to our good friend, the Euclidean distance. One incredibly useful aspect of Euclidean distance is that it works in higher dimensions. The formula $$\\sqrt{x_1^2 + x_2^2}$$ is for two dimensional Euclidean distance, but to move it to a third dimension, it is as easy as adding a $x^3$ variable. One thing to recognize is that Euclidean distance in all dimension is still a number. \n", + "\n", + "Why am I rambling on about something you learned in middle school? Well the realization that Euclidean distance is scalar in all dimensions means that we can preserve the variance of n-th dimensional data in two dimensions as long as we try to ensure that the Euclidean distances in the n-th dimensional is proportion to the Euclidean distance in 2 dimensions. That was a lot to take in, the following image explains the concept.\n", + "\n", + "![](https://raw.githubusercontent.com/bfkwong/data/master/IMG_0188.jpg)\n", + "\n", + "Notice how when we reduced our dimensions from 2 to 1, the distances between points A, B, and C remained the same. Meaning that x, y, and z remained the same between the two dimensions. While distances may not always be preserved perfectly between dimensions, MDS attempts to preserve it as well as possible. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "L4AjVFrwn78I", + "colab_type": "code", + "colab": {} + }, + "source": [ + "from sklearn.manifold import MDS\n", + "# We want to be able to plot this on a 2-D scatter plot so we choose 2 Dimensions\n", + "dimension_we_want = 2\n", + "\n", + "mds = MDS(n_components=dimension_we_want)\n", + "X_2d = mds.fit_transform(X) \n", + "X_2d = pd.DataFrame(X_2d, columns=[\"Dimension 1\", \"Dimension 2\"])\n", + "\n", + "display(X_2d.head())\n", + "print(\"Percentage variance explained:\", X_2d.var().sum()/X.var().sum())\n", + "X_2d.plot.scatter(x=\"Dimension 1\", y=\"Dimension 2\")" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mlIA-o_cEnSD", + "colab_type": "text" + }, + "source": [ + "By using MDS, we were actually able to preserve over 95% of the variance from the original datasets. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "88kI9JxzFO6n", + "colab_type": "text" + }, + "source": [ + "### Linear vs Nonlinear Dimensionality Reduction \n", + "\n", + "Thus far, we have explored PCA (a linear reduction technique) and MDS (a nonlinear reduction technique). While there are a lot of differences between the two methods. The key differences could be boiled down to just the following statement: **linear dimensionality reduction technique only stretch and shift the data while nonlinear techniques make more drastic changes to the data**.\n", + "\n", + "This sometimes leads to nonlinear techniques being better at capturing variance but losing the overall shape of the data whereas linear techniques are better at keeping the general shape of the original data but loses more variance along the way. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7uprQWlUGC37", + "colab_type": "text" + }, + "source": [ + "# Exercises" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UnMJKgsTkDoZ", + "colab_type": "text" + }, + "source": [ + "1. Consider the Iris dataset (https://raw.githubusercontent.com/dlsun/pods/master/data/iris.csv). Drop the \"SepalWidth\" and \"PedalWidth\" columns and then apply PCA on \"SepalLength\" and \"PedalLength\" with `n_components = 2`. How many percent of the variance was PCA able to capture in this case? What happens when we use PCA to compress 2D data into 2D data?" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "HCXKtA8AlSQq", + "colab_type": "code", + "colab": {} + }, + "source": [ + "" + ], + "execution_count": 0, + "outputs": [] + } + ] +} \ No newline at end of file diff --git a/10-Textual-Data/10.3 Distance Metrics.ipynb b/10-Textual-Data/10.3 Distance Metrics.ipynb new file mode 100644 index 0000000..f5137d7 --- /dev/null +++ b/10-Textual-Data/10.3 Distance Metrics.ipynb @@ -0,0 +1,660 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "name": "10.3 Distance Metrics.ipynb", + "provenance": [], + "collapsed_sections": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "PBgTRCHclclP", + "colab_type": "text" + }, + "source": [ + "# 10.3 Distance Metrics\n", + "\n", + "In the last section, we used Cosine Similarity to measure how similar two documents are. However, there are many other measures of similarity that exists. It will be up to us to decide which metric is best for our application. In this section, we will explore these other methods for measuring text similarity. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9BO29LPWh9Bn", + "colab_type": "text" + }, + "source": [ + "## Hamming Distance\n", + "\n", + "The most simple and intuitive method for measuring string similarity is the hamming distance. Given two strings of equal lengths, it measures the number of symbols that are different at each corresponding index. Take the example of the strings `\"BRYAN\"` and `\"BRIAN\"`. Here, we can see that the Hamming difference between the two strings is 1 because they are different at one location (third letter). " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "vhw83KbRkRC9", + "colab_type": "code", + "outputId": "a7706f2a-5a7c-4921-c2c2-446ab4789101", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + } + }, + "source": [ + "def hamming_distance(str1, str2): \n", + " if len(str1) != len(str2): \n", + " return None \n", + "\n", + " differences = 0\n", + " for x in range(len(str1)): \n", + " if str1[x] != str2[x]: \n", + " differences += 1 \n", + " \n", + " return differences\n", + "\n", + "hamming_distance(\"Bryan\", \"Brian\") " + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "1" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 41 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "N3B-DJ3HlO0t", + "colab_type": "text" + }, + "source": [ + "In theory, we can modify the Hamming Distance to accept two strings of different lengths, and whatever is left over is counted as a difference. The following is a simple implementation for this modified Hamming Distance" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "I7S6HPYxlKy3", + "colab_type": "code", + "outputId": "9bfdfa7b-15a9-49f9-f119-8e3a6bf596a3", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "def modified_hamming_distance(str1, str2): \n", + " differences = 0\n", + "\n", + " if len(str1) != len(str2):\n", + " differences += abs(len(str1) - len(str2))\n", + "\n", + " for x in range(len(str1)): \n", + " if str1[x] != str2[x]: \n", + " differences += 1 \n", + " \n", + " return differences\n", + "\n", + "modified_hamming_distance(\"Bryan\", \"Brian!!!\") " + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "4" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 23 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BethmiSBoTJj", + "colab_type": "text" + }, + "source": [ + "## Edit Distance\n", + "\n", + "A more applicable string metric is the edit distance. Generally, the edit distance is the smallest number of operations needed to turn one string into another. These operations may include insertions, deletions, substitutions, transposition, etc. Edit distances have different names based on their set of operations. For this chapter, we will focus on the Damerau-Levenshtein distance.\n", + "\n", + "### Damerau-Levenshtein Distance\n", + "\n", + "The Damerau-Levenshtein distance has four operations: deletion, insertion, substitution, and transposition. Here's what these operations mean:\n", + "\n", + "* Deletion: **s**cat --> cat
\n", + "\"scat\" and \"cat\" have an edit distance of 1 because we can delete the s in \"scat\" to turn it into \"cat\". \n", + "\n", + "* Insertion: at --> **m**at
\n", + "\"at\" and \"mat\" have an edit distance of 1 because we can insert an m into \"at\" to turn it into \"mat\". \n", + "\n", + "* Substitution: **m**at --> **c**at
\n", + "\"mat\" and \"cat\" have an edit distance of 1 because we can substitute the m in \"mat\" for a c to turn \"mat\" into a \"cat\".\n", + "\n", + "* Transposition: e**at** --> e**ta**
\n", + "\"eat\" and \"eta\" have an edit distance of 1 because we can transpose the successive characters of 'a' and 't' to turn \"eat\" into \"eta\". \n", + "\n", + "By measuring how many steps it takes to turn one string into another, it tells us how similar the two strings are. While the concept may be simple, it is infinitely applicable. One of the more interesting applications of the Damerau-Levenshtein distance is lingustic comparison. By computing the DL distance on two strings of different language but of same meaning, we can get a general idea of how similar the two languages are. \n", + "\n", + "Damerau-Levenshtein distance is conceptually simple but it is rather tedious to implement efficiently as it requires Dynamic Programming. NLTK provides us with an implementation of Damerau-Levenshtein distance with the `edit_distance()` function. Note that `edit_distance()` only provides the Levenshtein distance (without transposition). To make `edit_distance()` compute Damerau-Levenshtein distance we specify the parameter `transpositions = True`.\n", + "\n", + "\n", + "Below is an example of DL distance in practice using translation of an excerpt from Winston Churchill's famous \"We shall fight on the beaches\" speech into five different languages.\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "IKge7Vi6oSR5", + "colab_type": "code", + "outputId": "31720555-6f7e-4da6-987a-e1365eaffd38", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 266 + } + }, + "source": [ + "from nltk.metrics import edit_distance\n", + "import pandas as pd\n", + "\n", + "df_speech = pd.read_csv(\"https://raw.githubusercontent.com/bfkwong/data/master/we_shall_never_surrender.csv\", index_col=\"language\")\n", + "\n", + "def getDLDistance(language): \n", + " return edit_distance(df_speech.loc[\"english\"][\"text\"], language, transpositions=True)\n", + "\n", + "df_speech[\"Distance to English\"] = df_speech[\"text\"].apply(getDLDistance)\n", + "df_speech" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
textDistance to English
language
englishWe shall go on to the end. We shall fight in F...0
spanishLlegaremos hasta el final, lucharemos en Franc...276
germanWir werden bis zum Ende weitermachen. Wir werd...334
portugueseIremos até ao fim. Lutaremos na França. Lutare...281
frenchNous irons jusqu'au bout, nous nous battrons e...326
italianndremo avanti fino alla fine. Combatteremo in ...292
\n", + "
" + ], + "text/plain": [ + " text Distance to English\n", + "language \n", + "english We shall go on to the end. We shall fight in F... 0\n", + "spanish Llegaremos hasta el final, lucharemos en Franc... 276\n", + "german Wir werden bis zum Ende weitermachen. Wir werd... 334\n", + "portuguese Iremos até ao fim. Lutaremos na França. Lutare... 281\n", + "french Nous irons jusqu'au bout, nous nous battrons e... 326\n", + "italian ndremo avanti fino alla fine. Combatteremo in ... 292" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 73 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wHvsnPmzfu-A", + "colab_type": "text" + }, + "source": [ + "When it comes to the vocabularies in this excerpt from Churchill's speech. English is most similar to Spanish, followed by Portuguese, Italian, German, and finally French. It is important that we don't extrapolate this conclusion to the entire language, as these results only shows that with the words and punctuations used in this speech." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LSKNeVzBooXL", + "colab_type": "text" + }, + "source": [ + "## Jaccard Distance\n", + "\n", + "This method of comparing string similarity is more based on statistics than pure comparison like Hamming Distance and Edit Distance. The formulation of this metric is as follows: \n", + "\n", + "$$\n", + "d = 1 - {|X \\cap Y| \\over |X \\cup Y|}\n", + "$$\n", + "\n", + "What this equation tells us is that given two strings, we want to find the number of characters they have in common and divide it by the number of characters in total. Then we subtract it by 1, so its is consistent with all other distance metric where the more dissimilar two strings are, the larger their distance metric. Consider the following toy example: " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Qh_exrbztvKp", + "colab_type": "code", + "outputId": "340e9db4-4e95-4494-d7a5-f31df7bf987d", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "documentX = \"Republic of Korea\"\n", + "documentY = \"The Republic of Albania\"\n", + "\n", + "setX = set(documentX)\n", + "setY = set(documentY)\n", + "\n", + "intersection = setX.intersection(setY)\n", + "union = setX.union(setY)\n", + "\n", + "jaccard_dist = 1 - (len(intersection)/len(union))\n", + "jaccard_dist" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0.33333333333333337" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 94 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yQG8k-bptvfA", + "colab_type": "text" + }, + "source": [ + "Of course, this can be applied to words of a string as well instead of individual characters. Consider the following code: " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "KPR7puJQuslG", + "colab_type": "code", + "outputId": "b1a4d95f-0530-4e8f-a742-8ab394c73995", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "import pandas as pd\n", + "import requests\n", + "import re\n", + "\n", + "fed_1 = requests.get(\"http://dlsun.github.io/pods/data/federalist/1.txt\").text.lower()\n", + "fed_2 = requests.get(\"http://dlsun.github.io/pods/data/federalist/2.txt\").text.lower()\n", + "\n", + "fed_1 = set([x for x in re.split(\"\\n| |,|\\.|\\(|\\)\", fed_1) if x != '' and x != '\"'])\n", + "fed_2 = set([x for x in re.split(\"\\n| |,|\\.|\\(|\\)\", fed_2) if x != '' and x != '\"'])\n", + "\n", + "intersect = fed_1.intersection(fed_2)\n", + "union = fed_1.union(fed_2)\n", + "\n", + "jaccard_dist = 1 - (len(intersect)/len(union))\n", + "jaccard_dist" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0.7766895200783546" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 118 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6OfWaiK40pp-", + "colab_type": "text" + }, + "source": [ + "Fortunately for us, NLTK has a Jaccard Distance implementation for us to use so that we don't have to write all this code. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ZT_pa4cj03Xb", + "colab_type": "code", + "outputId": "0bab68e3-59f1-4a22-fa76-225dac2daf39", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "jaccard_distance(fed_1, fed_2)" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0.7766895200783546" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 120 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eiUHy1wPFKM0", + "colab_type": "text" + }, + "source": [ + "On the surface, Jaccard distance seems a bit generic, but that is what makes this such a popular metric. Its only requirement is that the inputs are two sets which means that we can compare basically anything. One additional benefits of Jaccard distance is that duplicates does not affect the results since Jaccard distance operates on sets. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YALHZEiVtxO3", + "colab_type": "text" + }, + "source": [ + "## Building Similarity Matrix\n", + "\n", + "Just like in our previous section, we want to build a similarity matrix that tells us which document is most similar to which other. To create a similarity matrix $S$, we want to ensure that $S_{ij}$ gives us the similarity between string i and string j. Note that in all the formulations we have given above, 0 represent identical strings, this can of course be inverted to fit your needs. \n", + "\n", + "The below code generates the a similarity matrix for the languages used for the translations of \"We shall fight on the beaches\" speech\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "oh6l7PK5frn8", + "colab_type": "code", + "outputId": "d64698fa-a6bb-4536-dac1-5009489c7f01", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 266 + } + }, + "source": [ + "import numpy as np\n", + "\n", + "mtrx = np.zeros((df_speech.shape[0], df_speech.shape[0]))\n", + "for i in range(df_speech.shape[0]):\n", + " for j in range(i + 1, df_speech.shape[0]):\n", + " d = edit_distance(df_speech.iloc[i][\"text\"], df_speech.iloc[j][\"text\"])\n", + " mtrx[i][j] = d\n", + " mtrx[j][i] = d\n", + "\n", + "mtrx = pd.DataFrame(mtrx, index=df_speech.index, columns=df_speech.index)\n", + "mtrx" + ], + "execution_count": 0, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
languageenglishspanishgermanportuguesefrenchitalian
language
english0.0276.0334.0281.0328.0293.0
spanish276.00.0338.0130.0280.0220.0
german334.0338.00.0344.0368.0345.0
portuguese281.0130.0344.00.0280.0190.0
french328.0280.0368.0280.00.0288.0
italian293.0220.0345.0190.0288.00.0
\n", + "
" + ], + "text/plain": [ + "language english spanish german portuguese french italian\n", + "language \n", + "english 0.0 276.0 334.0 281.0 328.0 293.0\n", + "spanish 276.0 0.0 338.0 130.0 280.0 220.0\n", + "german 334.0 338.0 0.0 344.0 368.0 345.0\n", + "portuguese 281.0 130.0 344.0 0.0 280.0 190.0\n", + "french 328.0 280.0 368.0 280.0 0.0 288.0\n", + "italian 293.0 220.0 345.0 190.0 288.0 0.0" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 87 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lP0zNA9Svyb4", + "colab_type": "text" + }, + "source": [ + "This similarity matrix tells us how similar languages are to each other. It makes sense for this matrix to suggest that Spanish is more similar to Portuguese and Italian than English, French, or German. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lUmS6CXZwtdT", + "colab_type": "text" + }, + "source": [ + "# Exercises " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Q3K2dw5oCd2Z", + "colab_type": "text" + }, + "source": [ + "1. Autocorrects determine if a word is correct by comparing it to an existing list of words. Download a dictionary of words from (https://raw.githubusercontent.com/dwyl/english-words/master/words_dictionary.json) and write a function that takes in a string and determine if there are any misspelling using the Hamming Distance." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "p3tY5LkR4qVY", + "colab_type": "text" + }, + "source": [ + "2. Damerau-Levenshtein distance is often used to measure how similar two RNA/DNA sequences are. Download the data from (link) and determine which animal HIV came from.\n", + "\n", + " Note: It may take a while to calculate Damerau-Levenshtein distance for long sequences.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bDt8YbtoHhlP", + "colab_type": "text" + }, + "source": [ + "3. Use the Jaccard distance to create a similarity matrix for translations of Churchill's speeches. Can you see any advantages or disadvantages of using the Jaccard distance in this context?" + ] + } + ] +} \ No newline at end of file diff --git a/10-Textual-Data/10.4 Visual Analysis of Text.ipynb b/10-Textual-Data/10.4 Visual Analysis of Text.ipynb new file mode 100644 index 0000000..eadfd70 --- /dev/null +++ b/10-Textual-Data/10.4 Visual Analysis of Text.ipynb @@ -0,0 +1,505 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "name": "10.4 Visual Analysis of Text.ipynb", + "provenance": [], + "collapsed_sections": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "Lv8WJDNEe4og", + "colab_type": "text" + }, + "source": [ + "# 10.4 Visual Analysis of Texts\n", + "\n", + "An important part of Exploratory Data Analysis if data visualization and it is no different for textual data. Visualizing textual data can help us see patterns that our brains could've otherwise not picked up. This section will teach us ways to look at text data. The dataset for this section will be the federalist papers which is processed below:\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "A-1uNmVtxQaT", + "colab_type": "code", + "colab": {} + }, + "source": [ + "import pandas as pd\n", + "import requests\n", + "\n", + "federalist_dir = \"http://dlsun.github.io/pods/data/federalist/\"\n", + "df_author = pd.read_csv(\"http://dlsun.github.io/pods/data/federalist/authorship.csv\")\n", + "\n", + "df_author = df_author.set_index(\"Paper\")\n", + "\n", + "papers = []\n", + "for paper in df_author.index:\n", + " response = requests.get(federalist_dir + str(paper) + \".txt\", \"r\")\n", + " papers.append(response.text)\n", + "\n", + "df_author[\"Text\"] = papers" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GXgnmrm-oPXD", + "colab_type": "text" + }, + "source": [ + "## N-Gram Frequency\n", + "\n", + "One way to visualize text data is by looking at how often an N-Gram occurs. The simplest way to observe this is by looking at a Unigram (basically its term frequency). This allows us to see how often a specific word is used. In the example below, let us examine how often the authors of the federalist paper mentions the 3 branches of government. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "_yzmXUqPkolH", + "colab_type": "code", + "outputId": "c9abdcc2-281e-4727-e9ff-56b3a4b7ea3e", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 85 + } + }, + "source": [ + "from sklearn.feature_extraction.text import CountVectorizer\n", + "from sklearn.preprocessing import StandardScaler\n", + "\n", + "# Create a CountVectorizer like in 10.2\n", + "vec = CountVectorizer()\n", + "vec.fit(df_author[\"Text\"]) \n", + "tf_sparse = vec.transform(df_author[\"Text\"]).todense()\n", + "\n", + "# Turn the sparse matrix into a Pandas DataFrame \n", + "df_unigram = pd.DataFrame(tf_sparse, columns=vec.get_feature_names(), index=df_author.index)\n", + "\n", + "# Get the three terms we want to examine\n", + "df_ngram = df_unigram[[\"executive\", \"legislature\", \"judiciary\"]]\n", + "df_ngram.sum()" + ], + "execution_count": 3, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "executive 246\n", + "legislature 190\n", + "judiciary 91\n", + "dtype: int64" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 3 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-v0j32ya10N0", + "colab_type": "text" + }, + "source": [ + "We can see here the authors of the federalist papers were particularly interested in discussing the executive branch of government. This aligns with consensus understanding of the federalist papers where the three authors were particularly concerned about the overreaching executive. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "4EaKhuJH12FV", + "colab_type": "code", + "outputId": "5f09fe54-3562-4867-e6e0-f61fe85c4179", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 296 + } + }, + "source": [ + "df_ngram.plot.line()" + ], + "execution_count": 4, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 4 + }, + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEGCAYAAAB8Ys7jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOydd5xU1fn/32fK7mxfFpbelt6bgIolStFYQGMiaiwYe4qJMSZqYkmiv8RE00z8xpBoxGgMCYogRkVBFMWIFFF6XWBhge19dmfuPb8/zp3Zzs4MbMF93q/XvGbn3jszZ2d3Pve5n+c5z1FaawRBEIRTD1d7D0AQBEGIDRFwQRCEUxQRcEEQhFMUEXBBEIRTFBFwQRCEUxRPW75Zt27d9MCBA9vyLQVBEE551q9fn6+1zmy4vU0FfODAgaxbt64t31IQBOGURym1v6ntYqEIgiCcooiAC4IgnKKIgAuCIJyitKkHLgjCqU0gECAnJwe/39/eQ/lC4vP56Nu3L16vN6LjRcAFQYiYnJwcUlJSGDhwIEqp9h7OFwqtNQUFBeTk5JCVlRXRcyKyUJRS6UqpRUqp7UqpbUqpM5VSGUqpt5VSu5z7Lic0ekEQOjx+v5+uXbuKeLcCSim6du0a1dVNpB74H4A3tdYjgPHANuA+YIXWeiiwwnksCMIXHBHv1iPaz7ZFAVdKpQHnAs8AaK1rtNbFwGXAAuewBcDlUb2zIHwBeHPzEfLLq9t7GEInJZIIPAvIA/6ulNqolPqbUioJ6KG1znWOOQL0aOrJSqnblFLrlFLr8vLyTs6oBaED4A9YfPPF9fxnXU57D0VoRVatWsWaNWvCj59++mmef/75dhxRLZEkMT3AJOBOrfXHSqk/0MAu0VprpVSTK0NorecD8wEmT54sq0cIXxhqLButoSpgtfdQhFZk1apVJCcnM23aNADuuOOOdh5RLZFE4DlAjtb6Y+fxIoygH1VK9QJw7o+1zhAFoWMStEw8UhO023kknYsXXniBqVOnMmHCBG6//XY+/vhjxo0bh9/vp6KigtGjR7N582YqKiq46aabmDp1KhMnTmTJkiUAWJbFPffcw5gxYxg3bhx//OMfAdPqIz8/H4B169Zx3nnnkZ2dzdNPP83vfvc7JkyYwOrVq/npT3/KE088wfbt25k6dWp4XNnZ2YwdOxaA9evX86UvfYnTTjuNCy+8kNzcXFqDFiNwrfURpdRBpdRwrfUOYAaw1bnNAx5z7pe0yggFoYMStI1wB6zOKeA/e20LWw+XntTXHNU7lYdnj252/7Zt21i4cCEffvghXq+Xb33rW+zYsYM5c+bwwAMPUFVVxXXXXceYMWP48Y9/zPTp03n22WcpLi5m6tSpzJw5k+eff57s7Gw+/fRTPB4PhYWFzb7fwIEDueOOO0hOTuaee+4BYMWKFQCMGDGCmpoa9u3bR1ZWFgsXLuSqq64iEAhw5513smTJEjIzM1m4cCE/+clPePbZZ0/qZwWR14HfCbyolIoD9gLfwETv/1ZK3QzsB+ae9NEJQgdGIvC2Z8WKFaxfv54pU6YAUFVVRffu3XnooYeYMmUKPp+PJ598EoDly5ezdOlSnnjiCcCUQB44cIB33nmHO+64A4/HyF9GRkbM45k7dy4LFy7kvvvuY+HChSxcuJAdO3awefNmZs2aBZiIv1evXifyazdLRAKutf4UmNzErhkndziCcOrQ2QX8eJFya6G1Zt68efzyl7+stz03N5fy8nICgQB+v5+kpCS01rz88ssMHz48otf2eDzYzlVVpLXYV111FVdeeSVXXHEFSimGDh3K559/zujRo/noo4+i++ViQHqhCEKMBJwve00ntVDagxkzZrBo0SKOHTMpt8LCQvbv38/tt9/OI488wrXXXsu9994LwIUXXsgf//hHtDYn2o0bNwIwa9Ys/vKXvxAMBsOvAcYuWb9+PQAvv/xy+D1TUlIoKytrcjyDBw/G7XbzyCOPcNVVVwEwfPhw8vLywgIeCATYsmXLSf0cQoiAC0KMdPYIvD0YNWoUjz76KBdccAHjxo1j1qxZLFiwAK/Xy9e//nXuu+8+PvnkE1auXMmDDz5IIBBg3LhxjB49mgcffBCAW265hf79+zNu3DjGjx/PP//5TwAefvhhvve97zF58mTcbnf4PWfPns3ixYvDScyGXHXVVbzwwgvMnWtc5Li4OBYtWsS9997L+PHjmTBhQr0yxJOJCp2d2oLJkydrWdBB+KKw5XAJlzz5ATNH9uBv85pyGL94bNu2jZEjR7b3ML7QNPUZK6XWa60b/ZNJBC4IMRKOwMVCEdoJEXBBiJFwGaFYKEI7IQIuCDESkAhcaGdEwAUhRixbkphC+yICLggxEpqBKQIutBci4IIQI5LEFNobEXBBiJFQElMi8LYlOTk55uc+9NBDvPPOO83uDzWqOh6vvvoqW7dujXkMJxMRcEGIkVASs1oE/JTh5z//OTNnzjyh14hFwEOzPk82IuCCECO1SUzpB95ePP7440yZMoVx48bx8MMPh7c/8sgjDB8+nLPPPptrrrkmHFXfeOONLFq0CID77ruPUaNGMW7cuHCnwbr89a9/ZcqUKYwfP56vfvWrVFZWsmbNGpYuXcoPf/hDJkyYwJ49ezjvvPMITVDMz89n4MCBADz33HPMmTOH6dOnM2PGjGbb254Isiq9IMRIKIkZisQ7HW/cB0c+P7mv2XMsXPRYRIcuX76cXbt2sXbtWrTWzJkzh/fff5+EhARefvllNm3aRCAQYNKkSZx22mn1nltQUMDixYvZvn07SimKi4sbvf4VV1zBrbfeCsADDzzAM888w5133smcOXO49NJL+drXvtbiGDds2MBnn31GRkZGs+1tk5KSIvp9m0IEXBBiJGhLErM9Wb58OcuXL2fixIkAlJeXs2vXLsrKyrjsssvw+Xz4fD5mz57d6LlpaWn4fD5uvvlmLr30Ui699NJGx2zevJkHHniA4uJiysvLufDCC6Me46xZs8Ltaptrb3sirQlEwAUhRoKOcFu2xrI1blcnW609wki5tdBac//993P77bfX2/773/++xed6PB7Wrl3LihUrWLRoEX/6059YuXJlvWNuvPFGXn31VcaPH89zzz3HqlWrmn2t5trQ1o2uo21vGwnigQtCjIQicJBKlPbgwgsv5Nlnn6W8vByAQ4cOcezYMc466yxee+01/H4/5eXlLFu2rNFzy8vLKSkp4eKLL+Z3v/sdmzZtanRMWVkZvXr1IhAI8OKLL4a3N2wvW7cNbchfb268TbW3PREkAheEGAla9QU8Ic59nKOFk80FF1zAtm3bOPPMMwFTXvjCCy8wZcoU5syZw7hx4+jRowdjx44lLS2t3nNDNovf70drzW9/+9tGr//II49w+umnk5mZyemnnx4W7auvvppbb72VJ598kkWLFnHPPfcwd+5c5s+fzyWXXNLseB988EHuuusuxo0bh23bZGVlNXlyiQZpJysIMfJ/q3bz6zd3ALD2JzPonuJr5xG1PqdKO9ny8nKSk5OprKzk3HPPZf78+UyaNKm9hxUR0bSTlQhcEGKkYQQudBxuu+02tm7dit/vZ968eaeMeEeLCLggxEhdD7zTlhJ2UEKr7HzRkSSmIMRIsE75oETgQnsgAi4IMSJVKEJ7IwIuCDESqBuBWzKdXmh7RMAFIUbqJjGloZXQHkQk4EqpbKXU50qpT5VS65xtGUqpt5VSu5z7Lq07VEHoWIiF0n5MmzYt4mNXrVoVniq/dOlSHnvs+DNIo3nt9iaaCPx8rfWEOrWI9wErtNZDgRXOY0HoNEgSs/1Ys2ZNTM+bM2cO9913fKmK5rW11uFp9O3BiVgolwELnJ8XAJef+HAE4dRBygjbj+Tk5HqRNcB3vvMdnnvuOQDefPNNRowYwaRJk3jllVfCxzz33HN85zvfAeDo0aN85StfYfz48YwfPz4s3KEFI8rLy5kxYwaTJk1i7Nix4fav2dnZDB8+nBtuuIExY8bwyCOPcNddd4Xf469//Svf//73W/X3DxFpHbgGliulNPAXrfV8oIfWOtfZfwTo0dQTlVK3AbcB9O/f/wSHKwgdh4Bl41Jg686ZxPzV2l+xvXD7SX3NERkjuHfqvSf0Gn6/n1tvvZWVK1cyZMgQrrrqqiaP++53v8uXvvQlFi9ejGVZ4Z4qIXw+H4sXLyY1NZX8/HzOOOMM5syZA8CuXbtYsGABZ5xxBuXl5YwfP57HH38cr9fL3//+d/7yl7+c0O8QKZFG4GdrrScBFwHfVkqdW3enNvPxmwxBtNbztdaTtdaTMzMzT2y0gtCBsGxNUpyJgcRC6Ths376drKwshg4dilKK6667rsnjVq5cyTe/+U0A3G53o34pWmt+/OMfM27cOGbOnMmhQ4c4evQoAAMGDOCMM84ATMQ+ffp0li1bxvbt2wkEAowdO7YVf8NaIorAtdaHnPtjSqnFwFTgqFKql9Y6VynVCzjWiuMUhA5HwNIkxLkpqw52SgE/0Uj5RKnbxhUat3I9UV588UXy8vJYv349Xq+XgQMHht+j4SIMt9xyC7/4xS8YMWIE3/jGN07qOI5HixG4UipJKZUS+hm4ANgMLAXmOYfNA058fSBBOIUI2jaJTgdCKSNsewYMGMDWrVuprq6muLiYFStWADBixAiys7PZs2cPAC+99FKTz58xYwZ//vOfAbAsi5KSknr7S0pK6N69O16vl3fffZf9+/c3O5bTTz+dgwcP8s9//pNrrrnmZPx6ERGJhdID+EAptQlYC7yutX4TeAyYpZTaBcx0HgtCpyFoaRJDFoqsytOmKKXo168fc+fOZcyYMcydOze8Mo/P5wu3dp00aRLdu3dv8jX+8Ic/8O677zJ27FhOO+20RgsVX3vttaxbt46xY8fy/PPPM2LEiOOOae7cuZx11ll06dJ2FdXSTlYQYuTq+R8RtDTr9hdx18yh3DVzWHsPqdXpCO1kCwoKmDRp0nEj4vbg0ksv5fvf/z4zZsw4odeJpp2szMQUhBgJWpp4rwu3S3VKD7w9OHz4MGeeeWaTq8i3F8XFxQwbNoyEhIQTFu9okXayghAjAVuT5HIR53bV64sitB69e/dm586d7T2MeqSnp7fbmCQCF4QYCVo2HpcizuPqVBF4W9qunY1oP1sRcEGIkaCl8bgdAe8kEbjP56OgoEBEvBXQWlNQUIDPF/nSfGKhCEKMBG0bj9tYKJ2ljLBv377k5OSQl5fX3kP5QuLz+ejbt2/Ex4uAC0KMBG2Nt5NZKF6vl6ysrPYehuAgFoogxIixUEwE3lkEXOhYiIALQowE6iYxO4kHLnQsRMAFIUYsuzaJKWWEQnsgAi4IMWIicLFQhPZDBFwQYiRoa7zuzpXEFDoWIuCCECNBS+N2uYjzdJ4yQqFjIQIuCDESsO3aCFw8cKEdEAEXhBiwbY3WiAcutCsi4IIQAwFnJRiPW4mAC+2GCLggxEDQWYXeK2WEQjsiAi4IMRAS8FASUyJwoT0QAReEGAg6FookMYX2RARcEGIgaJsI3BNe0EFj29JiVWhbRMAFIQZCnndoKj3IwsZC2yMCLggxEPLAPS5FvAi40E6IgAtCDIQtFLcLr9sRcElkCm2MCLggxEA4iemqY6GIgAttjAi4IMRA2EJxFnQApBZcaHMiFnCllFsptVEptcx5nKWU+lgptVsptVApFdd6wxSEjkU4iSkRuNCORBOBfw/YVufxr4Dfaa2HAEXAzSdzYILQkan1wGsFXDoSCm1NRAKulOoLXAL8zXmsgOnAIueQBcDlrTFAQeiI1FahuKSMUGg3Io3Afw/8CAj9h3YFirXWQedxDtCnqScqpW5TSq1TSq3Ly8s7ocEKQkeh7kzMeKlCEdqJFgVcKXUpcExrvT6WN9Baz9daT9ZaT87MzIzlJQShw1EviSkeuNBOeCI45ixgjlLqYsAHpAJ/ANKVUh4nCu8LHGq9YQpCx6JuElMhAi60Dy1G4Frr+7XWfbXWA4GrgZVa62uBd4GvOYfNA5a02igFoYNhNZHElDJCoa05kTrwe4G7lVK7MZ74MydnSILQ8QnYksQU2p9ILJQwWutVwCrn573A1JM/JEHo+ASt2iSmSylAygiFticqARcEwVC7oIMKz8QUD1xoa2QqvSDEQCBcRihVKEL7IQIuCDEQTmK6pB+40H6IgAtCDASaaGYlEbjQ1oiAC0IMBOvUgbtdCqWkjFBoe0TABSEG6jazUsokMiUCF9oaEXBBiIFQFYrXZb5CcR6XlBEKbY4IuCDEQNC2cSlwuUwNeLzHJUlMoc0RAReEGAhYGo+79usjForQHoiAC0IMBC0bjxN9g7FQRMCFtkYEXBBiIGhrEXCh3REBF4QYCNo23roWinjgQjsgAi4IMRC0NB53bQTudbukDlxoc0TABSEGApbG46qfxJQyQqGtEQEXhBgI2na9CFw8cKE9EAEXhBhomMSMFwEX2gERcEGIgaAlSUyh/REBF4QYaJjElIk8QnsgAi4IMRCwNW5XgwhcBFxoY0TABSEGLNvG22Aij5QRCm2NCLggxECgiTpwicCFtkYEXBBioKkkZrVE4EIbIwIuCDEQtDXuumWETgSutW7HUQmdDRFwQYiBYMOZmM7CxqG1MoXo+eOKXTzw6uftPYxTihYFXCnlU0qtVUptUkptUUr9zNmepZT6WCm1Wym1UCkV1/rDFYSOgWlmVT+JCbIy/YmwNruQNXsK2nsYpxSRRODVwHSt9XhgAvBlpdQZwK+A32mthwBFwM2tN0xB6FgEm1jQAWRl+hOhqsaiojrY3sM4pWhRwLWh3HnodW4amA4scrYvAC5vlREKQgckYDdc0MFttksEHjNVAYtyvwh4NETkgSul3EqpT4FjwNvAHqBYax36tHOAPs089zal1Dql1Lq8vLyTMWZBaHeMB96EhSIReMxU1VhU1FjYtuQRIiUiAddaW1rrCUBfYCowItI30FrP11pP1lpPzszMjHGYgtCxCNr1LZSQHy4tZWOnKmABUFEjUXikRFWForUuBt4FzgTSlVIeZ1df4NBJHpsgdFhMHXj9boQgEfiJUFnjCHi11c4jOXWIpAolUymV7vycAMwCtmGE/GvOYfOAJa01SEHoaDRXRihVKLETisDLJZEZMZ6WD6EXsEAp5cYI/r+11suUUluBfymlHgU2As+04jgFoUMRaLigg9skMSUCjw3L1uHPTgQ8cloUcK31Z8DEJrbvxfjhgtDpsJpYlR5EwGMlFH0DUkoYBTITUxCiRGvtNLNqykIR/zYWqmpqP7cyKSWMGBFwQYgSyylzq9dONjyRR0rgYqGugEsEHjki4IIQJUFHwN0ylf6kUc9CkTLCiBEBF4QoCc229LpkKv3JorKOaIuFEjki4IIQJSELxdNUBC4CHhOSxIwNEXBBiJJQy9gmk5hBSWLGQl0PXMoII0cEXBCiJGibKLvJMkLxwGOibgQuAh45IuCCECXBUATeZBWKCHgshCLw5HiPWChRIAIuCFESqkLxNtHMqkZW5ImJUATeLTlOIvAoEAEXhCgJOjZJ3SSmUoo4j6xMHyuhCDwzJZ5yaWYVMSLgghAl4SSmq/7XJ7SwsRA9oU6E3ZLjKfcH2nk0pw4i4IIQJU0lMQG8HpdMpY8Rf8DC53WR4vNIO9koEAEXhCgJNlEHDiaRKRF4bFTWWCR43SRJEjMqRMAFIUpCVShuF3zznW/yv9z/AYgHfgJUBSwS4zykxHsorwmitSSDI0EEXBCiJJTEDNiVfHDoA9bmrgUcAZc68JioqjEWSlK8B61rPXHh+IiAC0KUBBwLxcIPQEl1CRCyUCRyjIVQBJ4Ub5YokFLCyBABF4QoCUXgQV0NQHF1MSAR+IlQ5XjgKT4R8GgQAReEKAklMQPaicBrnAjc45JeKDFSGbBIiHOTFOcIuHQkjAgRcEGIklASM6jrWyjxksSMGb8TgSc7EbhUokSGCLggREmoDjwUgYcsFK9bLJRYqQwESYhzkyweeFSIgAtClIRmYgbsppKYIuCxUFVjGwtFBDwqRMAFIUosJwKvsasAqApWUW1VSx34CVBVEzQWSrxYKNEgAi4IURKKwKsdAQcThYuAx4bW2ikjrBXwMhHwiBABF4QoCU/ksarD24qri50yQqkDj5Yay8bW4PO68XlduF1KIvAIaVHAlVL9lFLvKqW2KqW2KKW+52zPUEq9rZTa5dx3af3hCkL7Eyoj9NuV4W0l1SWOBy5lhNESaiWb4HWjlCIpzi0NrSIkkgg8CPxAaz0KOAP4tlJqFHAfsEJrPRRY4TwWhC88IQGvtvzhbcXVxaaMUKpQoia0mENinBswq/LIyvSR0aKAa61ztdYbnJ/LgG1AH+AyYIFz2ALg8tYapCB0JEIWSrVVhVsZ0REPPHZCfU8SQgLua7kj4Ysf72f5liOtPraOTlQeuFJqIDAR+BjoobXOdXYdAXo085zblFLrlFLr8vLyTmCogtAxCCUx/cEqeiSaf/vi6mK8bhe2rhV4ITLqWigASfGeFssI57+/l5fWHmj1sXV0IhZwpVQy8DJwl9a6tO4+bXo/Npm90VrP11pP1lpPzszMPKHBCkJHIGjbuF2KymAl6b504t3x4QgcZGX6aAlZKAl1LJSWBLy4MiA2CxEKuFLKixHvF7XWrzibjyqlejn7ewHHWmeIgtCxCFoajyPgiZ5E0uLTTBWKrEwfE6EIvK4HfjwLxbY1pf4ApbL0WkRVKAp4Btimtf5tnV1LgXnOz/OAJSd/eILQ8QjaGq/bRVWwikRvIunx6RRXF4cjyFBEKURG6PPyRWihlPmDaA2lVRKBRxKBnwVcD0xXSn3q3C4GHgNmKaV2ATOdx4LwhSdo2XjcispAJQmeBNLj0ymtLiU9wQuYy3shchp64C1ZKCVV5vMtkwgcT0sHaK0/AFQzu2ec3OEIQscnYGs8LlfYQrHjbfYU7yEt0Qh4UWVNO4/w1KK2jNDIUchC0VpjDID6FFeZz7eixnJOpp13PmLn/c0FIUaClo3HpagKGAsl5IF3SYwDoEQi8KiobBiB+zzYunkrqu4VTmdPZIqAC0KUBG2N222aWCV6EhtZKEUi4FHhb1CF0lJHwpCFAiLgIuCCECVBS+N12wR1MOyBB3UQb5y5tBcLJToqa4K4XQqv29glKfHHX5WnuI6Ad/ZKlBY9cEEQ6hO0bdxuI9KJ3kSSvEkA+K0yfF4XxSLgUVFVY5Po9EGB2gi8uX4opXUFvKpzC7hE4IIQJQFLo1xGOBI9iaTFpQFmOn2XxDixUKKkKhDE59gnAEnx5uey6qY/x7onyFKxUARBiIagZeP2GHFJ8CaQ7ksHzHT69MQ4KSOMktCK9CFS4k0uobkIvLgyQKg4pbNbKCLgghAlQVvjcpte4KGZmBCKwL1ioURJaDGHEKEIvLnZmCVVAXql+gBJYoqAC0KUBC2Ny7FQQklMCEXgXkliRklljRWehQmEV6ZvblWe4qoAfbokAOKBi4ALQpQEbRtctUnM1LhUwETgYqFEj79BBN7SupilVQG6JMaREu8RC6W9ByAIpxomiVlroXhcHlK8Kc5kHi/FVQFMg04hEiobeOAJXjcudZwywsoA6YleUnyy8IMIuCBEiWVrlBOBJ3jMpXzd2ZiWrWVR3iioClj1qlCUUsdtaFVcVUN6YhypCV6xUNp7AIJwqhGwbLRyInBvIgDp8emU1BgLBaC4onMLSzRU1Vgk1onAofmGVv6AhT9gk5bgJdXnFQulvQcgCKcaQVuHPfBwBO5Lo8RfUmc6vSQyI6UqYIWn0Ydorid4KOJOSxALBUTABSFqgpaNphqvy4vXZQQ7Lc6xUJJEwKOlqqaxgDdnoZTUEfDUBInAZSq9IERJ0NZoVU2iJzG8LT0+PVyFAvUbLgnNY9ma6qBdL4kJzVsooT4oksQ0SAQuCFEStDS2qmkk4GWBMlJ85itVVCEReCT4A/WXUwvRnIUSKtFMT4gzHngnr/gRAReEKAnaNjZ+I+DvPwF5O8OzMXFVAtJSNlIa9gIPkRTvabKMsL6FYvqGV9S00RJ2m/4Fe95tm/eKELFQBCFKApbGppoElxdWPgKBStKyTgOgPFhKqs8j0+kjxN9gPcwQKb5mLBTnc01L9JLiM/mGMn8gPPmnVVn5KGQOh8Hnt/57RYhE4IIQJUHLxqKaROWITunh8HT6kuoSuiRJR8JIqaypv5xaiKR4NxU1ViN7pLTKNLJKifeQ6gh4myxurDWUH4XSw63/XlEgAi4IURK0tRHw0FKxpYdq+6H4nY6EksSMiKrwajz1pSgp3oNla/wBu9724qoAaQleXC5FaoIR/TapRPGXgFUDpYda/72iQARcEKIkaGss7SchFByWHq7tSFgjHQmjoXZF+voReEozy6oVVwbCtfZ1LZRWpyLP3PtLoLq89d8vQkTABSEKtNZYtiag/SRajriUHia9zqIO6QnSkTBSqgLmM2yqDhwaC3iJE4EDpDpdC9vEQik/WvtzWW7rv1+EiIALQhQELBN2W7qahJCABypJsgJ4lKd2UQeZSh8RVTXGImmqDhwadyQsrgqQ5tTapzpC3iYWSvmx2p87kI0iAi4IUWDZGnAi8EB1eLsqPUxqfGq4oVVZdZCAZTf/QgJgFjSGpuvAoXEEXlonAk8J9Q2PYjLP+v2FjH7oTXJLqgB49H+Pcvequ1t+Yj0B7ziJzBYFXCn1rFLqmFJqc51tGUqpt5VSu5z7Lq07TEHoGARsG1QQjU1CoAoaVKKYKhQjMDIbs2WaKyMMLerQsBa8uLIm7IHHe9zEe1xRdSRcv7+IihqLDfuLAfg8/3M2529u4VlAxbE6f+tTKwJ/Dvhyg233ASu01kOBFc5jQfjCE6zbC7y6HLqPMjucSpSS6pJwhCiJzJapLSNs2gOvqKkVcNvWlFSZXuAhUnzeqBY2zi4wE6225ZYCkF+ZT15VXsuzOcuPQnJ3SOwKJaeQgGut3wcKG2y+DFjg/LwAuPwkj0sQOiRBq85qPP5y6DkWlAsaWCjQOWdjllcH+c4/N4Qtipaoai4Cj29sj5TXBLE14RMkQGpCdKvyHKgj4K6I2yAAACAASURBVJZtke/PJ2gHKa4uPv4Ty/OMgKf2PrUslGboobUOpWKPAD2aO1ApdZtSap1Sal1eXl6MbycIHYNAncUcEv0l5gud3CNsodQT8E7YD+WT7EKWfZbLezsi+65X1VjEe1y4Xare9oykOOLcLg4WVoa3lVTWTqMPEeqHEin7CysAI+BF1UXY2uQpjlUeO97TTASe1B1S+3whBDyMNtcezV5/aK3na60na60nZ2ZmnujbCUK7YlkalNML3LbqRGWH6nQkdCyUTuiB7zlmaqT31xHe49FUL3AAr9vFkO7JbHWsDqjfByVENB0JA5bN4WI/yfEeDpf42VNYa4XkV+Uf/8kVeeZE7fytOwqxCvhRpVQvAOe+hdOXcEqgNXy+qENNVOhoBGy71gO3db3L6rT4NKqtahLijC3QGT3wPXkmwt1fUNHywQE/I4/9lwRP0zI0slcq23LLwo/DnQidKxwgqp7gh4qqsGzN+SO6A/Dp4QPhfceNwLU2VSjJmeZvXVUIgcgsotYmVgFfCsxzfp4HLDk5wxHalfyd8PLN8Pm/23skHRaTxHQsFG07l9V9ofQQmQndACiz8vG4VKf0wPfkORF4QQQR+PZlXHPo/zHFvbPJ3SN7pZBfXk1emTlhFleZz71uEtNYKJFF4NnOSeXLo3uat8+LMAKvKgI74ETgfc22DmKjRFJG+BLwETBcKZWjlLoZeAyYpZTaBcx0HgunOgV7zH3R/vYdRwcmUDeJaevay+qacoYm9gZgV9EuM5mnE0bgex0BP1BQ2XJlR1E2ACNVdpO7R/VKBWorRpqyUFJ9noin0h9wbJ3JA7vQLTmO7GKTxkvyJh0/Ag/VgCc5ETh0GAFvsQej1vqaZnbNOMljEdoCraGyAJK6Nd5XtM/cFx9ovK+98JeAxwee+PYeCdBgRXqtay+rgcEqHrdys6NoB10Sx1HUyWZjFlfWkF9eQ680H7klfooqA2QkxR3nCeb/bKhuOmAYWUfAzx2WGbZQ6leheKkO2vgDVriSpSZoUxWw6h0H5qrA53XRPSWekb1S2V2RR0ZqBl3iu5BXdZyka4Uj4MndIaVjCbjMxOxsbH8dfjsSSpvo51DoCHjJwbYd0/H42yx452ftPYowQbtOBO6Kg/hUU5kAxFfkMTB1IDsLd9IlMS58yd9ZCNknIY+5RR/c+T/LsvY1ubtLUhw9U33hCLy0KkC8x1Wv5DC1idmYv317Jxf/YXWjK4D9BZX0z0hEKcXIXqmU1OTTLSGTzMTM4wt4KAJP7gGpvczPpTnH/93aCBHwzkbup6Yt5tEmZp+FI/AOIuDVZZC/A3I3tfdIwgTqTORJSMwEpepdVg/LGMaOoh2kJXrDEWNnYc8xI9jTh4cEvAUf3Pk/6xfIBrvpVXVG9koJJzKLK+tP4oGmOxKuyy7kUHEV+/Lrn0AOFFbQPyMp/Lq4S0lydyEzIZO8yggEPCkT4pLAly4RuNBOhKLs/CYSR4V7zX35EQhWN97f1uTvMvdFTUdo7UEoiRmvFe5kpyw2pRegoPQww7sMJ7cil+SEmk7XkXBPXjlxHhfThnQFWhBwraHkIEWkEaera/MvDRjZK5U9eeVUBy2Kq2pIT6hvydT2BDcRuG1rth8xgr/xQO3kHNvW7C+oZGBXs47piJ6pKE8Zyk4NR+DNevblR8HlhQSnY0haXxFwoZ0oakbAraDxJFOcS8SSDnCJGBLw0sMQ8LfvWBxCForxv535a54444+W5DA8YzgA2ptLUWXnWnB3T145g7olkRjnoWeqLzxppkkq8iDoZ7UyS9Fx9PMmDxvZK5Wgrdl9rLxeK9kQDSPwnKKqcAOsjQeLwscdK6umOmgzwBHwgV0TUJ4yqv3JdE/s3uRszDWH13DL8luoCU2jV85ko9TeHCk9yDXLriG7JNts2/APWHRzi5/RyUYEvLMRisDzGgh4aQ7YQcj6knncERKZ+TucHzQUd4zKmKClUaqGRNs2l9QhnFrw4V2MgFe7cqgJ2o1WlPkis/tYOYMzkwEY0DUxPG29SRz7ZJU9Hgs3HGm6oVRtIrOM4soAaQ0slIbLqoUm/mQkxdWLwEN+fP+uxkKpsEpQSlNankA3p/yzYSXKewff4+Pcj3m/fK8R8PCb9uYVK5/NBZt5I/sNs+2Tv8HmRW1uP4qAdyb8JWYSgnI1jsBDwp51rrnvCInM/J1mrFA7vnYm6EzkSbSt2ggcwlOsuyV0I8OXQZltTjidxUapDlocKKxkcKYRyAFdE48/G7PEBAg7gj0oTBjYdE4GyOqWhM/rYltuab1WsiEaLqu2LbcUl4IrJvZh+5GycLva0FgGZJgIPCTWRwvj6Z5oxLlhLfjeEmMpLg3kmXp/B53Sm6VxJhr/6PBHUJFfm6fZs7L537kVEAHvTIREsO8UqMyHyjo9ykLWysCzjGh2iAh8lxkr1PrzdVi/v5Clm06CF3lsO6z7e0SHBiyN11VpIvAGURmlh1FKMazLMAoC2UDnEfD9BZXYGgZ3D0XgSeSVVTdakCGME6ketLpSmDKs2Qjc7VIM75HCttxSiqtql1ML0dBC2ZZbysBuSUwb0hXL1nyWUwKYunS3S9GnSwJQK9b9Kw7h27cRaByB7ysx34nVrgBFibUdszd6NIe8HgYk9eazvM8o2/UmoMEdJwIutCIhERx2obkPecyhfe54SOtval3buxLFCprEVv8zIS6lyUTmb5bv5N5Fn1EdbLqCIWL+9xQsuwv8pS0eGrRtvK4qxwNvIODVJVBdxvAuwzlSlQ1YnaYSJdQDJWSh9Hci3QPNReHFB9DxqZSSREnqcCg7XD+gqMPIXql8fqiEyhqrURVKUpwbl6q1ULYdKWVkz1Qm9DOCG7JRsgsq6JOegNdtJC8k1g+zmN7v/QqoH4FXBCo4WnmUiwZeRFAp3nDVTp1fWr6XBNvmR4OuwNIWa3cuMQnOMV+FvauarahpDUTAOxMhERx6gbmva6MU7oMuA8DlgvT+7W+hFGWb6cuZwyFjYCMLxbI1mw4WUxWwWJ9d1ORLREwo+ju6pcVDg5bG5fI7HnhdAQ9Nsc5leMZwgjqAKy6/00Tgux0BH1THQoHjVKKUHCSYYj6z8vSRZlszNsrIXqnhOu+GFopSKtwPpcwf4GBhFSN7pZCRFEdWtyQ2HDD/GwcKK8NjAsJlg6PtfNIqj5DqTa4XgWeXZgNwYa9pjKiu4TW/mXZfbVWzvGATsyoqOdOVRqInkTWFW2DQeTBkJviL4fDGlj6uk4YIeGeicJ8Rne6jTLQdThJiBDNjkPk5vV/7R+Chk0u3YWZcDSLwnUfLqHAWA3h/Vwud5I6HbcGxbebnZgSkLkFnJmZiUxE4QGkOw7oMA8Dly239CDxQ1SHq5PfkldMnPYHEOONJD3DqrQ80V4lSfJCaZDMBqrqrI+AtJDKB8HqYFB8w3jO1HQlD5YOh4yf2S2fjgWK01uFJPCHyqvLIUF7cGC87051QbzLP3mJztZrlTmR2eQWb/cfYW7yXVQdXURasZHZ5Bd7yI0zNGMmHHo0edD4MOh9QbWqjiIB3JoqyISMLXG7oOqTWQtHaicCzzOO0fqZlptUGq303R0jAuw4x4yraX+/SNHRp3LdLAqt3nUCf+YI9EHQuj480XcpWl6Blo10BEprywAFKDzMobRAelwdXfG7r9kOpLIQFs+Ev58K7vzB/x3ZiT15FOPoGSEv0kp7obToCd2rA/UlGwFVKDxNYNHMCHdErpfZ1E7zm+c9fBotuAmp7godmbIYFvH86+eXVbDlcSklVoH4EXnGEbjVV/C9pOjV4ybR1PQHfV7IPt3LTT7u4uKICNy5e2/sar+15je6J3ZmifVB6mGnaxyGvh4M9R0JSV+g9AXaviPFTjB4R8M5EXZHuNrRWJCvyIFBhxB2MhaIt40u2F/m7TJVHQroZlx2oV5u+4UARGUlxXD2lH1sOl4Y71kVNSDQSu0VkoQQsjeWySFRuiEuu3RGqny89jNftZVDaIOISj7ReR8Lig/DshZD7GQyZBe/9CpZ9v0391xBaa/bk1ZYQhhiQkdi0gPuLobqUygRz0kuMc0PPMc2eQFN9Xvo6ycf0BK/J1xTuhX3vQfEBI+D+ANtyy0hL8NIrzQfAxP7GB1/yqbE/BnStPcHkFe0hMxDg0KAr+czOoltVeb3ZmNml2fRL6Ye3spBuls20zAks3rWYDw59wKWDLsXt9AU/K99cqa4pc/JLg6dDziem4qsNEAHvLAT8JqoOiXTmcBORB6trk5shcU/vZ+7b00bJ32nsE6gdVx0bZeOBIib1T+fcYaYW+8PdMdooRzeDywOjLoNjW1sUwKBlEVQ2CZ7E2okdAF6fOQk4zf6HdxmOK/4IRRU1WLZu8hbVJB/bqr0d3WrEu+woXL8Yrv0PnH03rP87/Gdem096OlLqp7LGClegYNtgWwzI8HGwoLTxlYHzf1WeYE56CXFu6DEG8raD1fQJb0RPE1WnJ3rrWxSf/TtsoWzLLWVkrxSU83cZ0TMFn9fFkk9NIFLfAz9Kd3cCPcZMZ6M9hMzy+mtj7ivZR1ZaVnga/Zwhl1HgL8DSFrMHzTZXXIV76XdgHX3ciXx4+EPzwoNnmOBn3/sxfprRIQLeWSjeD+g6Efgw0LaxEJwEYUF8b874xQo+LnQildYoJdQa/nUtLP7m8Y/J31Er4KGTjjPOksoAe/IqmNi/C6N7p9El0ctHOw7Ck5Ng4wvRjefIZvM+fU6DQGWL9eZVlh+tINGb1HhnWu1yW8MzhqPdJSz+bCeDf/zfJm8zf/setq0hfzc8dTq8+8um3/SNe+HnGbW3P59pJl19479O2aeCmQ/Dlx+Dba/Bq8f5bE8GW16F348zvWqoTWAOyUw2Vtdj/eDnGTy5cwbvVX0N/eez6ou4kyDfUGJEOSnOY9YWtWpqbb3sD+F3Y8zvA4zuHRLwOGNRdBkIA86CTf8i1eehuDLAjiNl9fxyj9vFuL7pHHOuzkIeuFWSQ74O0i1zFOP7Z7DBHkqPYE14NmbQDpJdmu0I+FFwx3Ne1kUke5MZmTGSIV2GmL/1sa2ooJ9p3cazNnctATtgyl7jktvMB2+xnazwBSEkTKFEZbeh5j5/p4lslYvF+9wcKfXzzOc+TofWqUTZ/Q5sXwbKDbN+btqxNqQiz1yChgQ8tY+psXUi8E9zjP89sV86bpfi7KGZeHa9CtYe+PhpmHhd5OM5utkIQc8xzuPPoduQZg+vcvzyhPi0xjtT+4RPeqFE5txpbvr4hjU6dF9+BYs3HmLfZ6sZvPxG0+J39W9g/FW1fyOAQxvM7zT8EuOvgslhjJ1be6UU4oxvQlUxvPcYnH479D8jss8gWj5+2gQE2R/A8ItqSwi7J8HWRVBTDmffzeZjNezdtp45xz4yieIeowDQxQdQwG/WVvHl0f0Y0TMF3KHPf7O5Ilx0E1jV8L8/w8jZ3DhtICN7pZDm1ZC9GsZdBb3Gw2vfZUTmLhaVGnGuK+AAk/p3Ye2+QjJT4sMJ1qJPX8BWiu79ziYtwUtRxni6OaWoxyqPUVpTStAOGgE/sAOSe+DzJvDk9CdJC/3dnQ6UuOM4a+hl/OfoR2w6tonJPSebyXBtJOASgXcWQvZDKJrt6ohU/i4o3IdO7cN/NhoPcMXuEqyk7ic/ArctePshk7DSlpl63BThChTnJONyQ/qA8Elow/4iXArG9UsH4Jyh3ZhV86459sjnzVYzNKKy0FgePcdA5ghzUmnhudWOgCf60hvvTO0d9ulDPVFGDSznuzOGNrr9+OKRnO36nP5L55oOdzf+F9xeWPHz2tfT2nxeiV3hK0/Dl35kbuf8oLF4hzjru5DcE5Y/2DpJzcK9cOAj87OTrNuTV0GKz0NmcrzZljEIZj5M5Zk/4JeBr5tj95hjg5bN6k82UKXjuHDKaJ66dhIulzJ/a3ccrHkS/n29icjP/A7s/xCKsumSFMeXx/SCnLXmBDF4Ooy+HNzxTC5+Kzy8kT3rC/jE/ubvFJqBidYc2/YKAJmZ5oTSf+AQPJa5osqvyg9P4AlH4E6QMaXnlPCJOZy07n8mU/udi1u5WXN4jdk2eLqxJ5tp0HUyEQHvLBTuMxNiEk2nOOKSTLWJE4FXJPVjx9EybjhzAJatKXD3OPkR+KcvGp/54seh1wTY9FLTx9UtIQyRkRUW8I0HixnWI4VPjq5mwZYFnNfL5hzXZ2zrfYXxsz/7V+PXXPtX2Phi/W2hBGaPMWbBiG7D4Ohmnnp3N+/vbLqypTpobIOkhK6Nd6b2Ngm6f3yFjP/cQiZuduxZ3uTrZOa+x9/jHuewqyfc/LaxQqbdCVsWQ846c9DOt0y0+aX7wJfa5Os0Ii4Jzv+xEbptS+vv27IYXvgq/OOKpm//ntdy8u2zfwMKeo6DPSvNbMdDJQzOTEZZNZTs/4AHu/fgSMURBnRNJJeuFCcNgj0r8Qcs7nhhAxXH9lGZ2JtfXDG2djV6t9ecRI98DoNnELz+FR7xBdke53Xe02HPSnOizToHfGkw4hJGFCzHSxC3SzG0R/1EaljAQwnM3E3kl5g2B5mJmc4xXcit6Q+YCLy+gNefRh8mJOCDp5MSl8K4zHH1BTw01lZGBLyzULTPTIipm3jrNsyIZeFedtZk4nUrvj9zGOP6prHdn35yI/CaClj5/6DvVJMwHH+NqV8+urXxsXk7wZtUe5kKTinhPmzL5tMDRYzu5+EnH/yEJ9Y9weHtf8WtNM9al5hJSp/9p34yMm+n8ZGX/6R+m9xQtN1zrHM/Bn/OJh5/awd3//vTcFe7uqgaI+yJiU0I+JBZZuaovxT8JUyqtvgg/1MCztJhYQJ+eP0HFCUM4CtVP6EizlkdadqdpkHW8gdNCefbD0HGYJj8jRY+3AZMvA4yR8I7P4WgU8b40VPwnxuhYLcR6Ya3qkLY+mp9sWyI1uakm3UuTLgWCvfw0+dfZ9PBYi4d1wsOfsz8ZC+vVh/ipe0v0T0lHp/XxY6kyej9a7j5b++zYvtRpnSpoGufIeFkY5gzvw1n3QXXvMSa/E38O/u//Kz3AOxN/6y9mti9AvpNNeINMP4afIESznN9yqBuSfUWewDonuLjprOyuHyiI7irHuOYz4h8qAfKxP7p7AmYK6b8kmz2leyjW0I3UuNSnQi8CQHvO8X8H4+9EoBz+57LloItHCw9aK5Aug0zJ8xWRgS8s1C3hDBEt2HGm6ws4MPCFKaP6E6XpDiumNiHrZXp2MU5pqLgZPDRU6bP+AWPmpPImK82Hy3n7zQ+tKvOv2dGFtSUs//gfkr9QUri3qAiWEF6fDq/2b+Mg0mjWHIoierRc8377F1V+9x3fgposzjtrjoR8dHNJrpyvqB29zH4KnPJSqwmv7yG+e81cQlcY6oSEpJ6NN7Xaxzc9CbcugJuXcHscx+myO3iw7d/VP+4tfOh5AB50x6m0Erkf3sLzPb4FDjvfjiwBhbdaBK5M39qotNocDn5hcK9sO5ZePtheOvHMHIOfOvj8Pjq3W5bZa5ENjXx9whx8GNjDYy/hop+pumZtWslD1wyklvOGUTOjtd4KTUFl3KxbO8ybG3TPyOR5dWjUUE/cYc+5o/XTKRb8Ki5+mvI+Kth1s/A7WXpnqW4lIvNqobl1UdNaV6oaVQowgUYPJ3q+K5c4V7dyP8O8dDsUZwzNNN49jvfIC9rGgBdfeYkPLR7CrvdI0m1LI7lb2NvyV4TfduW6RnUlID70mDu8yaZCVw66FIUimV7l5n/73FXhe2f1kQE3MEfsHh/53GaurcGAb+5zGrt97Qtk3TKyKKqxmL5liNYtja+o2Ui0i3+blwxyUxtnj2+N7l0w2XX1K4H2BAraBKSkUz2KTsKH/4BRs6G/qebbcmZJmJtGC2D8eW7NUj8OSefvbs2o7wFrC9axuVDLud7g7/Gp26LVYOmUhO0+dg7xXy5QkK0fw3seN3YEEnd6wvUkc9rk5fAmnKzWvlNp+UwcfRm/vbZC/xl4/O8se8NLGeMKmD6dSQm92zx15427DIyXHEszV9fW+NcWQirn6Bo8PmUD+5DgtfN6rozSSfNM7/7tteg3xnmMzsOFdVB/rX2AAvWZIdvBwoqYegsEym/dT98+HuYfBNc+Zwpd2yO8VfDoXX1e+Q4rNpxjB3L5xNw+XixdDxzF+VzSHfj2/33c8s5Jun65KEVuHFxz+R7OFZ5jLVH1tI/I4l/Hu1Hjfbw6NhjXDoizSRsm/PwgdKaUt498C5XDruSYelD+H1GF2o+/adzUtb1BdztoWDQZcxwbWBCt+N8j2wblj8AqX3I65pFhi8Dr3NidLsU7r4T6GbZ5JXsNyWEqVlmnNqu33WyGXom9WRqz6m8tvc1oyHj5podx7uiOQmIgDv8ZvkObnh2bbhmtE1Y8TP4x1dg88ut+z6lh0yJVsYgHntjG7f9Yz3f/ddGarrUVlsUxffmfGcprK7J8WT0GQyAVdhMH+7/PWX81I//fPz3LskxswWtAMz4af19468yk4Xq1szWVJpWow0F3KnMyD+wg+Seb+Nxefj2hG9zecERBtcE+JfeSXK84oVPjpjofttrxspY/oBpzjXtTnO5u/MtI6JWwNQd9zAC7g9YPLbRw0GPm9/mP8Nu+wXcma/yp88e50fv/4iFOxYCoIKOgIcm7hwHr8vLRYPnsCoxgZK3fmw2rv4NurqMe9J93Pr2TYwbVFnfb3d74MJfmlzFhf+vvuXVBA8t2cJ9r3zOw0u3hG/X/PV/+IO2udqJSzYnr0t+ayLz4zH2StOJ8rOF9TZ/uDuf2//+Ib0OvsFrgcn85L/7yC6oRA2ZTp/CT8AKsvnAat7wBLihyxiuHHYlKd4Ulu1dxsT+6SQlp1HT53T6FHxUO7cgfUCzw3g7+21q7BouH3I5P5j8Qw553Pwr+3XY8YZZzqz3xHrHeyZdR5yyuLhySfO/25ZXTI+S6Q+Q5y8K9wAPMaZ/TxKDceyoMlUog9IH1V9KLQJmD57NwbKDbMrbZCbDDTzHBAytGKCJgAMHCytZsMYI1eNv7cAfaIPZbIV7TWINjJC35hJmTvLvsKsnL358gBE9U3j9s1zuWlHbp2Ls2AnEeWr/HSaOGw/Arp1NeNSVhfD+b8zP7z/ebBc58nbAMxdAWS5c93Lj8rxhF0F8Wv2ouGC3uQ9VoIToMgBQ7C/aCMmfMm/0PLrHZ+DZ/DJ3Jw7lQEUOZ0/azfKtR9na/RIzPf6VW+HQepj+AMQlmgjTDpgTZv4uc1JzBPzvH2azuTSB33frgQdYcvkSLuvyVyp2PcjoLpN4etPTlNWUgWWSfAlOI6aWmD38awSU4q38DfDJM7B2PqtHf5m1hVvQaCqTlrA3v4KDdbv2DZ0J9+yCvpOP+9pbD5fyysYcbjoriw0PzmLDg7P42w2TOVRcxfMfZZsyu3uz4fz7WzwRAJDS0/Tz2LQwbJ3ZtuYX/93GlSmbSVWVzLz6e2x4cBbrH5xF70kXQ3UJOmcdv/nkV2RYFjdN+DY+j48LBl7A2/vf5sazevHR/dNJHnWBSWDnrDXv1ZSF4rB0z1Ky0rIY3XU00/pMY1r6cP6S5KVk6yumaVSDE1H3oaehR11Oz83zoexI4xcMVpvvWI+xMO4q8qrywgnMEBP7p2MHU8lR5ooyK9WpQIGIInCAmQNmkuBJYOkeJ3k8/moo3FOblG4FTnkB11qHm7a3iG2biQZF2eZWfAC05vG3duBywW/njudQcRUL1mSbCK01RXXFz423efnTZhwhMW926DZ78gs5WFgZvll202d2y7bwB+vMxnNKCP+00SLO4+IfN5/Ob64cz1v7NaUkka9TuXTK8HqvMXVCSMC3NX6D9x+HmjL46jMmyl39m8bHHPzEzBa0AnDj66ZqoCFeH4z5CnrbUpNoLMqGA/8z+7qZ8VQGKskpyyGnKo8D6b15P3kHPlca3xjzDXNJXX6UcybeypSeU9hatYjuaZqffOJDZwyCnW9C99HmiwQmWdl9tDlhhCpQeo6hsKKG/3t3N6ePKGd5gocbgnEMShvEPTMnkexJI5h/CUXVRfz+k79gB02/jcQIo7JRGaMYnDaIZekZ8PrdBJWb37pKGZA6gO9N+h7ZVRtwJ+7ig4YzSVuKloFfvrGNVJ+X780YSkZSHBlJccwc1YMvDcvkTyt3mz4sEbxOPcZfY66ADpiKilc/PcSWw6V8J2MdpPQideQMMpLiTLIw60ugXLy3+R+sK9/PN8sDJPUztedzBs+hKljFyoMrTQvXkO2xfoG5b8ZCOVh2kA3HNjBn8JxwkvPus35GmcvF/6WnkdNvkvl/KMshaNd+79WMh8z/2qomJkOt/av5jl3wc3C5yavMIzOhoYB3oTxQuy0Lj7lCg6Y98CZI8iYxo/8M3sx+k4LKCnJ6zsL2+Chf+w8OFlZSEzz5qzOd0gJeHbT4zj83MvHnb/P21qPHP7iqCJ67BP4wDv4w3tx+P5ayZy7jnU17ueXsQVwxqS/nD89k0bv/w/q/afCnKa1Ty5mzzmSop90JE64x02/ff9yMsQmKqyo47x/f4LLXZnLeU3/mnF+/yzm/fpcr/ryGgvL6J5lifzHz3pzHhS9fyJZ8p7dH4T5sl5d/7bS5/dzBZKbE89XT+vK3G6awQ/fjsKcv4/vWn5jiS0qj0p1K6dG9vFf3Er9wn/lCTLwOxn7NVCOsnV8/WbPrbXh+jrncvXm5Se41w4q46ahAJTw1xfxN3vihWUA2YxCb8zdz0SsXhW+XdHGzP6Gaywd8g6SaKljxU0joghp2IT+Y/AOKqovoN3wJG3Py2dnT8Y6dLy1gotCQz7vlVXDHUZk6iLsWfkpFTQAylpHhiuem3ANgBUlPjOM704ewbmcSgZIJ/Hvni1RYpjLH5zmOl1wHpRSzB89howcOejwsGXcRe8oOcNeku7h+1PX0TupN/kZsJQAAECpJREFUcq+3eG9HC/+/DXh/Zx6rd+Vz5/QhjZYZu//iEZRXB/nTyt1RvSYAIy4xtsuml/AHLJ54awe3dfucHsdWG1+37gkhMYO9vcfxSN4HDAzafLXntPD+id0n0ie5T2002mOMyUEcWmf+vs3kEJbtXYZCcUnWJeFtw7uNZnZif/6ZlsJFO/8W/n+45vVravuXdB0MU26GDc+bBTpCZH9gRH3wDBg8Hcu2KPAXNBLwjKQ4arzGpkuwbXr8dZZJ/CpXxAIOxkYpqynjvD89xdl/WMfS6kkEP1vEjF8vb743+glwQgKulPqyUmqHUmq3Uuq+kzWoSCjzB7jx2U94/fNceqT6uP0f61j4STNlb6W58PeLTSZ75k/h8j/D5X9Gn/djknJW82/fL7hjislgP3S6i+f1AwSKD5kJA89ccHL7+2ptysSSusO075pts35uSrmaiGRzSgq54KXrKdQbSfVmkNT/ea6fmc8Dl4xke24pVz79ETlF5h/jSMUR5r05j20F2/C4PNz01k18dPgjdNE+clV3uqUkcOu5tZUo54/oTu95z9Htumcbl3QBcd0GMiyukJuf+4RXN5oeH+Erh/McT3f6T0xd7opHzONNC+Glq81EoZuX104cavQxaJ56dzc3v+vh/vif8IOaO1jQ414Cs5+CeUtZc2wDN711EwmeBH565s+Y4LuD2UcyefxYJfeNnm7+Lvm74SvzwRPP6K6juX/q/ewq+5iugxfwrf0TCVy31PRorkvI593xOsGuw7jm2Q18sCuPb8yqZGvRp3yr9/kkBf1QYBJ5t5w9iPnXn8aDo+fiVQHeTVH4VBwuFflX55JBl6BQ/Pusb/CUP5sJmROY0X8G8e54vjvpu9hxOXx45G2CVmQRmuXYGv0yErj+zMZe8oieqXzttL48/9H++tZMJMQlmvK4LUt4YfU2zit/nfvLH0P1nWz6rdRhU94mbvCVY1kBnjh6FO/QWeF95sQ1m49zP+ZIxRFTUTT4fLMzrW/9CiMHrTWv7XmNKT2n0Cu5fo7hwUv/wa9H3cqjZz3Ko2c9yr1T7mV/6X6uf+N6DpQ63/tzf2ROPu88bB5vXWrq21P7wJw/AlBUXYSlrXAJYV0yu5q8y8CkXrgcjeCGJaY6KEIK8/ujg6nEd9nI418bR9dpN5CuKnju7CK6p8ZH/DqRErOAK6XcwFPARcAo4Bql1KiTNbDjkVdWzdXz/8cn2YX87qrxvPG9czh7aCb3vvw5T727u34lSf4u82UvPgDXLYKzvw8Tvg4Tvs7KHjdyW833GeE6SPKLl8LmV8ha+lWSvPA1/4McvmIJeBPhuUthz7snZ/A7/msuT8+/H+KdSQc9x5gxffwXY/E4bDl6kNmLrqHStZevZ93H8rmvMrXnFF499ATejNW8eMvp5JdX89U/r+Ht3Zu47r/XcazyGE/PepqXLnmJPil9+NaKb7E4bys7ajK5e9aw8HTiEH0GjaR31sgmh+rp0p9J6eVMHtiFuxZ+ypLXl5pk0JnfgVTnC5ba29Tvbl4Er/8AFt8GA6YZ26SZyMW2NT9ftpXH39rBZRN687Mf3sPIi27n4f3juWH9EF72F/Dtld+mf0p//jbrOd5ZO4DVGwdyQbfT+XJFPu6/X2gqBG5YAsMuCL/u10d+nV+d+yusuGxyU/+Ppw8lN37z1F5O32Z4p7A723NLeeraCXxS+gIDUwdyxchrzXFOjbjLpbigax7XbvwB11YGsJUiMb6J1z0OPZN6MrXXVJ478BZ5Vfn8YPIPwifMi7Iuok/CUOwub/DJ/mYqfhrwyoYcth8p40cXjiDe07RFcves4bhc8Ou3djS5/7iMvxpqyhj9/h38wvsMaugFcP2rpjOkw+qc1dzy1i2kxqfxj9wjDK8JhD/XELMHzUajeX3v62bD4Bnmvhn7ZFPeJg6WHWT24MaVN77Erlw05btcNuQyLhtyGdeNuo5nL3yWykAl179xPVsLtpp2rufcbayzZXebxl69xpvSTqfcL7RoQ8MIHOD/t3fvwVFXVwDHv2d3s3ltyJOEPAhBQdAqokMFojgGK1V80alVOxUZW6p0VNCx7ZQ608Hpw7bj0Fpn7NhqOz461GqZkWktloo6PhmlVGuFqgWhEB4RSCAhkOzu6R/3l7DJLnmRZn9Lzucf8tvfb3cvd++evXv33nPPGefKVVEyvTtGdO8ROwBPvrWdO1e9S4nOIpq9hbmfyWPOvC9BpIL6w+u6N2AeTjLUaXMiMhtYoaqf946XA6jqCTLywIwZM/SddwY/oH/HrxvYSvLKuFAgQNdCLsX1TGKqJPYlQ8RQhH2BsXQQ7nH/aFwJBoQJYwRp2emWdwfDRMfU8MmBTgIBCEuc8ngTWXTSOQypY0LEiBGkMdCzhxEkRnW8kTgBYt7nanMQOgV+2hqmIeBe/A6U5dkd/DUrRl1cCCh0xuI0hSA3Dvftg4leQrdWgR+UwwfZUBbNoaCsGmEAP2Z1ad0L7QfRYJhoXAloFBB2BapIrGVBqY43EiRGm+TxqZRCH88Tx5W5OC9MWSS7+8pDRzvZe+gogXATWdHTKT58G23tWew5dNTNNS551y1GKaiChauhPPUHzxu73uAb65YSiwfJ0uS5wXl6hLH6KZ9SRH5ROYFAlJ2tO3mw4UHmVl0EP6pywaprsU7zDsgtpuXGJ5n/ylIKwgWs/eLagdcj7oe5e1+7l8smXMbKS1b2OLf+k9dZ9soSJFpCsFcbTaUzroSDgR4bFKSyv+0Y+9s6CAcDg3nVAaiONxIiSiy7kOCYSnq/ntsPbWdy8WQebvgFZQ991n2Q374h6XEWPr+QDw9+SGV+pUvAtf8jN7SWYhZP87Fm2jrbePmGl8lPlSwshW0t21iybgkHjh6gOlINxGH/VvdjdTjiet8J35bao+00tjXy1PynOHfsuT0ea/3Hm1n2+vVkH55Pccf8AT1/l7gq/2lq49Kp5dw9fwxf/st1VOZXkhfKc1NxjxzgoWueYXzFOYN63C4islFVk37VPpmIVA0krrXeCcxM8cS3ArcC1NbWDumJSsJlHDnW86tgfjjUvb9doraOKJ0JX0XjBNkTriUvkEtycxdOG5vvksQX1Lq50mVnQCibwqyjNLa4vBcdWkNZxzZCevLJ+RVhb9Z4ioLJvbhANJ/S6PFeWE2nsDhnHBdUHA9CYeCnqkyJ7mVL3JUvFlTqjsa4pq2QwqwsDiR80N91OM7qeAstVaeTX5hi9WBfckrdNxjvQ771WJTdwXEUBpPzgEg0Qk68lZasGooGMOOhtCib2tLkV2R/pIN9B2cwIbSAQK4LZldPq+KKcyqhvRguuNUNPfUxj7i+up6Vcx7hx288QlST05OKxinoDFE+bgqRiHsdFkxaQMP4BjdOfun33Fhtl+oZ0LCcwsIa7p9zf9Lu5QMxb8I8Nu/fzM1n3Zx0bm7dhdS/dwsfNve/oQS44YlJYyNEcvp++04Yo/x7z+Eh7RkajOZRktVJ0fipKc/PrJzJnefdSSQccVkQ80pSXrfs/GWs2rIKxesoatBdm1uc8vrZVbMHHLzBLXd/4oonePjdh90sIYCsYrcAp+yMlLNv6qvrObMk+cN/zsQzmLrxCxQE68mRwX3LArhyWhVL504iFAxw27Tbune1J6cM4lsIJ04sGCYn0wO/DrhcVRd7xwuBmap6x4nuM9QeuDHGjGYn6oGfzI+Yu4DEblCNd5sxxpgRcDIB/G1gsohMFJEwcCOwpp/7GGOMGSZDHgNX1aiI3AG8AASB36hq/5sKGmOMGRYnNa1CVZ8Hnh+mshhjjBmEjF6JaYwxo5kFcGOMyVAWwI0xJkNZADfGmAw15IU8Q3oykSbgBDsEJCkDBr/kbXSxOuqb1U//rI765pf6maCqSQlcRjSAD4aIvJNq5ZE5zuqob1Y//bM66pvf68eGUIwxJkNZADfGmAzl5wD+q3QXIANYHfXN6qd/Vkd983X9+HYM3BhjTN/83AM3xhjTBwvgxhiToXwZwNO5WbIfich4EXlJRD4QkX+JyDLv9hIRWSciH3n/pt7mZBQRkaCIbBKRP3nHE0Vkg9eWnvZSH49KIlIkIs+KyBYR2Swis60N9SQid3vvsfdFZJWI5Pi5DfkugKdzs2QfiwL3qOpZwCzgdq9OvgO8qKqTgRe949FuGbA54fgnwM9UdRJwEPhaWkrlDw8Ca1V1KnAurp6sDXlEpBpYCsxQ1bNxabJvxMdtyHcBHLgA+FhVt6pqB/B74No0lymtVHW3qv7d+/sw7o1XjauXx73LHgcWpKeE/iAiNcCVwKPesQBzgWe9S0ZtHYlIIXAx8BiAqnaoajPWhnoLAbkiEgLygN34uA35MYCn2iy5Ok1l8R0RqQPOAzYAFaq62zu1B6hIU7H84ufAt3Eb3wOUAs2qGvWOR3Nbmgg0Ab/1hpgeFZF8rA11U9VdwAPADlzgbgE24uM25McAbk5ARCLAH4G7VPVQ4jl180FH7ZxQEbkK2KeqG9NdFp8KAecDv1TV84A2eg2XWBuSYtw3kolAFZAPXJ7WQvXDjwHcNktOQUSycMH7d6q62rt5r4hUeucrgX3pKp8PXAhcIyKf4Ibd5uLGfIu8r8MwutvSTmCnqm7wjp/FBXRrQ8d9Dtimqk2q2gmsxrUr37YhPwZw2yy5F28s9zFgs6quTDi1Bljk/b0IeG6ky+YXqrpcVWtUtQ7XZtar6leAl4DrvMtGbR2p6h7gvyIyxbvpUuADrA0l2gHMEpE87z3XVUe+bUO+XIkpIvNx45ldmyX/MM1FSisRuQh4Ffgnx8d3v4sbB/8DUItL03u9qh5ISyF9REQuAb6pqleJyGm4HnkJsAm4SVWPpbN86SIi03E/8IaBrcAtuE6ctSGPiNwH3ICb+bUJWIwb8/ZlG/JlADfGGNM/Pw6hGGOMGQAL4MYYk6EsgBtjTIayAG6MMRnKArgxxmSoUP+XGJNZRCSGm3IZwuWNWaSqR9JbKmOGn/XAzamoXVWnexnlOoAl/48nEcfeQyZtrPGZU92rwCQRudrL6bxJRP4mIhUAIrJCRJ4UkTe9nNhf77qjiHxLRN4Wkfe8BR6ISJ2Xq/4J4H16pn0wZkTZEIo5ZXn5K64A1gKvAbNUVUVkMS5r4T3epdNwedbzgU0i8mfgbGAyLr2xAGtE5GLccuvJuGGZt0by/2NMbxbAzakoV0T+4f39Ki6PzBTgaS9hUxjYlnD9c6raDrSLyEu4oH0RMA+3dBogggvcO4DtFryNH1gAN6eidlWdnniDiDwErFTVNV6ulBUJp3vnk1Bcr/t+VX2k1+PU4VKxGpN2NgZuRotCjqcBXdTr3LXe3oelwCW4jJgvAF/1crAjItUiUj5ShTVmIKwHbkaLFcAzInIQWI9L2t/lPVzK0DLg+6raCDSKyJnAmy6zKK3ATUBsJAttTF8sG6EZ1URkBdCqqg+kuyzGDJYNoRhjTIayHrgxxmQo64EbY0yGsgBujDEZygK4McZkKAvgxhiToSyAG2NMhvofVetmFLppymgAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [], + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eMlGFN_qx0x1", + "colab_type": "text" + }, + "source": [ + "Looking at the line graph, particularly at Federalist 46 (the huge spike in the middle), there is an unusal spike in the number of times all three branches of government are mentioned, which leads us to assume that this paper discusses the government as a whole and not any particular branch. This assumption would be proven right as this essay by Madison is comparing the difference between the state and federal government. \n", + "\n", + "Another way to interpret this chart would be to treat the papers as a time variable. Since the papers were written in chronological order, we can see that as time went on, the authors wrote more and more about the branches of government.\n", + "\n", + "N-gram analysis provides us with a very simple way to analyze our text by seeing what word(s) occured most frequently. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GKjN-Y2t03aQ", + "colab_type": "text" + }, + "source": [ + "## Dimensionality Reduction\n", + "\n", + "While N-grams are great for visualizing a few words over time, it does struggle in visualizing all the documents together. To visualize the whole Count Vector, we will use dimensionality reduction to construct two \"Principle Component\" vocabulary to present the entire corpus. Recall from the previous chapters that the Principle Component does not represent any real vocabulary but rather it is a mathematical construct that maximizes variance. \n", + "\n", + "### Representing Documents\n", + "\n", + "In an exercise in section 10.2, we used a the KNN algorithm to predict federalist papers authors. However, one question we may have is whether or not topics plays a role in the classification. Below, we run PCA on the 85 federalist papers, making sure we don't plot points where the author is unknown to simplify our analysis. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "nPKk-reyzq5l", + "colab_type": "code", + "outputId": "42cc2807-f769-4fa5-b17a-214cedddbfe3", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 368 + } + }, + "source": [ + "from sklearn.decomposition import PCA\n", + "import numpy as np\n", + "from altair import *\n", + "\n", + "pca = PCA(n_components=2)\n", + "df_data = pca.fit_transform(df_unigram)\n", + "\n", + "# Turn numpy array back into dataframe and reset index \n", + "df_data = pd.DataFrame(df_data, columns=[\"PCA1\", \"PCA2\"], index=np.arange(1,86,1))\n", + "\n", + "# Merge to get Author name \n", + "df_data = df_data.merge(df_author, left_index=True, right_index=True)\n", + "\n", + "# Drop all papers where the author is not certain \n", + "df_data = df_data[~df_data[\"Author\"].isnull()]\n", + "df_data = df_data.reset_index()\n", + "\n", + "Chart(df_data).mark_circle().encode(\n", + " x=\"PCA1\",\n", + " y=\"PCA2\", \n", + " color=\"Author\",\n", + " tooltip=\"index\"\n", + ")" + ], + "execution_count": 5, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "alt.Chart(...)" + ], + "text/html": [ + "\n", + "
\n", + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 5 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-eTIHF96DnO6", + "colab_type": "text" + }, + "source": [ + "From above, we can see that the way that the papers are clustered is very much based off the Count Vector of the author. Each author forms a very distinct cluster. Topics doesn't really play a role in this chart as their location in the plot is rather random. For example, Federalist 78 to 83 are papers by Hamilton about the Judiciary but they're they're scattered all throughout the graph.\n", + "\n", + "There are many other dimensionality reduction techniques that could be applied in a similary fashion, experiment with all of them to get the best results. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3CQRLXxh6wut", + "colab_type": "text" + }, + "source": [ + "### Representing Words\n", + "\n", + "We can apply a similar version of the dimensionality reduction technique to the words of the document. Instead of reducing the number of words there are so we can visualize the documents, we can reduce the number of documents so that we can visualize the words. In this context, PCA creates two new document, PCA Doc 1 and PCA Doc 2, where each value represents the number of times that a word appears in the PCA docs.\n", + "\n", + "-|PCA Document 1|PCA Document 2\n", + "---|---|---\n", + "**word #1**|times word #1 occur in PCA Doc 1|# of times word #1 occur in PCA Doc 2\n", + "**word #2**|...|...\n", + "\n", + "This technique has the added benefit of capturing the meaning of the word. Consider the following block of code where we reduce the number of documents to 2, and plot the words counts." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ktPaYIbg9BOk", + "colab_type": "code", + "outputId": "d1f0b81e-9863-4996-8210-0b820e05d06e", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 296 + } + }, + "source": [ + "from sklearn.decomposition import PCA, KernelPCA\n", + "from sklearn.manifold import MDS\n", + "\n", + "from sklearn.feature_extraction.text import CountVectorizer\n", + "from sklearn.preprocessing import StandardScaler\n", + "\n", + "# Create a CountVectorizer like in 10.2\n", + "vec = CountVectorizer()\n", + "vec.fit(df_author[\"Text\"]) \n", + "tf_sparse = vec.transform(df_author[\"Text\"]).todense()\n", + "\n", + "# Turn the sparse matrix into a Pandas DataFrame \n", + "df_unigram = pd.DataFrame(tf_sparse, columns=vec.get_feature_names(), index=df_author.index)\n", + "\n", + "# Get the three terms we want to examine\n", + "df_unigram = df_unigram.T\n", + "\n", + "# Use a non linear model\n", + "pca = KernelPCA(n_components=2, kernel=\"sigmoid\")\n", + "df_data = pca.fit_transform(df_unigram)\n", + "\n", + "df_data = pd.DataFrame(df_data, columns=[\"Principle Document 1\", \"Principle Document 2\"])\n", + "ax = df_data.plot.scatter(x=\"Principle Document 1\", y=\"Principle Document 2\", alpha=0.35)\n", + "\n", + "ax.annotate(\"monarch\", tuple(df_data.iloc[list(df_unigram.index).index(\"monarch\")]))\n", + "ax.annotate(\"evil\", tuple(df_data.iloc[list(df_unigram.index).index(\"evil\")]))\n", + "ax.annotate(\"good\", tuple(df_data.iloc[list(df_unigram.index).index(\"good\")]))\n", + "ax.annotate(\"executive\", tuple(df_data.iloc[list(df_unigram.index).index(\"executive\")]))" + ], + "execution_count": 6, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Text(0.4415900652159762, -0.09777202180619515, 'executive')" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 6 + }, + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdeZxcVZn4/89z7621q/clnaSTdPYQIBBoIKKgOCgiCDii6MiIP0dRR3TUcb4j43x9qd9xxnHUn/MdmVFG/bmPqOOCAhNkExEh6bAkJGQjaye9r7Xf7fz+qK6m0umlknT1lvN+vfqVrls3dU+q0/Xcc55zniNKKTRN0zRtIsZMN0DTNE2b/XSw0DRN0yalg4WmaZo2KR0sNE3TtEnpYKFpmqZNyprpBky1uro61dzcPNPN0DRNm1O2bdvWo5SqH+/5eRcsmpubaW1tnelmaJqmzSkicnii5/UwlKZpmjYpHSw0TdO0SelgoWmapk1KBwtN0zRtUjpYaJqmaZPSwULTzhJZ12MgZZN1vVn5etrsNu+mzmqadrK2/hSbd3bgegrLFK45t5Gm6uiseT1t9tM9C02b57Kux+adHUSDJo2VYaJBk807O067R1D4enWxIALct/247mHMczMaLETkDSKyR0T2i8gnx3j+3SLSLSLPDX+9dybaqWlzWdr2cD1FNJgbSIgGLVxPkbZP78M9/3oZx+eJ/T08d3SQLQf7eakrMZXN1maZGQsWImICdwHXAuuBd4jI+jFOvUcpdeHw1zentZGaNg9EgiaWKaRsF4CU7WKZQiRonvbrgaL1UB9ByyAaNAkHDH6/r3tKehfF5EJ0vmT6zWTO4lJgv1LqAICI/Bi4Edg1g23StHknZJlcc24jm3d2MJR2R3IMIev0gkXIMrlidT1bDvZjGh6GIbQsq8Ye7q2c7utCcbmQyc7Juh5p2yMSNM+oLdqJZjJYLAaOFjxuAy4b47y3iMiVwF7gY0qpo6NPEJHbgdsBli5dWoKmatrc1lQd5dZNy6bsQ3RlQ4xLl1djmUJVJIjt+fjDr326CnMh0aBFynbZvLODWzctG2nveOe8rWUJnq/oS2Z5dE+3TryXwGxPcP8aaFZKbQB+C3x3rJOUUncrpVqUUi319eMWTdS0s07hcE3IMqmKBqfkbjtkmVy3YREg9CRsUrZ3Rr0VKC63MtY5vQmb7z55iB8+dZh/emA3tutPSSJfO9FM9iyOAUsKHjcNHxuhlOotePhN4IvT0C5NmxdKPb11qnsrhbmVfK9hdG5l9DnxtMO+rjhXrq7DNAwsQ9jbGaehPEQ0aDGUds94aEzLmcmexVZgtYgsF5Eg8Hbg3sITRGRhwcMbgBensX2aNmdN9XTZ8Ux1b+WacxtJ2R4dg5kxeyujzxlIO6yqj1ERCRKyDEIBk4zjY3v+GSfytRPNWM9CKeWKyB3AZsAEvq2U2ikinwNalVL3Ah8RkRsAF+gD3j1T7dW0uWSs4Zq5cJddTG+l8BzTEH7SepSU7RI0DZZUhdnVHqcvaRMOmOMOjekk+Kmb0RXcSqn7gftHHft0wfd3AndOd7s0ba4rZkhntgpZk3+AF55zzbmN3LP1CLuOxwFY01jBVWsbWNkQG/N19Orz0zPbE9yapp2GYoZ05ov68hCxkMUrV9XyxvMbWVoT4Y8Hesc8d7qG5+YjXRtK0+apqU5Az1a52VJCbSwEgGUa4w65zdXhudlA9yw0bR6bygT0bHUqK9QnO1evDB+f7llo2iyhk66n51RWqE90rs5lTEyUUjPdhinV0tKiWltbZ7oZmnZK9AfVmTuVYDv63Kzr8YOnDp+wMjxleyesHp/vRGSbUqplvOf1MJSmzTCddJ0aEw25jR5eGn3uVFfmnY/0MJSmldhkd7w66Vpa4/XaCn8uxUw1PtuHCXWw0LQSKmZ4aS6viZjtRhceHErb/PLZY1x9zgL+8FLPCT+XifIeephQD0NpWskUO7x0Nq2JKLXRw02Fvba+RJZtRwbYeqiPL/zPi2Qd94SfS315iFs3LeOWS5Zw66ZlI8FADxPm6J6FppXIqQwvnS1rIkpprLv/+vIQlikMpW2ePzaIALGQhev57O1KsqAicsLPZaycR/7nGDQNklmXkGUw5J19w4S6Z6FpJXKqO9SdDWsiSmW8u3/IlQMZTLsMphwALlxSRTRkkXU8su7kBQcjQZNE1uGR3V08daCXR3Z3kcg6Z90woe5ZaFqJTPUOddr4JurFNVVHue3yZgCqIgHKIwFcX/HMkX76kzbh4PgFB/NcX+F4PiL5+2sp8b9o9tHBQtNKSA8vTY/JJgmUhwPctHExm3d2kBzMELQM7rx2HTVloUl/Li91JdjbkSASMFAKLlxSja/OfAvZuUYHC00rsWKqqGpnpphe3OkE7qzr8ft93YQDBrFwAIXiuaP9bGiqnHAYaj5Os9XBQtO0eaGYYHCqgTtfpLBlWTXPHxvE9xUZx+eK1fXjvs58nWarg4WmafPGVPfi8sNb4aDJFavqGEjbuJ5iZUNszPNHr+tI2S6bd3bwtpYleL6a0z0NHSw0TdPGccLwlpcb3rpuw8JxP/ALp9nmd+87mkjz3ScPETCNOd3T0MFC0zRtAqeS68hNs3XZ3jaAaRhkHZfBtMsbzl1AdSw00tOYiwUK9ToLTdO0SZzaGphcJe9U1uVgT4rjAymeOTpAX9Ke0wUKZzRYiMgbRGSPiOwXkU9OcN5bRESJyLjlczVN02Za2vaIhQJcubqOYMDgvMWVlEeC+L5ie9sA8bQzZ+t+zViwEBETuAu4FlgPvENE1o9xXjnwV8DT09tCTdO0U5NPiCdsF8sQApawuCqMYQgDKYeBtDNnF2bOZM/iUmC/UuqAUsoGfgzcOMZ5/wf4ZyAznY3TNE07VfmEuOspklmPRMblFSvruKS5mouXVXPb5c1zMrkNMxssFgNHCx63DR8bISIXAUuUUvdNZ8M0TdNOV1N1lHe/cjl//fo1bGiqxHZ9bE9x08bFlIcDM9280zZrZ0NJrgjLV4B3F3Hu7cDtAEuXLi1twzRN0yYRskzWL6pkZUNs3qzknsmexTFgScHjpuFjeeXAecBjInII2ATcO1aSWyl1t1KqRSnVUl9fX8Ima5qmFW8+VRKeyWCxFVgtIstFJAi8Hbg3/6RSalApVaeUalZKNQNPATcopVpnprmapmlnrxkLFkopF7gD2Ay8CPxEKbVTRD4nIjfMVLs0TdO0k81ozkIpdT9w/6hjnx7n3NdMR5s0TdNKZXQ12rlUnXbWJrg1TdPmk9HVaC9oqmTb4X7Stk8kaHDdhkWzelqtDhbavDWX7tq0+W10NdqhtM1XH9pLeShAwDLwfEUi63LHa1fP2v+rOlho89J83VNAm5tGb/uqFHQMZqlvClMeDmC7PruOxxlMOTRUzM5goQsJavNO4V1cY2WYaNBk884Osu7cK96mzQ+F274CpB0PEbCM3EewGi4+mP9zNtLBQpt3Rt/FzeVKn9r8kC8DkrI9OgZzlYteuaqOlO0ykHJIZFzWL6qgKhqc4ZaOTw9DafNO4V1cfreyuVrpU5s/Ru+L0R3Pct/246Qdn0ggl+CerfkK0MFCm4dO2N0s7Y7kLGbzL6J2dijc9jVfQ2quTMLQwUKbl05ldzNNmylTvWd4Kelgoc1bc+kXUdNmO53g1jRN0yY1abAQkZMKsItIXWmao2maps1G4wYLEblKRNqAdhF5UESaC55+sNQN0zRN02aPiXoWXwSuUUrVAXcDvxWRTcPPSclbpmmadhbJuh4DKXvWLh6dKMEdVErtBFBK/UxEXgR+LiJ/C7N4maGmadocMxfK00zUs3BEpDH/YDhw/AnwGWB1idulaZp2Vpgr5WkmChafBBYUHlBKtQGvBr5QykZpmqadLfLlaYKmQTLrEjSNWVmeZtxhKKXUQ+McHwQ+X7IWaZqmnUUiQZNE1mF72wCmIXi+YkV92awrT6PXWWiaps04GefP2UOv4NY0TZtBadsjFrJ47boGbM8naBr0JGzStjerKhAUsyjvrcUc0zRN005dYZVkpZi1VZKLGYa6s8hjp0xE3iAie0Rkv4h8coznPyAiO0TkORF5QkTWT8V1NU3TZouQZXJBUyWP7+vht7s6eXxfDxc0Vc6qXgVMMAwlItcCbwQWi8j/LXiqAnDP9MIiYgJ3Aa8D2oCtInKvUmpXwWk/Ukp9ffj8G4CvAG8402trmqbNFlnX4/m2QV69uh7TFDxP8XzbIOc3Vc2qgDFRzuI40ArcAGwrOB4HPjYF174U2K+UOgAgIj8GbgRGgoVSaqjg/DL0YkBN0+aZ/NTZutjLZfiSg5lZl7OYaOrs88DzIvIjpZRTgmsvBo4WPG4DLht9koh8CPg4EAReW4J2aJqmzZi5srNjMTmLS0XktyKyV0QOiMhBETlQ8pYNU0rdpZRaCfwt8PdjnSMit4tIq4i0dnd3T1fTtNM022vgaNp0Gr0/d8r2ZuXOjsVMnf0WuWGnbcBU/nYfA5YUPG4aPjaeHwP/MdYTSqm7yRU7pKWlRQ9VzWJzoQaOpk23ubCzYzE9i0Gl1ANKqS6lVG/+awquvRVYLSLLRSQIvB24t/AEESmsQXUdsG8KrqvNkLlSA0fTZkLIMqmKBk8pUExnL72YnsWjIvIvwM+BbP6gUuqZM7mwUsoVkTuAzYAJfFsptVNEPge0KqXuBe4QkasBB+gHbjuTa2ozK5/IiwZz/+2iQYuhtDvrEnmaNhdMdy+9mGCRTzq3FBxTTEGyWSl1P3D/qGOfLvj+r870GtrsMVcSeZo2G2Rdj8GUgyJ3g+X5amSIqrCXnv9d2ryzg1s3LSvZjdekwUIpdVVJrqyddfKJvM07OxhKuyN3Q7pXoWk5WdcjbXv0JbP84tljvHBskETGJWAZXLikmtpYkKvW1mMaBhnHoy4WAqanlz5psBCRBcA/AouUUtcOr6J+hVLqWyVpkTavzYVEnqbNhPywUsb2eOZIP2nbI2E79MQdfJXrXVwQqOKfHtjNeYsqeOH4EIYIi6oi09JLLybB/R1yeYVFw4/3Ah8tVYO0+e90EnmaNp8VDitVlwXxPJ99XQlMDEIBg6BlcLQ3xc7jg1iGUF8eZuOSSp450k9bf2paptsWk7OoU0r9RETuhJHEtJ6+ommaNkUKJ390DaVpH8qQtF264xk8QBT4lkHW9aiMBglZBoury1BKuH7DIhorwyW/+SqmZ5EUkVqGS22IyCZgsKSt0jRNO4vkJ38MpW12tsdZWhMlGjBJO7kkd9r1yNgeadtlTUMZlmmQsl3Cw9PQp6OXXkyw+Di59Q8rReQPwPeAD5e0VZqmaWeR/OSPwbTLYMohFg7w1pYl1JWHqSsL0hALsbQ2immYJLIzs9K7mNlQz4jIq4G15LZv2lOiWlGapmlnrabqKLdd3gxAVSSAYQhLaqIcG0izvrGcsnCA7niGoGnw5o2LqYwGpjXvV8xsKJNcqfLm4fNfLyIopb5S4rZpmqadVcrDAW7auDg3Kyrt4fo+DeUhysIBbNcnFDAJmLmE93RPECkmwf1rIAPsAPzSNkfTNO3sVji9/Or1DXzpwb10xzOEAiZrGsoIBcwZWchaTLBoUkptKHlLNE3TNCCXw8hPMb/z2nO4f0c7AoQC5owtZC0mWDwgIq9XSj1Y8tZomqZpJ1hRH+N9V66Y8YWsxQSLp4BfiIhBrqCfAEopVVHSlmmapmnAyz2NmVRMsPgK8Apgh1JK7xWhaZo2C+TrSE1Xb6OYYHEUeEEHCk3TtNlhJjYRKyZYHAAeE5EHOHE/Cz11VtM0bZrkS5bbrs99O45TEQmMlCe/b/tx3ryxqaRrL4oJFgeHv4LDX5qmado0autPcc/WI+w6HsfxfFzf53XnLCAatMg4PlsO9pO2fcojVsl6GcWs4P7slF9V0zRNK0rW9bhv+3EOdCepLw/h+YoXjg/yxwO9vGJFDU8f7CccMFhSE8H2/JJtglTMCu5HGS4iWEgpdcY75Wmz33Qn0TRNO1Ha9kjbPqYhBK1cOb/asiD7uxIMpl36kjZvPG8BlmlgmUbJNkEqZhjqEwXfh4G3AO6UtkKblWYiiaZp2okiQZNI0MDzFbbr4/mKnkSW5toor1hRxzNH+9nfnWRpbRm255dsE6RihqG2jTr0BxHZMuUt0WaVmdjjV9O0E+V79q9b30gi67LreJx41iFje4gY7OoYYu2CcnYcG+JoX3okZ1GK39FihqFqCh4awMVA5VRcXETeAPwrYALfVEp9YdTzHwfeS64n0w28Ryl1eCqurU2scDMWmJ49fjVNe9nonv2bNy7mbS0G92w9wv6uJFXRIArFvq4EFy+r5uaLSzsbqpj9LLYBrcN//hH4a+AvzvTCw9Vs7wKuBdYD7xje37vQs0DLcG2qnwFfPNPrasXJb8aSsnMjjtOxx6+maTmFPfvGyjDRoMmje7oJWSbl4SCXLa8hM9zryDg+r1lbT0NFaTdBKmYYanmJrn0psF8pdQBARH4M3AjsKrj2owXnPwXcWqK2aKPkN2PZvLODobQ7krPQvQpNK73xevaKXC8jHDS5YlUdPYksacdjSU3pc4mT9ixE5EMiUlXwuFpE/nIKrr2Y3OrwvLbhY+P5C+CBKbiuVqR8qeRbLlnCrZuW6eS2pk2T8Xr2VdEg15zbSMr22N+VoPVwP7ar+EnrUdr6UyVtUzHDUO9TSg3kHyil+oH3la5JJxORW4EW4F/Gef52EWkVkdbu7u7pbNq8ly+TrHsUmjZ98j37lH3yFqpN1VHe1rKEspDFq1fXs7axnGjQZPPODrKuV7I2FTN11hQRydeGGs41TMVK7mPAkoLHTcPHTiAiVwOfAl6tlMqOfh5AKXU3cDdAS0uLrmGladqcV7gJ0uh1Tp6vCJgG5ZEAMD0TUIoJFv8D3CMi3xh+/P7hY2dqK7BaRJaTCxJvB/6s8AQR2Qh8A3iDUqprCq6paZo2Z4xXmrxwmCo/tb3UE1CKGYb6W+BR4IPDXw8D/+tML6yUcoE7gM3Ai8BPlFI7ReRzInLD8Gn/AsSAn4rIcyJy75leV9M0ba6baJiqVKSYyuMiEgTWkiv7sUcp5ZSsRWeopaVFtba2znQzNE3TSm4qy/GIyDalVMt4zxezKO81wHeBQ+R2yVsiIrcppR4/o5ZpmqZpp2R0cJjOHfSKyVl8GXi9UmoPgIisAf6L3EpuTdM0bRrkV3RnHA+l4I3nN7Kivnzarl9MsAjkAwWAUmqviARK2CZN0zStQH5Ft+367O9KkHF8nj3Sz0deu5p1iyqmpXdRTIK7VUS+KSKvGf76T3LlPzRN07RpkCvr4bG3M07QMogETY4NpPnyb/fynT8cLPmCPCguWHyQXAmOjwx/7Ro+pmmapk2DSNBEKcg4PpYhHOiOEw6YlIctLFNKviAPiqsNlRWR7wPfV0rp5dGapmnTLGSZvPH8Rp5vG6BzMIvjwfLaCJZlUBUJ0pOwS14RetyeheR8RkR6gD3AHhHpFpFPl6w1mqZp2phW1Jdz57XncN7iChpiIRRwweLKkm54VGiiYaiPAa8ELlFK1SilaoDLgFeKyMdK2ipN0zTtJCvqY3zwqlX8zRvWsqGpEttT07IgDyYehvpz4HVKqZ78AaXUgeGifg8C/29JW6ZpmqadJGSZrF9UycqG2JQtyCvGRMEiUBgo8pRS3XrqrKZp2syazgV5MPEwlH2az2mapmnzzEQ9iwtEZGiM4wKES9QeTdM0bRYaN1gopfRuN5qmaXPEVBYVHEsx5T40TdO0WSxfN8r1cnt0X3Nu45Rvg1zMCm5tlsm6HgMpu+QrNjVNm/3ydaOiQZPGynDJtljVPYs5ZjruIDRNm10Kh5iAE4ab0raH6ymiwdzHeam2WC0qWIjIMmC1UuohEYkAllIqPmWt0IpSeAeR30px884Obt20bFqn0GmaNn0KbxATWQcQYiFr5Gaxvjw0LVusTjoMJSLvA35Gbi9sgCbgl1PaCq0oY91BuJ4ibevhKE2bjwpvEOtiQQ50J9nbmbtPNyVXQBCYli1Wi+lZfAi4FHgaQCm1T0QaprQVWlFmYpN2TdNmTuENYjLrMphy2N+d4FBPkoBlsHFJFWnbo6k6yq2blpV0NlQxCe6sUmpkEZ6IWOT24tam2Uxs0q5p2swpvEFUSrG3K46JUBcLYQo8+VLPyMhCyDKpigZL9nlQTLD4nYj8HRARkdcBPwV+PRUXF5E3iMgeEdkvIp8c4/krReQZEXFF5OapuOZcl7+DuOWSJdy6aZlObmvaPFZ4g3i0P03YMqiJBRlM2/QlbBzH4wdPH541mx99EugGdgDvB+4H/v5MLywiJnAXcC2wHniHiKwfddoR4N3Aj870evNJqe8gNE2bPfI3iLdetozVC8pZVhvFMoTqWIDKaIj6suC0bH40abBQSvlKqf9USr1VKXXz8PdTMQx1KbBfKXVgeJjrx8CNo659SCm1HfCn4HqapmlzUsgyaaqJ8sHXrCRhu8QzLp4Pl6+spTwSmJaJLuMmuEVkBxPkJpRSG87w2ouBowWP28jtl3HKROR24HaApUuXnmGzNE3TZqeW5lq+/NYYX/3tHjqHsrQPZWgbSLOivqzkE10mmg11fUmvPIWUUncDdwO0tLTo5LumafNWLGxRVRaiL5Vbc5G7p5eSX3eiQoKH89+LSCO5YSMFbFVKdUzBtY8BSwoeNw0f0zRN08aRtj1iIYvXrmsgZbsAJVmxPVoxi/LeC2wB/hS4GXhKRN4zBdfeCqwWkeUiEgTeDtw7Ba+raZo2J8UzDod7k8Qzzrjn5KfTdg6l2XZkgCdf6uW5tgH6ktmStq2YRXl/A2xUSvUCiEgt8CTw7TO5sFLKFZE7gM2ACXxbKbVTRD4HtCql7hWRS4BfANXAm0Tks0qpc8/kupqmabNR66FevvH4gZG6b++/cgUtzbUnnReyTK5a28Dn79sFIpQFTTY0VfDonm4WV0dL1rsoJlj0AoV1oOLDx86YUup+clNxC499uuD7reSGpzRN0+ateMbhG48fIBayKA8HRh6vbaygPHzyLtYZx8NXipCZCwzhgInt+iUdiiomWOwHnhaRX5HLWdwIbBeRjwMopb5SkpZpmqadJfqSNq6nRgJDeTjAQMqhL2mfECzy2xM8uruTaDAXWBSK1kN9bGiqLOmMqGKCxUvDX3m/Gv6zfOqbo02XU9lVK+t6DKYcFEovBtS0EqgpC2KZQjzjELIM+hI2KEVNWXDknHz12XjGZUfbIOctKudwfxrfV2QcnytW15f0d3PSYKGU+mzJrq5Niazr0T2UpTueAYHKSJBYyCISNEnZLsmMR18yi+crGisjvNQT54l9vYQsg6pogOs2LKK+PETa9jANIW17I4GhO57lnq1H2HU8NxK5flEFt1yyZOT8UhUt07SzSXk4wPuvXMEX7n+Rg71pBGiui7KnY4iW5toTqs+GTcHzfV5sj/Mn5zSQsF1cT7GyIVbSNk60KO+rSqmPisivGWNxnlLqhpK2TDvJ6N5A1vXYfXyIu3+3nz+81EPS9lEKDIH6WJBgwCRlO3THXXxenpGdJ8DCygDPHupjYU2URMblUE+SjOsRsgxWNVQgwNH+FFWRAJYh7Dg2QDzjEAtZIBA0DS5bXsP6RZVjjq1qmlactY0VREMWG5oqqI+FcHw1krfwfIXrKToG0jy8u5uM6zGQcqiIWJyzqJLrNiws+U3bRD2L7w//+aWStkAryoHuOPfv6CBte/QnbRZXBtnZkeCx3Z2k3FEnKzg2ZJ/0GqMjvgKODzocH+we85qthweB3PxqEzBMIWgZ/PGlXmJBA9MwyDgu92wJsLKhnPdesZy1jRU4riJgiR6y0rRTsOv4IMcGMlRGAsQzLktry3A9RV/SprEyjOt5PLirk1jIIha2CFsGXXGbOy9cTG0sVPL2TbQob9vwt61AWinlw0gBwNK3TCOeySW4jvYl+acHdtM1kKQ7Nf1lsvzhLzxF1svVn8kVLct9P5S1yToD3PHDZ1hYFSbjKOpiQS5uruGmCxezpCaK56sTekR6CEvTXpZ1PVoP9RE0BVPANA32dsZZUB6ipix309XSXMPPnz2Op0B5PmsaK+hP2iSy7swGiwIPA1cDieHHEeBB4PJSNepsls8/PPxiOz9tPUrXYIqu1OyvYNKTGg4cnSkEaB9Is79ziD/u76G2LMSKhhgLKkJcvKya59sGcT0FKK5YXc/KhpgOGtpZLW17WKbJ69cv4OHd3fi2i+PBrZuWjQzvrl9UybKaKAFLqCkLkXE8LFNOSIKXUjHBIqyUygcKlFIJEdGbKEyh/J12W1+Kr//uJX6/r5vBzNzdKlUBtg92VvHi8Tjl0RT7OuPUVQR5cGcn15zXiCHCc0cH2HKwn0uXV3PdhkV6bw7trJVfld1YFeHPNy2lO5FFgMtX1Y2cM5h2WFIT4cmX+tjflWRhZZiPXr162nKFxQSLpIhcpJR6BkBELgbSpW3W2aOtP8VPtx7l6QM9bDs8gDP7OxGnxAX6Ux79eBwbypUjONYXxzBN1jRUEAwYJDIuP95yhLdfspRI0KQvaRMJmNRXhHSPQzsr5Dc52ryzA9dTRILWCbtgZl2P+7YfZ2ltlBW1UbqSNkFTOL+patraWEyw+CjwUxE5Tm4CTSNwS0lbdZbY1xHnjh+1sqer9LtczSYdiVy+4/hgDyEDth7qRynFQ7tyc8gt0yBoGbxyVR3vvWKFnqarnRXqy0O8acOiMdczvdSVYMvBfkDROZSlsSKMGj6+flHltLSvmHUWW0VkHbB2+NAepdT4Va60CbUPpNnTOcS9247y8+2dM92cGZf1IZtyMIBkxsUyhcpogJqyIE8f7MMyDGJhk4BpEA7k7r70cJU23+QX3OXrQhX+P8+6Hr/f103AFDqGskQCBj3JLAsrw/x+X/e05fyK6VkAXAI0D59/kYiglPpeyVo1T/3z/S/w7T8cIevNs7GmKeADvgLHVWSGbBzHJxIy2byznVUNMaqiQdYsKOe+7ce57vxFBCwhGrROWEAI6B6INucULriLBi1StsvmnR3cumkZIcsc3gFPWL+wgkO9SU2yaTwAACAASURBVCRg4niK8xdXAVLy0uR5kwYLEfk+sBJ4jvxcyVwOUweLImVdjw9+5yke2T8w002ZExTQm3Yh7VIVsWgfyJCxfbqGMtiez2N7uvGVwnY9QqZJWTjAoqowlgFloQDlYUsnzLU5I217uJ4iGsx9HEeD1gn7U0SCJomsy77OOAaQdjwWVISJBgw8KPkOeXnF9CxagPVTtO/2WaetP8Wtdz/BoX49cnc6BtIuA+mRyXgEDKiKmMSHZ4tVlQVZU1/Gb57vA6WojoVYUhUhkXW547WrdQ9Dm/XyM6FStjvSs7BMGRUEFKYhVEUD7O9KkrI9LEP44GtWTtv/8WKCxQvkktrtJW7LvJJ1Pf6wt5v3fG/b5CdrRXN86E6+PK24e8gmnnZIOgoDSDlpBlM2WddncJNDQ4UOFtrsVjgTaijtjuQs8kEgtzNegCtX1/G7fT1csTpC0va5aFkVz7cNcn5T1ewYhgLqgF0isgUY2YpJ14Ya356OQd7+70/Qf3LFjRlh8vL4YZ4AAQF7jvcXPSA5PN84n/cYyrh0DGWIZxyCljFyhzaQshFyCXTd49Bmk6bqKLduWjZmzs00BMfzGUjbWIZgWQYRoLYsSE/Cnj05C+AzpW7EfHLvc0f4yI93zNj1DSBoGZj4BIMmleEAS6vLSDkeG5oqyTg+uzsGSNqKBbEgh/uTZBwfT8FQyiEWNkmkPRyVyx0YDJf6mCMcXyGAZRh894+HsUTw8PE8aB/MALB+UTlv3thETVlQJ8O1WSNknfx/MT9LKmV77OkYIm17VJcFaWmuwfb8MYarSqeYqbO/m46GzAcf/+EWfr5j7KJ8pRI0clVmxTBA+VRGgiytjdBcW053IkNzXRmNFeGRMhsZx6M7meXKxnL2dSe5MBZGoWiIhfjFc8eoLQsSDnqkMllStiIYMBGlMAQcz8f2yNWmmdZ/5alRwNG+JD/ZkkQBSkE0ZLFpRQ1V0RAvtA2y6/gQFy2r1tNxtVljrKrS+VlSdY3lLKoMc7gvRVXEwnZ9fKVOGK4qtYlKlD+hlHqViMQ5ubK1UkpVlLx1c8iln7mPrkxpXnt0aXEDsAxYVhsl64GgqI8FWb+wnLdcvJQVDTE8P5cQKyzgd35TFWnb49rzFnL/jnZStkdV1GBDUzU1ZUFSjsfhviTVnqJzUFhYaRLPupiGUF0WzA3jKMEwwHY9ehJZ0u7LbVLMniCSHdUdstMu248OcsnyGroSWRorItSUBXE9n5+1HuVNFyyiPBwYGaKaqNjh5ZdfzpNPPsmhQ4e4/vrreeGFF6bxX6bNR2Ots4iFrBNmSZVHAtTFQty0cdHILKnp7BVPVHX2VcN/6h3xxhHPODz9Ug8f/eEzJEowVhMAXrGqlre1NPHbXZ20DaYRhFX1MW7dtIyGijC266NQhCxz0rH4fDe3KhrktsubAaiKBCiPBEjZLisbYvz169eSyLrEQhaer4ary+aGtvqTDv/9zFH2dyZBFK87dyHrF8Z4Yn8vHYOZXFGzaIigJTzf1s++rtlVFeb4UJYHd3VSHjRori0jkXHYerif3e0J7n3+eK4H1lzNq9fUjxQ7HL1ACuDJJ5+cwX+FNt+Mt87ibS1LcrvnpR1MU/CG/z/OVOn/CYehhsuR71RKrSvFxUXkDcC/ksvBflMp9YVRz4fIree4GOgFblFKHSpFW05V66Fe/vJ7W+lKlabgXzQAf3JOIx973RpW1JfzuvMWTmmCtjwc4KaNi9m8s4Pk8Af9Nec2UhsLjVvueEFFhE9cs+6kLVbfdGHTCW0D2Hqgl4/9+Bm6x3h/YgFIOjPTC0k7PhnH50DP0EgJeNfzSO36Hc8++t/8j+/y9RXr2XTxRhK97bztQ3eyeWcH/p7HeP7ZZ/ja175GLBYjkUhMfjFNK8JY6yz6kjYdgxmWVIW5p7UNX4FlCu+/csWM5dgmDBZKKU9E9ojIUqXUkam88HAgugt4HdAGbBWRe5VSuwpO+wugXym1SkTeDvwzs6AuVTzj8Lc/e27KA0UsCIuqItx2eTOvWNnA4urIyH+MkGWyoCIypdebaAbGeEKWedJ01LHadsmKWt56aTPbDvdyrD9NJmvjeIqlDeVEAybHBzI0VUfZfrR/ZDbTdFHAi+0pXiRFQ3kAa6idtm2PsPI9X8ZBOPabuzgw4LH3oQe49SOfYshz+ek99/CZT//vaW2ndnbIr7OIpx18FJ2DGZ490s/TB3ppH8ywuDrCJcuqqYgEpnWq7GjFzIaqBnYOT51N5g9OwdTZS4H9SqkDACLyY+BGoDBY3MjLs7F+BnxNRGSmFwhub+vnpZ6pS1A0xCyW15YRClr83RvXs27h9KWDxpqBMVWv+2eXLaUyYpHIupiGwbXnLaA6GiKecfjH+18kFsrtsHd8MEUy42MZPvFpmm6czwN1xR1Sz/6RvkO76f/3DwGgXJs9kQr8snq+/YsHOXftWvbu2UPLZZtG/n7W9RhK27MmR6PNXSHL5IKmSr760D7aB9IMpB1iQZN1CyuIBk0GUjZ7OhO8ek09qWmcKjtaMcGiVLdTi4GjBY/bgMvGO0cp5YrIIFAL9BSeJCK3A7cDLF26tETNzWk91Ms7v7n1jF9HgIUVAcQwuPHCxUSDFm88v5EV9fMnRdRUHeXdr1x+Us+loSLMB1+zkm88foCykEV1NMSVqyqojAaIZxx+vb2d7OitYqdY4Ye8q6DsvNdS/ep3n3BOaseDPP3Qb9j+/Haiqzdx16MvccslS1DAD546TEdbOwNJm13HB6e8mNu73/1urr/+em6++eYpe01t9sm6HgMpm6cP9BILWaxZUM6+7gTxlEP7UAbLMPB89fI6i2mcKjvaRLOhwsAHgFXADuBbSqkS/wqfHqXU3cDdAC0tLSW72YtnHG7++lNn/DomcOnyalY0xGjrS3PteQtZ01g+L+f7j9dzaWmuZW1jBX1Jm1jIwjRyvwS5KYHCb3e2k7TVtKzxCC+7gO6f/x8qWm7CLKvCS8dRdpro6ss5/sefYHcdYNmb/5J9nXF+9WwbnqeIBk2ClkHG9fjyg3sn3MBpoplVnudhmvPv565NLj8DKp5xeeZwP+5wQOgcyJDxPFzls35RBT0Jh4yTy2tct2HhrMxZfBdwgN8D1wLrgb+awmsfA5YUPG4aPjbWOW0iYgGV5BLdM+Lyzzx4Rn+/JmqRcX1iQZPmujJStkcwYNBcVzYvA8VkysOBk3b5StsejZURLlpaw3NtA6RtD7vEEcMIhPDtLMe/+QF8J4MEwlS98p2kdj+Om+gn5dh4sXo2f/UT/KqvHTub4fiBPezsskkP9fO7b36O3w118vc97dz5N3/Nxz/2UQBuuukmDhw6TO9gkqvechtX3fQOrjm3kXVLGnj/+9/PQw89xF133cWBAwf40pe+hIiwYcMGvv/97wPw+OOP85WvfIWOjg6++MUv6l7GPFI4A6oqEmDX8UFa2wZwFRiGIC4ksi7H+tNcvb6Rmy5cPOPbD08ULNYrpc4HEJFvAVum+NpbgdUispxcUHg78GejzrkXuA34I3Az8MhM5Ssu+OR9xE/z7zaWWzieUBYyWVxpEQpadAxlR2Y3TNe2iHOBaQj7uuLUlId4zdoGth/t51BfhqAJnp8bMgKImjCV8wv81AAL/5//S6BuKR3f/Rh210sseOcXSe9/msSOh9j1m/+PJQtX8sHPf50D25/iy5/6CGs+8O/ELryWYy89w7r3fJG1MeFzn3sXH77jQwQCAf7j7v/k/r1xLN/h0++5kSuvuY7NOyGZTHLZZZfx5S9/mZ07d/IP//APPPnkk9TV1dHX1zfSpvb2dp544gl2797NDTfcoIPFPDJ6BtR5iyt5+kAfjuuiEEIBg4BpctHSKt77qhU0VIRnuMUTB4uRMqnD+YIpvfDwa94BbCY3MvNtpdROEfkc0KqUuhf4FvB9EdkP9JELKNOu+ZP3nfbfXV4T4T/+vIWgaeApRWNl7ofel7SpKQvqQDGK5ytW1cdoH8qggFULKojbHqYIIoLr+9REA1y0tIb7nj9GxlWYBtjemZUlsaoWEKxvBiBQt5TwsgsQEQL1zbiDnbiDXXivejNbDvSgylbS29vLsa4+fAVVay9DmUG6bINQeTW7Dxyhoq6Rf/3yV/nJf/+cgCH0dB5noP0IlcvOxTRN3vKWtwDwyCOP8Na3vpXyqmoGUjZlFS/venbTTTdhGAbr16+ns1NvlDWfjK40m3E9EHB9MIxc78IwhK64PWM5itEmChYXiMjQ8PcCRIYfT9kKbqXU/cD9o459uuD7DPDWM73OmTiTQLGqLkpdeYhI0GRZbdkJz+kgMbZI0KSuPMSiqjCmYZB1XNKOS8Qy8QDLENY2lvPmC5vY3zXEod4UiaxPwFQ4niIw/HuVPcVeh5gFPw8xRh6LCPgeGBZDWUX7YJrBjIfng+3m6lCZgSAiQlnQxFXw2Xu3k+h5kGd/8Rve+0/f47xlDXz+L99OPJmi1hTC4fAJeYr+lM1/Pn4AEUbKjwCEQi+vd9E7BMwvhZVmOwYyPHOon6AJKXI3TPG0R0UEVi8ox/Nnx8/eGO8JpZSplKoY/ipXSlkF358VpT5ON1CETbhkaQXrF1dSHsltEaoVJ/9LZHuKeMbFR/jo1WtoWV7DhsVVXLysmlsuWcrimgivWNXA2gXllIVMyiMBqiImi6oiuV7IOK8/7n/4SYSXnEty56N0DtkkDz6PGa1AQlEcX6F8j7AJAVPIOj7loSCVpkNZeSWHB11efHE3e7Y/w1Da5aq19Se87uoLN/HD/7qHbXuOsL8rQX9fH5t3dsyaDwitdJqqo6xfWE7rkT4O9yToT7gjs/QsA5QS1HC5ntmg2G1VzzqnGyhesbwKyzCwLJO04+mcxGkYa7Fgvq5V4YyiWy5ZQtgSxDDwfJ+ehA0oxBBCFrieQqmXy7MHBBDBP40P4spX/hm9D/wrR751B2KFqH3jx0aeSznQGXdYWBXCNAQRWHz+K9jywD38+n+/g/PPXceycy7Adn1+u6sTxctTJndnKrj4xvdw3xc+AGJQt2wtH/rMl3WwOAvEMw7f/sMhqiJBEAPTElxXEbByq4AClqCmdvT/jMh86962tLSo1tbWM3qN0w0UH3/dav70oiYqIwGdk5gmWdfjxeOD/Nsj+3F9RW/CJhIQjvWlMUxBEDwgZEB3IrcpklOC2VUmEAoYWKIIBEyyrk/AyPVjFlaEqCoLUBkN4fqKBeUhltRESTs+zx8ZIGAJNWUhgpZBdzzLuYsqeN8MlnXQpsfh3iSfuXcnDeUhdhwboK0/TSLjEQvnpmWfu7CSS5fX8M5Ny0b2mC8lEdmmlGoZ73nds5gC779yOTeOmtqmg8T0CFkmzXUxLlpaTXVZEFNyJdS7ExmuXteYuztTit9s7+CZQ30c6k0ykLLxlCIzhauGPCDl+ERMsH0Xx4Ok8okEhBc7k4QsqIwEqYoEefbIAG9raWJlfYx9nXESaYcBsfGVQgFvPH/6yk5rM6emLIhlCmnHIxq0WFARxvfTLKgIEbRMLl5WRTho6mGo2arl70+tV7Hl7/5E77w2wyJBk3DQzCWIh6t2xkIBVi14OXg318UYSNk8vqeLf7x/N66v8HxvynsZ6VGJ9dRw3auMCyTt4YS54kB3kpX1MVY1xLh/Rwe1MR/TEG6/YsW8WsWvja88HOD9V67gG48fwBDwleLVa+pJOh5rGsopCwemdb+KyehgUWBn2yA9p3C3ubwmPCvmP5/tJtvDOH9OVTTIscEMDeVhQkGDjoHMcC8j1xsptYwPXUNZKiIWA+ksQ2mb/V0JFlWGiAQtlFL84OkjLKmJsqK+fKQqbuEK99nywaGduazrsaqhnA9ftZrNuzowJPf/9OpzFtBYGZ51P28dLAq86WtPnNL51563oEQt0U5VMRV007ZHwDBYWhelczBDxvUREWoiFggMpR1ioQBVZRYdg1lAkZ7ikiOWAb7nsqc9geu1YwhUlYWoigYxDegYzPKLZ49x/uJK7tl6lGTWpTdps3FJNSsbyk4qKTJRKRFt9sqX+sg4Hs8dHWDjkkoWV5eRsl3+8FIPt25aNut+njpYFDjVD4X/9cbzStIO7fRMVkE3P1y1cUkl20UYTDskMlARCTCUcSkLBbh0eTWhoEVzrUMy6zGYtmnrTxEfvfXeaVJ+bgFhAJ/OgQyO5+F4CuUp2gYypG2XfR1D/GJbGxVlQTxPkcp4PLa3i55EBQNph49evYaQZY65u5reHnb2Kyz1ETQF34cXOxIsqIgQDVoMpd0Zqyw7ER0shhU7A2p5TZhrz1ugA8UcVDhcdc6icjKuxwWLK6krD9HWn+bhF7vwVW7VaUtzDb7v43iKrOvz021tDCRszjQn7gCOBxnPJ2ln8YGORG7tqwCmAZZhYAiUR3I9nMqIxVDKo20gw9H+NNesb+ScRRVj7q42G+9ItRPlS31kbI9njw7QMZTG8xXrFsSoiYVmtLLsRHSwAD798+eKOu/QF64rcUu0UiscrnrTBptH93SRsn0aKsJ87sb1PHNkEJHcNrLXnLsIgF8+e4yV9WW0BwyO9membA+L0X0VRa7cA8pHKehK2GQdl07XQxDSWZeAJTy2t4uFVWEyjkc0aOJ6/qy+Ix3L2Tx8lgsEitbDA5SHA6yoi7G/O8HTh/q4aGk1N1ywaFa+JzpYAD/bOrrY7YnqgtD6OR0o5ovCvcgXV0dO+NC6uLn2pA+x2y5vZjDtUB42sT2fjiEbg9yHeyny4vliib2JXHk2QVEdNfGVQjDxfEXHQIbnjg5gGQbhgMGaBeUELeOU70hn4kP7bBs+G/0ehyyTK1bXs+VgP6bhYZnCn6xrYPvwvu+P7ukmaJmz7j3RwYLcdqap7NjP/egvLuXy1fVjP6nNeaPzHGPlPYJW7gM5ZFmsa6xgKNNHwICFVWHqykIMZhw6BtLE0x6ZKZ6Ka5ArLDeUccm4HlXAs0f6yToeqxqiHOhOk8y6PHOknzuvXXdKH/gz8aFdOF4/V4bPxguokwXarOvxUleC3+/rBuSE93hlQ4xLl1cTMA3KQhaP7+2mPGyxvK4M2/Nn5XuigwXwV9es41P37j7p+OdvWKcDhUba9oiFArx2XQNZ1+echbl8QU00TCRk8a7Ll/OdJw8xkHY42JUg46opHaoyhl/MdRVD6Vzi/b4dHVRELETldh5c0xCjpiw05mtkXY/BlIPt+gQtg8pobsHoqX5oT0UvZHRp7tk+fDZeQJ0s0Lb1p7hv+3G2HOwnHDBoaa4hYAi/fPYYb7moiXjW4eKl1Ww70k/7YIaM4/OqVbVYpoFlGrPyPdHBAnjn5Sv50ubd9Bf0LqpDueOali8nbXs+ZSGLpbVRbr54Ca9bn5sPH7QMdhwbZOexQdK2RyrrMpiy8RVTsnFTfljKBCrCFomMS28iSyxkEgtZdCeymIZgGicXEmrrT3HP1iNsO9xPdzxLQ3mYi5ZVce15C0/pQ7vYXshkAWV0ae6U7c5IQreYwDdeL+htLUsKZjMZDKYd7tt+nD+7bBmerzANYfPOjty/K2AQsAye3N9DJGhypDfFT7YeBRFClsFly6u5+eImIgGD8PB7MFPvyWR0sBj27Gev44dPvsQDL3Ry7XkLdKDQRoy16O+GC09c73DLJUv5ldWG6+dmNMUzDkd60/Qk7Clbp+EpaB+yMQ0b14e9nUnCllAWsjh3UcVI8cH8B6FpCPdtP86+zjgp26UyEiBpO+zvTPCo1YllGkV9aMczDr989hiVEYu6WGjcXkgxAaWYBZRjmcrcSrGBb7xeUF/Szs1mcny2HOzD86EnkaUnkaUuFsbxfIYyDrGQwcHeJIYI3fEsaxrKaB9KYyAELYOKsMm2w/0srAxz7fmLeHRP1ym9J9NNB4sC77x8pQ4S2pgmW/TXVB3lvVeu5Kp1CR7b0822w/2sWWBgGNA5ZE/ZsJQC/OHoI0DWVSAOL7YPcag7STiY4Xd7uxhIOaQdD9fLzazyFYQsA9/J/UXHg6vW1fPHA70TfkC19af45bPH2Ha4n8pogAsWV1ITC53UCzmVXEQxCyhHt2GqcisT9Ra84XLg+faM1wvKbTmgaD3URyxs4bo+XUMZAqJYuDZMMuvxyItdiOQmJygleL7i+GCGkGlgDBeYDFgm4njEMx41ZcFTek9mgg4WmlakyRb9hSyT9YsqWdkQ4zVr63l0dyeuAhMhnnVGFvZVhE1cX2EZgohiIH1qfQ8fTpiNlXZgx7E4t39/KwHLoCxo0J9y8HzBdl2qoyE8X9Fn2JSFLHwVJBI0WNkQY2VDrmaWIFRGAyfcwUMur1EVCVA1nOd4/tggFy+tOqkXcqq5iMney7zJglBhL8or2Pth9LH8tcZq59G+NN998hAB0zghGI3XCyoPB0ZmM2Udm4M9cXqTDu2DaQ72pghaBiFT6E44mGauDcvrIsSCAY4NZYinXQKm4LgeSkF52DxhptSpeuyxxwgGg1x++eUAfP3rXycajfKud73rlF9rIjpYaNoUyweN2lgQx4c3bwyTzDp0DmZ4+lAfISs3Bbc/YVMTC3CgO4UhimTWJ1tkkaqxwkt/0sFTuedCphAwhYyr6E/b1ESD9CZtElmP2rIAFy+rHlkFft/246RtH9vzMA2DWMjCMoVXrKglY3tEykzWL6xgV/sQgymHwbTLTRsXn/DBVqpcxERBqDueZfPODnoTNvu64qyqj2GZAghZ22Vvd4I1C8pZUBEeCQCj2xlPO+zrinPl6joqIsGTglG+FzSYclCokVLhKxtiXLS0kmcODxAJBvDiWTwv9177fq4Xt6gqQni4XH3IshBD2NhUxVMHenF8GEi7vGpVLTdubDqhh3aqvYvHHnuMWCw2Eiw+8IEPnNF7Ph4dLDStRKqiQcrDFr5S1MbCRIIWNbEQ152/iKBlYBpCIuuyv3OIf/6fvVSGfdoGUlhGbpjIPYWxK1PAKTg/6ylsb3hWloKlNVE2NFWTsV3Oa6pg2+F+1jZW8L0/HGBXexyloH0wRW0sxE0XLsYwDX69/Th7O+KYhmCZBqvry1hZH+O2y5tHSvAXfridTi5iMuMFoXwSOWQatA+miYUsjg2ksV2fdNalJ2UjQE/C5pr1C04IAIXtTDsuTVWRCXtE+aA0ehjsqnULePpgP0nbxXYVhuS22h164REGt/6aw6YitnQdTZdexx9++i888PDjmAZ89xM3c/PffImFS5by7I+/yJ9+fg+O43DHJ+6EZS3Ytssv//OLHHruSUIBi/e97318+MMfprm5mdbWVurq6mhtbeUTn/gE3/nOd/j617+OaZr84Ac/4N/+7d94+OGHicViXH/99bzrXe9iy5YtABw6dIg3velN7Nixg23btvHxj3+cRCJBXV0d3/nOdyb9WehgoWklMtYwxuhCgLWxEMtqy1hcHeXX29s50BXn0T3d+EphUHy9srE6JPlDCdtnMG0TDpgcHUgRCBjYrgLf57+2HiXr5DaEUsCh3gwdQzZXr6vnUF+KxVVhth0ewPVhf2eCz9ywfiRQjJVLmOpx9/GGgjxfDfc4BM+HqmiArqFMbsFiwsYyoLosxFDaYXdHnPWLKkYCQL638FJXLr90pG+IrniWlmXVhIeDk2kIAyn7hKAUDQqep0YCz8qGGC3N1bT1pwgFDEwRBtsPEt/5exa+84ssqo1x5Nf/RoPfQ+VrXs/3vvYv7DjcxRXXvpmrX9nCt7/6j9Sv3Mjvv/cdBgYGuGBjC//0g/t5/qFfMtB5nL/+j19y26tWkhwaHPf9aW5u5gMf+ACxWIxPfOITADz88MMArFu3Dtu2OXjwIMuXL+eee+7hlltuwXEcPvzhD/OrX/2K+vp67rnnHj71qU9N+rOYkWAhIjXAPUAzcAh4m1Kqf4zz/gfYBDyhlLp+OtuoaVOh2GTuuoWVLK+PcaQ3xWDGBRR7O+L0Jd2TAoYMf51KpmN3e4KXulPDOYsElWGL//jdAZJZ/4SV6D5wtD/Nj7YcpTJscLAnRCRgEAkaVIYtnjkywMXNNcD46zSmYle3wh7LWO9h1vUAxVDGARTxjEPQMkhmXJTyCVkB0nYuN+D6uSR/4ZCY7fo8sruLyojFphU1bDnYx+N7e9i0soaLl1Xzk9ajuJ7C8Xw6hzIksi7e8Ey3hZW5Vf9V0SBXn7OAJ/d3M5iyiWc90oe3k+3YT+cPPk63COI5HKqr528/9Bf8+XWvpqppNRvf/jH6kjY7tzxB+ne/5f9v78zj7KqqfP9dZ7hT3VvzmDmphEAiYTDQDAJBwdeAgi1ItO0WbFDQtm0/jvRTm0Y+bYv0IE+6W32+Tzcq3UbBFgRFEIMiBEJQEgiQOYGkqpKaq+54pv3+OKeKSlGVqspQVans7+cT6tx79jl3rXsuZ529196/tfKBe0KbnBK5zv1sfPZJLr/2z1FiUnB8qqurD/t7vPbaa1mzZg233HILa9asYc2aNWzZsoWXXnqJSy+9FADf92lqahrzXFPVs7gFeFwp9TURuSV6/YUR2t0JpICbJtM4jeZoMt7EZdwyKU9aJG2LqjKb7pxL0c3j+IpUzCBb9AnUGzf2gaABbwQOC0YUOwz1roSS59Pa43HAEAKlMA1GLABV8hUHcj52IQwwMcsgX7JxogV+Bden6PjUpsOFgCMN34w0/j6eMfnRZj8Nbd/eXyJb8ni5pZ+C62MInDq7AhEo+QGmCF05h5p0jEAdXH1w6Awv2xQUYY6h5HmcOa+SjXt7B9dQtPUVeGZnJyc3ZEglLIpOwLYD/YNrWprr05w5vxo/UOxoz1EQqD79Ei750KdozxaZU1mGHwQ8sWkXSikCJ4/vOGza24MfBHz87/+Nz157MQA/eGYPqSiglVz/TTkfy7IIoqlwxWJx5B/RMFavXs373vc+3vve9yIiLFmyhBdffJHly5ezbt26Yb+RQxf8Y0pihwAAHT5JREFUnqpgcRWwKtq+B3iCEYKFUupxEVk1/H2NZqZSmYqxbFY5r7b2oYBM0iYdt1jWmGFTSx8l1yNX8ik4AV6gsC0hbgqur0Kp81HyHF4A2WI4QwiBkhvgE06pPRRuAI7rU3R9Co7Hb7YeoOgFVCRtXohueLWZBH4QHHRzG+mGD4w6BXborKaxpuAOzJBqyCSoXxqn4Pi4fsD7Vs6jImWzrzvPz19sw1cBphhcfmoTi+rSBx1bmbTJJCx2duSwDYPm+jIU8NT2TmzL4KF77uY3D/+EeKYSN1HF7nlLaTj5LDb/+J+J4dJ63yl8/57/JJUpp23XFp6480vk8nmMZDmlnv2sf+U9qGw76x/+BqiAwPOoqK7hosv/hN/eexcr3vdpFp9xPtvW3kfsA+9ARJjtt7HPaWTRaefy6P33ct/7ryRumXR1dVFdXc2CBQt4/vnnueyyy7j//vsHr1Emk6Gvr2/E69fc3Ixpmtx+++2sXr0agKVLl9Le3s66des499xzcV2XrVu3jvnbnKpg0aCUao2224AjqiIkIh8FPgowb968IzRNo5k64pbJ6rPm8sAf9lLyAhrKEyQsAyeA2ZVJljaWs6Wtj0LJo7W/hOsFFFyfTNygO//mIasBAgAFga+wzDB5Pt78ed4L8yeup9jTUaC/uJ8PnD2P5toyHtrURlNFgphtcNOFiwA40Ffk4U0tlCftwRXOD77QgmmEtUOGB4GhCWTXD8g7PrWNYWnZgR7Lgb4igQrrVvuBojPr0NpbGBwaqsvEKbg+FdgsqsvwkQtTI/ZgBmZX1aZtTm4sZ0d7bjBQrVxQTb7ksePljTz56MN8+J9+hOO4/N/PXEvtgpPZ8qM7ePt1n6N68elUv/pTbrvtNv7uq1/nh3d+gXM/8GmqF57Oz793F4Fh8dq9X8Tt3Eussp63v+u9PPXIT0mkUlzxwZv533/xJ+zfsoHLP/Rxnl9zFytWrCAIAhYuXMj9P32Aq1Z8htv/tpPLLzoH27b5yEc+wic+8QluvfVWbrjhBr785S+zatWqQZ/e/e53c8011/DAAw/wzW9+803Xb/Xq1Xzuc59j165dAMRiMe677z4++clP0tvbi+d5fOpTnxrzdyBqtEeRI0REfgU0jrDri8A9SqnKIW27lVJVo5xnFfDZ8eYsVq5cqTZs2HAYFms004ehInReEPYYLj+1kdlVKZ7e3sH31u1mT2ce0xDOWlDFs7u66Su4uJ5HzwhqhrbB4OK8hA2mCDlHYRlhr2M8+Q8DKItbgGLFnAoyCRvHDThzQSXpmEVnPiwBW3ACXtzXy6mzy9nTlccPoCfvsLC2jKWNGeJWqH/U1lvk8lMbeeCFFiqSFuXJGH0Fh99u6+CiJXVkkjb9BZeX9vWyv7/IgBjfn58zj+8/8xrpuEUmYbO/r8Cu9jxXnNZIIlJ0ba5PjzjMVfJ8/vOpXVimkI5ZPLG1HccPWDm3koKrCETR8uR9/HrTbv7o6ptwPJ/f3fsvEEvRtv4XfPjun5OJW5xd63LrX9/IPfc9xPlnv5Xzv7QG04CtW3fQ8pOv0nzd19j6rb/ktM/+gKtXzqZ11zZ+9PXPcdmt9w5qRSVsg7zjTxvBQBF5Xim1crT9x6xnoZS6ZLR9IrJfRJqUUq0i0gQcOFZ2aDTHI0MX+A1P7O7syPH2k+vpL3hs2tfLro48vh9Qm46xr9t/07lMwnyFH4RTOz0fHMJysRN5VAwgrOznKTa+3hMly8FTPgnboitb4p3LG2mqiPNii+KRzW2cOrsCywA/CPjd9nbaegsk4xYn1ZdRcAPu//0+Nr7ec9Dq8IXVKVr7irzWnWdbWz8727MkEzZLGzIopfj2b3dyUl2a7qJHV7bEvu4iDRUxSm7Aptd7eWpHx2BdiEV1mYN8CHMdPi+3hMM2lSmLva05vr27G8cLKE9YGDu7KLo+r7b1UXACegsuKSPAMITWnjxPdxb4dXY/r3Xm+a9n92CbBrYZ5jw8pTBESCfCVd5Zx2Pj6z00CaRiJqfOqWBuVRLLDFdxT0fBwNGYqmGoB4HrgK9Ffx+YIjs0mmnN8OT4G8MoccqTMRoqEmw/kMUUaM86IBA3oeSHPQFTIBkzsCyDkqswDR8vgLJY2KMoehMLGMVo8UfWCcg6DjEDNuzuIW4bFN0AAdIJG8cPONBXYnc8R971yRVD1dueggsCz+3qZmFdmlkVMVIxE98LeG5PN4vrytjVleeUxgyb9/dTXWaRawkT+8/u7KAiadNf9AZVWjMJm76CS2tPkVdb9yISyqF4nmJzSx9/c9kpg/mKzmyJ/17/GnVlMd65vIG23gLrd3WRd31sUwiU0JV3yNtz6fzDg6TOugZRPr1b1mOvvAzHTLLxuWdYfNrZ7HlmLRWLTuP51hLlFRUsk7301i3hlYeeoGzBqaQyGcxEmvrsLq467Qruvft7YdC3DRw/GNTlmo6CgaNxzIahDvmhIjXAj4B5wB7CqbNdIrISuFkpdWPU7kngZCANdAI3KKV+eahz62EozUym5PmDs2YGxv7zjs8pjWn+5bGtbG4JZ+pUpmJYZpijqMnESdom2/b3Y5oGrq+YU5lgy/4shoAhQswUskUPQ6Dw5s7JmMTNcDaNHyjSMZO4bVJwfSxDqE/b7Ogo4CmwBBbWpkgnYniBouT5JCyDlt5iJJNhcumyeubXlvHwpla2tvWTdz0sI7y5DkS2VNwiHTO5+JQGntjSTq7k4gUK1wsT7c31GWrLYixtKufmi5rZ0tbH3Wu383pnHsswqC+PYxjCxtd78P0AN1CIgFIKx4Oe391L9uXfYKersNOVzFp+DmbDYrb/9BuYvkOiZhZnfPBv8OwUf1Tex//cfRuFfJ507Wz+9PNfxbPK2PDcBjb+9x1YpsHp517I+t88zmNPPce6nZ3TsvDTWMNQUxIsjiU6WGhmOqNNLe3MlvinR7fQ2lsgFbMIAkVTRYLaTAIBHtjYQsoW2vsdKlI2OztymBIuQEvFLLpyDgYBjh/ek8epPAKEQcAQ8INwOm/cBss0KLnh+QLAljfOWRY38IJQaM8LoDxpAcJJ9WkaKhKsnF/F99btob2/iGUKPTkXJzp3U3mcRMykJ+8wtzpJS1eB3lIY4fxAkbCFxvI4McuiPpNg2awKfv9aJ/XpBPt6i7T1FgFFVVmMtp4ifUUXP1qUaEqY1/GdArFEEvFL7P3BLSy86q9JNi2h6PpUp+OYhlBwPBbUprlgcQ2WaQ5ei7pMnJ68w0//sO+ghP5AfgKYloKBU5az0Gg0x4bRFvrVpON8/OLFg1pPyZjBFSvCOuIPb2qhusymO+cwrybF/r4SMdOgoSKJKeE6jO68Q01Zgu68Q8kL757phIkImIZBZ84d1SYvWtkn4R8MH1w/GKznYcCgbhVAfykYXCMihDOt6sttuvMOmaRFW28JxwvIlnwStkFVKk5HvoQJxGMmSikKbqgPlXV8iNaM+AJ5R9HaW+L0uXEStsHmlh6278/h+oqqZIwDfUWKnqLkBNRkYihCGXY3CP1IWsLen92N0/EaynOpPO0dxBuXEI8qJhYdHyQMsDdduIjzFte+Ka8Ut0wuXdY4quz4dAoS40X3LDSaGcZoi+F2HMiy9tX9uD6IBLT2lOgtuJiGkHM8unMOs6tS9BVcWnoK4ZNzTRkHsg45x6Mn7+IPCQDDGQgUQzFlfD2UlAELGjJUpcKkuSXC/n4Hx/MoeQo38MkWfVw/7Dm4gSLwwx5JzvHCPEUQ9l4CgZRthbXJ69PUZOI8vaOTukyMhG1RdHzcIGB+dYq2viIxy6QibrG9vZ/2rEMqZmKbBpmERWc2XNhnmoIooaEiztuaa8m5PrYh3Hhh8yFrely8tI7qsvi060WMhO5ZaDQnGCOtGB9pdlV7f4mHN7XQX/RAwPcDZlUmMQ2DXNHhqZ1dvK25lo37eunLO7zS1kcmEaejz0GJwvWg5AWYRphUHhpDhHBYyjIFfxyKiKUAOrNFXD/GOQtr2NOVZ+W8ch59tYOiG0ptpCwD3wRXqXCluqHCBXlDlqwbppBJmMQMEzcIeK0rR7/jk45bdGQdYqZLQ3mS0xoqsAxhZ0eOqmSMynSMVdX1rN1ygETMwo+mK2cSFovrM6yYU4ltwjM7u8m5AZmE/SahxJHk1NduaZ82U2OPFB0sNJoTiKGBZE5VikuXNfLzF1sRoIhwoL9EwjJRwEcvWMQrbf3MqUzyYs5hVmWSvoLP/NokfiB05xy6Cw5mWOUHxw0I6/oIXqDwg3B4yZJw4VzpEInzuAVFJ+DK0+qYW5MKZyrt6aUyadKHIlfyKAZQnbIREdJxm53t/YOS7gM1PvxA0VCeDNVoewq051xScZsz5lXSk3fY1ZFj2awKdhzoww0UdZk4bqCoTtns7swztypJzg3wA4XrKaozMc5ZWE1VVCHw7IVVvOeM2VSmYm8KAMdbffGJooOFRnOCUvJ81m45QE06Nvgk/FpXHjcIsE2DV9r6uXhpPWVxk/s2vE4iZuI4Pr9/vZei41NVZlOXrubV/f1kCx4dQYk5lQl2dxZQUcLYIEwYq0MECgHEMCh6Pi+39rFlf5Z93Xk6sg62ZWCIsLAuQ2e2FM6wMg2UCteLDO3NBNF/OvpL1GfiLGlIs709x5zqFI4XcCDrMLe6jPb+Iv1FH9OEJfUZsiUPyzRCmZW2foqeg2WBH3g016Qp+QFtvcVB1eCG8uSIfkyX+uLHCh0sNJoTlOFPwjHTYGtblvMWV1ObTkTDKAd414omLNOkMhnjyX0d1KTj5B2fBTVJXmnr552nNOIpxamzyvnuk7tIWkUKrkIgWqX8xlDVSANStilEunxsbevHtk0IFLYpVCYsso6PaYDjBxSdANuENrd4UJJkYNMSKLNNWnqKtPYW8f2AouuzrCmDGwRYhoHrKUxTGMjX2qZJzDIIVBjcmuvLcD2FQjG3JjVY6GmsvMPh1hc/XtDBQqM5QRn+JNxbCGc7VSZDifGBYRSJZDZ6Cg5BEN5ETQPm15QRt0zeddosGisS9OQdyp59nVnVZfQXXfKOT67oYRmQjFvYnkfJCxPediQzooCELSQsC1N8DEMIAkXcMojZBk2VCV5u6WdPZ46YZZKMm4CQtBS+MsgWg3D4CYgZUJ+JI2aoqBs3DSqTcbbu78f1A9Jxi7c0Zdi0t5e23iIGgqKf2rI46XiGU5oyPLZ5PyJgWwZva64hYZsjDjmNxkTrix9PGFNtgEajmRoGnoTzjk9bbxHXD1g2K4Pjh4M7A8MoFakwmev5ilzJJ1v0WNZUTm/BxTKFxooEccsMg4plsLCmDAPBNozB9QvFkkcyHiNhQ8yEuG2QTphkEhbVqTgnNaapTsepL09Qn4lTm45TFrc5c1415zbXUJ6waShP0FSRpKbMpq8QYBJKmSDhjcy2DGzLYFYmQcw0mFddxqyqFA3lcQwRbr5oETHbIOd6lMUMyhImjhfg+AGrltax7UCOd61opLk+zeyKBDs6cly8tG7CN/y4NbEAc7ygexYazQnM8CfhAQXY4cMoc6pSXH/+Qi5YUsfPNrXyzM4uAJbNytDeX2JOVYqKlM2yWRl2tudYXJ/mta48xahIUcEN6M07mIbBZy9t5rwlDSRtk85ckad2dGIgrN/dhW0IhiG09BQwDfj5S61kEhY5x2d+jcW8mnA2V97pYlZlnC1tWXIlD8sS4pZBZcoGCWcxWabg+QGmISRsg/k1aebXpCk4AXVpm/6ST9wS8k5AwrbwfMX82jSzq1KUvIDunEN1WXxqL9A0QgcLjeYEZ/gMqdGGUeKWSXN9msqkxXmLq6lMxnD84KB6E6vPmje4KPDkpjRP7eikvd+hPCEEhEWGcq5iXk0KP1DMrk6yYm4VBcfnXafN4rGX2yg4YQ9n/a5uFtUlwmBR8tmyP0s6buMFirnVKS5dVk9FsnNwCO285lpyjsf5zdXc9fg2OrJhWdSGTJzlsyuoSIXlYH0V8PTOLkwjlCdZVFdGdVnsoCE5xw9IxMwZk5w+GuhgodFoDuJQlf0Kjg8ItekEEEp6DJ0eOtADKTg+/UWXzS1ZlBLK4hYqCMg6Pl05h3ue3o1tGgfJlVSmYoPHHugv8tK+/sESrctnlbPx9R4aKhLUpWMUPYVlCKm4hRDW4E7aBoZhsWJuFV+56lQe3LgPz1dkEhZXrJg1pBzr0LXjChBiljGjk9NHAx0sNBrNuBnP9NCBYJOMmSyfleGJLe3kHR9ThJoym309BU5qyJCMmfQUHB7e1ML15y8cPC5umZhGmFTvL7pkEjYKWFBbxg1vW0h9eWJwuKypPMH29ixL6jOU/GDwBr+oLs3HVi1+Uw+pEC3Qe/vJ9Th+QMw06Mg6FBx/RienjwY6WGg0mnEzkemhccvkg+csQMTgpX29mCIsri/DtkxcP2DD9m6CIEyaX7CkjmWzKgaPzSRsbrpwEd/+7U568mEi/WOrmplbXQYcPFw2MJw00rDZcLsGgp3jByMGu/HWSz8R0dpQGo1mwoykP3Wotj15ByG8Kf/Xs3vYtLc36jEoskWPFXMqBnsXQ+kvunTlHKrLYmQS9lGxdTTV3hMdrQ2l0WiOOhN5Ao9b5kGrni9YUsf6Xd2YRrjYbuWCahwvGFEWI5OwDztIwOhy7nq4aeLoYKHRaCaV5vo0Zy+swjYNKpJhVb1AqaM+82gkYb+hM7d0kJgYelGeRqOZVOKWyRUrZqGAjqxD3vGPycyjkYT9PF9FM7o0E0X3LDQazaQzGUNBM13Yb7LRPQuNRjMlHGtZjOFyJseqB3OioHsWGo1mxqKT2UePKelZiEi1iDwmItuiv1UjtDldRNaJyGYR2SQiq6fCVo1Gc3wzU4X9JpupGoa6BXhcKbUEeDx6PZw88CGl1HLgj4FviEjlJNqo0Wg0moipChZXAfdE2/cA7xneQCm1VSm1LdpuAQ4AdZNmoUaj0WgGmapg0aCUao2224CGQzUWkbOBGLBjlP0fFZENIrKhvb396Fqq0Wg0mmOX4BaRXwGNI+z64tAXSiklIqNqjohIE/B94DqlVDBSG6XUd4DvQCj3cdhGazQajWZEjlmwUEpdMto+EdkvIk1KqdYoGBwYpV058DDwRaXUM8fIVI1Go9GMwZQICYrInUCnUuprInILUK2U+vywNjHgF8DPlFLfmMC524E9YzSrBTomaPZ0Zib5o32Zvswkf2aSL3B0/JmvlBo1LzxVwaIG+BEwj/DGfq1SqktEVgI3K6VuFJE/A/4D2Dzk0OuVUi8chc/fcCh1xeONmeSP9mX6MpP8mUm+wOT4MyWL8pRSncA7Rnh/A3BjtP0D4AeTbJpGo9FoRkDLfWg0Go1mTE7UYPGdqTbgKDOT/NG+TF9mkj8zyReYBH9mXKU8jUaj0Rx9TtSehUaj0WgmgA4WGo1GoxmTEyJYjEflNmr3iIj0iMhDk23jWIjIH4vIFhHZHq1NGb4/LiJrov3PisiCybdy/IzDnwtF5Pci4onINVNh43gZhy+fFpGXI/Xkx0Vk/lTYOR7G4cvNIvKiiLwgIr8TkWVTYed4GcufIe2uFhEVTd+flozj2lwvIu3RtXlBRG48qgYopWb8P+DrwC3R9i3AHaO0ewfwbuChqbZ5mF0moS7WIkKNrI3AsmFtPg58K9p+P7Bmqu0+Qn8WACuA7wHXTLXNR+jLxUAq2v7YdL024/SlfMj2lcAjU233kfgTtcsAvwWeAVZOtd1HcG2uB+4+VjacED0LxqFyC6CUehzonyyjJsDZwHal1E6llAP8kNCnoQz18T7gHSIik2jjRBjTH6XUbqXUJmBEPbBpxHh8WauUykcvnwHmTLKN42U8vvQNeVkGTOcZMuP5/wbgduAOoDiZxk2Q8fpyzDhRgsWEVG6nIbOB14e83hu9N2IbpZQH9AI1k2LdxBmPP8cLE/XlBkIZm+nIuHwRkb8UkR2EPfZPTpJth8OY/ojImcBcpdTDk2nYYTDe39nV0XDnfSIy92gaMGOChYj8SkReGuHf8CcjxfR+GtLMUCIJm5XAnVNty5GglPpXpVQz8AXgS1Ntz+EiIgbwz8BnptqWo8TPgAVKqRXAY7wx0nBUmDE1uNVRULmdxuwDhj4lzIneG6nNXhGxgAqgc3LMmzDj8ed4YVy+iMglhPL8FymlSpNk20SZ6HX5IfDvx9SiI2MsfzLAW4AnohHbRuBBEblShdJD04kxr40KZZQG+C5hz++oMWN6FmPwIHBdtH0d8MAU2nI4PAcsEZGFkRrv+wl9GspQH68Bfh31oqYj4/HneGFMX0TkDODbwJVKqen8oDIeX5YMeXkFsG0S7Zsoh/RHKdWrlKpVSi1QSi0gzCdNx0AB47s2TUNeXgm8clQtmOos/yTNJKghrPW9DfgVoSQ6hEMC3x3S7kmgHSgQjgn+r6m2fYhtlwNbCWdEfDF67yuEP26ABPBjYDuwHlg01TYfoT9nRdcgR9hD2jzVNh+BL78C9gMvRP8enGqbj8CXuwiVoF8A1gLLp9rmI/FnWNsnmKazocZ5bf4hujYbo2tz8tH8fC33odFoNJoxOVGGoTQajUZzBOhgodFoNJox0cFCo9FoNGOig4VGo9FoxkQHC41Go9GMiQ4WmmmLiPiReuZLIvJjEUmN0u7pwzz/ShH5P2O0WSAiL03wvAN2bxaRjSLymWi18LRHRE4XkctH2VcjImtFJCsid0+2bZqp5bj4AWtOWApKqdOVUm8BHODmoTujleoopc47nJMrpTYopY6FttGA3cuBS4HLgFuPweccC04nnM8/EkXgy8BnJ88czXRBBwvN8cKTwGIRWSUiT4rIg8DLACKSjf6uEpEnIhG1V0Xk3gHlXRE5S0Sejp7014tIJmr/ULT/70Tk+yKyTsK6Jx8ZboCImCJyp4g8F4m13TSW0Spcsf1R4BMSkhCR/4hqQvxBRC4ecu5/jHpRm0Tkr6L3d4tIbbS9UkSeGGLvPdF3sUdE3isiX4/O+4iI2FG7t4rIb0TkeRH55cAq3+h7uiP6LraKyAXRyuCvAKujntHqYb7klFK/Y3qrs2qOETNGG0ozc4l6EJcBj0RvnQm8RSm1a4TmZwDLgRbgKeB8EVkPrAFWK6WeE5FywlX6w1kBnEMovf0HERmuRHoD0KuUOktE4sBTIvLoKHYMopTaKSImUA/8WfiWOlVETgYeFZGTgA8T1vA4XSnliUj1Ib+UkGbCWhnLgHXA1Uqpz4vI/wBXRPZ/E7hKKdUe3fz/HviL6HhLKXV2NOx0q1LqEhH5W8JVzJ8Yx+drTiB0sNBMZ5Ii8kK0/STw/4DzgPWHuEGvV0rtBYiOXUAo196qlHoO3qjJIG8u9/GAUqoAFERkLWENgReG7H8nsELeqNxXASwBDhkshvE2whs4SqlXRWQPcBJwCWHxKi/a1zWOc/1CKeWKyIuExXEGgumLhH4vJRTKeyzy1QRahxz/k+jv81F7jWZUdLDQTGcKSqnTh74R3fRyhzhmqKKrz8R+48O1b4a/FuCvlFK/nMA5EZFFkS2HIyLo8cZwcWLYvhKAUioQEVe9od0TEPothJpa545y7oHvaqLfk+YEROcsNCcCW4AmETkLIMpXjHRzvCrKKdQAqwiVPofyS+BjQ/IBJ4lI2aE+WETqgG8RlrtUhD2kDw4cD8yL7HsMuGnAriHDULuBt0bbV4/b45AtQJ2InBud0xaR5WMc008o3a3RHIQOFpoZjwrLUK4GvikiGwlvzMOf0gE2Eap1PgPcrpRqGbb/u4RJ9d9H02m/zchP5MmBqbOEirOPArdF+/4NMKKhozXA9Sqsb/Fd4DVgU2Tjn0btbwPuEpENhD2Aifp9DXBHdM4XCIfxDsVaYNlICW4IE+6EBYOuF5G9IrJsIjZpjl+06qxGQzi7CMgqpf5xqm3RaKYjumeh0Wg0mjHRPQuNRqPRjInuWWg0Go1mTHSw0Gg0Gs2Y6GCh0Wg0mjHRwUKj0Wg0Y6KDhUaj0WjG5P8Dspl4ZDDMnz4AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [], + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "j7uONWzz9-Nz", + "colab_type": "text" + }, + "source": [ + "We know that the writers of the federalist papers tried very hard to convince the American public that the executive under the constitution will be good and just because their actions will be checked by other branches of government. This is in contrast of their view that the unchecked monarch is evil and cruel. This conclusion is definitely supported by our graph above where we see that the word \"good\" is close to \"executive\" and \"evil\" is close to \"monarch\"." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UWsQHmBc_HP0", + "colab_type": "text" + }, + "source": [ + "# Exercises" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mtXxcvtgH4E3", + "colab_type": "text" + }, + "source": [ + "1. Download the nucleotide sequence for Immunodeficient viruses and plot each virus on a 2D scatter plot. Treat every character as its own feature (column). Can you tell which viruses are most similar to each other? " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vz5jx7wvc5N5", + "colab_type": "text" + }, + "source": [ + "2. Download the spreadsheet of names at (link) and visualize the names on 2 dimensions. Are you able to derive any meaning from this chart?" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "_9mOjCNLL9Xh", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 296 + }, + "outputId": "ec768d94-5473-47b5-dc33-c7a44615636a" + }, + "source": [ + "from sklearn.manifold import MDS\n", + "from sklearn.decomposition import PCA, KernelPCA\n", + "import pandas as pd \n", + "import numpy as np\n", + "import altair as alt\n", + "\n", + "\n", + "df_names = pd.read_csv(\"https://raw.githubusercontent.com/dlsun/pods/master/data/names/yob2017.txt\", header=None)\n", + "df_names.columns = [\"Name\", \"Gender\", \"Number\"]\n", + "df_names[\"Names\"] = df_names[\"Name\"]\n", + "df_names = df_names.set_index(\"Names\")\n", + "\n", + "def count_alpha(string): \n", + " counts = [0] * 26\n", + " for x in string.lower():\n", + " counts[ord(x) - 97] += 1\n", + " return counts\n", + "\n", + "pca = PCA(n_components=2)\n", + "df_data = pd.DataFrame(pca.fit_transform(pd.DataFrame(np.array(list(df_names.Name.apply(count_alpha))))), columns=[\"PCA Alphabet 1\", \"PCA Alphabet 2\"], index=df_names.index)\n", + "df_data = df_data.merge(df_names, left_index=True, right_index=True)\n", + "pd.DataFrame(pca.fit_transform(pd.DataFrame(np.array(list(df_names.Name.apply(count_alpha))))), columns=[\"PCA Alphabet 1\", \"PCA Alphabet 2\"]).plot.scatter(x=\"PCA Alphabet 1\", y=\"PCA Alphabet 2\", c = df_names.Gender.map({\"M\": \"blue\", \"F\": \"red\"}), s = [5] * df_names.shape[0])\n" + ], + "execution_count": 33, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 33 + }, + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOxdd5gTVfd+J73sAkvvIkpRwAKiqPQiYEFUEERF0U8UFeVTFBABRYRPsf3EAhakKkVRaSpFQHqVKr0j0hZYtm+SeX9/nCSTyWZ3s41dZN7nuU+Sycyde6ecc+6pCkkYMGDAgIHLD6aiHoABAwYMGCgaGAzAgAEDBi5TGAzAgAEDBi5TGAzAgAEDBi5TGAzAgAEDBi5TWIp6ALlB2bJlWaNGjaIehgEDBgxcUti4ceMZkuXCt19SDKBGjRrYsGFDUQ/DgAEDBi4pKIpyONJ2QwVkwIABA5cpDAZgwIABA5cpDAZgwIABA5cpDAZgwIABA5cpDAZgwIABA5cpDAZgwIABA5cpDAZwuYAEfvkFWLKkqEdiwICBYgKDAVwuGD0a6NoVuPtuYPLkoh6NAQMGigEuqUAwA/nApk1ASgpgMgFbtxb1aAwYMFAMYDCAywXDhwN//QU4HEC/fkU9GgMGDBQDGAzgckHt2obkb8CAAR0MG4ABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYwGIABAwYMXKYoMgagKIpDUZR1iqJsURRlh6IobxbVWAwYMGDgckRRrgDSAbQmeT2AGwB0UBSlSRGO5/LErFlAy5bApElFPRIDBgxcZBRZOmiSBJDk/2n1NxbVeC5LpKYCDz0EZGQAa9YA7dsDFSoU9agMGDBwkVCkNgBFUcyKomwGcArAQpJrI+zTW1GUDYqibDh9+vTFH+S/GRaLFIgxmbTvBgwYuGxQpAyApI/kDQCqArhZUZT6Efb5guRNJG8qV67cxR/kvxlWK7B6NfDGG8AffwAlSxb1iAwYMHARUSwqgpE8ryjKEgAdAGwv6vFcVrj2WmkFgb//Bl5+GahYEXjnHcBuL5h+DRgwUCgoMgagKEo5AB4/8XcCaAfgnaIaj4ECQJ8+wLx5gM0G1KoFPPdcUY/IgAED2aAoVwCVAExUFMUMUUXNIDm3CMdjIL8oUUJsCSYTEBtb1KMxYMBADihKL6CtAG4sqvMbiABVBc6eBcqUARQl98d//jlQty5QvjzwyCMFPz4DBgwUKIqFDcBAMYDHA9xyC7BtG3DvvcD33+e+j9hY4PXXC35sBgwYKBQYqSAMCPbuBXbuBLxe4IcfhCH8G7BzJzBgALB4cVGPxICBYgeDARgQXH21GG6tVqBTJ/ksDkhLA378Edi9O/fHkkDz5sC77wIdOgDz5xf8+AoCO3cCzz8PzJ5d1CMxcJlBkYDcSwM33XQTN2zYUNTD+PfC5wNOngQqVcqbDaAw0KEDsGKFEPPNm4VJRQsSiIkBUlLkt9MJnDol24oTKlcG/vlHxvfnn0CdOkU9IgP/MiiKspHkTeHbjRVAdvD5RCpbt66oR3JxYDYLMSouxB8AtmwBkpNlTHv25O5YRRF1VmA+Pp+kvQjg779lhdC8OXDsWMGNObcIVbd5vUU3DgOXHQwGkB3++1+gRw+gVStg0aKiHs3lic8/B6pUAdq1k5ZbdOgAzJwJtG4NTJgAlC6t/TdiBLBqFbByJfDWWwU25Fzj11+Bbt1krvXq5b+/I0eA9etlBWTAQDYwvICyw6ZNIn3a7cCOHUDbtkU9ossPnTtLyw8eeEBaOOrWlXurKPK9qNCoETBtWsH0tXkzcPvt8v3ZZ4HRowumXwP/ShgMIDt8+CHw8MOiFnnssaIezcUFCUycKLrp55//dwZ2vfCC3FurVVxfcwmvF/j6a/ns3buY2M3XrJF4jrQ0WVkYDMBANjBUQNmhcWPROy9dCpQqVSBdqqpoHh56CNi/v0C6LFB4MojjkxeDzVsAzzwjieJ69y7qYRU8fD5RCz34oNh58mD3eO890RK+8krRapB0eOABoGZNMXS//XZRj8ZAMYexArjImD0b+N//xDFl3z5R1RYXJJ/34Ibyx3HY0wxd0RtTsVzSOpw7V9RDKzicPi1qvbQ0kZYB4JtvgLFjJYdRLhAfL9I/Kd0WC5QrJ+pKAwaigLECuMhwOgH6fDDBB6eSVtTDCYIq8Z8ai7DfUw0e2DAD3YT433UX8MUXRT28gsHGjUCNGsA11wC//AJUqyY3pEWLXBN/ABg8GOjaFbj//mK0AjBgIBcwVgAXGXfUPoRPfSOxE1fjv//MBFAISwCfT3RNuVBKb/1+D35OaAHCBIB4DBOBuXOBjh3zPx5VlTTRy5eL3qRly/z3mRcsWiRuoF6vpLrYtg04cACoXTtP3ZUqBUydWsBjNGDgIsJYAVxkKBcS8Lh1Kt7hAFS8kEu/9miwfbskc4uNBX77LerDKtUrDTN8cCEZV2MfvrI+CzRtWiBDyvhyIn74+Bg2bVTF+FFU6NJFqLbNBrz6qngAXXON+OF/951cL1UtuvEZMHCRYTCAi43rrweGDhXiOmtWwff/7bdAQgKQng589FHUh5WvVw4b2w7E56bnsQa3SlK3AvL8ebx/GTymfoOmWIF1KfWFCHfvfvGJ7VVXSSRwcrLm9bN4saiBevSQmAGLRa6hAQOXAQwGUBQYMEDUIW3aFFiXCQlAv37Aa8f6IM1RSqTbhx+OvoO1a1F7+dfoqU5AGcQDw4cDXbtiVLvFsCheNInZjrTzebNZbPY1QDJioIDYnVhZBjtnjkT5XmwoihB5AEhMBNu2xed4Bi/gIxxHJbHoDh9+8ccFiGoqGqZIAj//LM0I9jKQH5C8ZFqjRo1oIDJ69yatVtLhIEcOTiKPHctdBzfeSAo50ZrdzhhcIEDGIJG/v78pT2Nb+tM5XlU6nnc2/IfJN95OxsSQcXHk2bN56q/AMH8+5+AuupBEMzxshmWkzUYOHHjxx7Jokdy8kiXJrVtJkl4v2aMHWaUKOXVqyL4ffki6XNI++ujij9XAJQcAGxiBphorgEsdKSnA2LEwHT0cdGU3xbolfUJu0KBB5hq+ioJ2Zf6EC8mwK+mo37FanobY4t5S2HciFvO+TYBryTzJz7NzJxAXl6f+CgzNm8NrdUEBQSjwNGwinkIjR+a9T58PeOkliTHYvDn64z78UFxTExKAyZMBSA68n3+WlEXPPhuy765dsm9amlzHvMLIO2QgElcors1YAURAt26kw8ELzvJ8tct+vvGGyvQ5v5Fr1+aun/R0snx5kfydTrJXL3LJEnrTvdz42Rqeq30zWbYs+fvvuR+jzycrDIdDPn2+3PdRGPj1V6pWG9/Fy+x11TIeOlQAff74I+l2y3W89troj5syRa6Pw0EuW0aSPHxYFktuN3nbbSH7Hj1K3n472bSpfM8t9u8nW7eWMV5/PZmcnPs+DFxSQBYrgCIn6rlpBgOIgFtv1Yj2V1+RgwcLxXA6yblzc9fXM8+IWsFiIa+4QogSKSoHu13Oc8st0fe3aBH51ltCvRRFjrdYyBMncjeugoSqkidPauMJbdWry5hfe41s357csiX3/a9YIdfQZiNbtcp+v/Hj9cT3yJFM12bvXvL778mkpNwPJSLWrdPP3eWSORv4V8NgAP9WbN8ukmDPnmRqKtmunUZoR47MXV+qSo4bpyeKXi+5cqUwFLdbiGMU3Sz/ahd32q8nzWZ9f506kfHx0u/FxtGjZOXKmQl/eAuMuW7dvJ1n3jzyvfeytnGsWSPX0+Ui77knuj4TEwuGcV57rX6upUuTZ87kv18DxRpZMQDDBnCpo1498SiaOBFwOIB33pHqXo0aAU8+mbu+FEX84kNBArfdJvrs336TREY5YOhQoMNzV6Fh+ios990q/bpc4qZUvboUjb/mGiApKXfjyy9mzYouZ4PPJ7URypTJ23nuvFMC3z7/XHLydO4sfQZw6JBck5QUKcWZEzZulDTWlSvnP7nbtddq32vWBA4fzvs8DVz6iMQVimszVgAXCa++StasSX7zTZ4Ov+02ES5tJg/fu/IT8VSZNk2kfqdT/oyJIZcuLdhx54QtW0THHi7xV6okLlSB31Yr+eST5H33kR9/LEua7HD2LPnmm+TEifp9AysJl4vcuFHbnp4u7j3XXUeuWpXzuNu21cZWpkze5k6S/fqRFSqQderI/DIy8t6XgUsKMFRABi4Wli0Te3L9+hG0Fk88IfrxK64gExIu/uCWLBHCHiDOJhM5ZAg5cyZZrZoYur//nqxVS/53u4NG2SzRqZMwDZeLnDVLiHrLltJfTIwQ7XPn8j7m7t01BlC+vLb95EmyXj1R4yxYkH0f8+bpmZ7FImrDWbNyZnAGLnlkxQAMFVBu8cUXomopVy5vhcqLM0hRMfTqJaqBPKJ5cyktvG0bUKFC2J9ffQUcPChptkuUyN94c4uFC0U9M3++JLoDRF0WGysVuY4eBc6cER2WzSb7kDkniktI0FwqExMludzSpdLf66+L22Ze04nv3w/ccosWvFa+vPbf9OmSUvbsWWDQoKz7+PRT4O679du8XmDSJMlk99xzeRubgUsfkbhCcW15XQH8/DM5ZkwBeFIcOyYSY0CKGjQonx2GQFVJj6fg+ssLfvpJpFiTKXfePpcKRo0SyTegSmnThty9m+zSRS8du93kwYPkiy+GRWBlgQMHyAceIPv3l3sY6mUzcWLexztunKisnE7xSmrUiFy/Xvt/zRq5X2539sFrkdReoU1RjFXAvxwobiogANUALAHwF4AdAF7M6Zi8MIB58+QdcTjIrl1zfbgeW7dqBAQgFy/OZ4d+/PKLDNBkIidMKJg+84K5c+Vimc3iWRQJM2eKJ03nzqLLzgkJCaJH/+23gh1rXvDPP+L3brEIUS1VSvT3O3aI6idwXx98MH+xCmPHCoO5++78jffqq7Uxvf565H327ROVU3YE/OabMxP9uDjte2ysHJ+RQV64kL8xGyiWKI4MoBKAhv7vsQD2ALg2u2PywgC+/lqzO+qCaXKL06fFMAqIDjsKd8iokJSkYyp/lmvHAbcs4apxWwum/9xAVYV49e1L/v23bNu3T+IL7r1XT0AcDvKdd3KWHO+4Q2IIXC5yzhxxVS1K+HzaKs7pFDdaUnT0gViHQIDUP/8U7VhDJffJk/PeT1KSBI4F5l27tsQffPQR+Z//kLt2SdRZmTLyLH7+ecHNwUCxQLFjAJkGAvwMoF12++SFAaSlid2xVSvyr79yfbhg3jwx6IUu7WNiyNWr89hhCMaMCfaZDqs/945KJ5J57tD5/PefH+zdqyeK4c3pJD/9NPs+6tVj0LPGbBaD5eHD5KlTokLr3DlvAVf5wf/9n3jDPPWUxsBeeSXy/LYWASMO4K67hHHGxcn1ygtGjJDn1mZj0CNp4kRRWQHCoAPxHwGGk9f4BwPFFsWaAQCoAeAIgBLZ7VdkXkCB0P5IBGLfvrz1mZoqhKhfPyGyisKUBjfTjjQCpB2pPLk9jy89yR9+kHc8L5kbSEqwVokSkecdsBMoCvn449n3s3GjSJ+BvpxOqkOH8R9LVXpg5l+oy0OlbxRJtEwZWVUUBSpUyDxPm03uUVHB4xE9/+nTcl0eeUTSOOQGgetuswkTvuYa6SPgBWWzSQTy/v2SiM5mkyA2A/8qFFsGACAGwEYA92fxf28AGwBsqF69emFdn6zh8eh9xC0W7bfLlfdVQNu2QkT9y3IPzPwblTj3xQXsWG4dZ/x3ZZ6H/M8/pN3m82tq1Ly5ey9dGln6b9yY/PZbkRZLlCA3b46uvw8/lOtWqhS74TtakMFSiKcDKXQimfOVjvwLdemDIsu2i43HH9fm2Lmz3OcyZcgaNeQ+d+ggRPLLL+X/IUMiR+auXi3XpUwZctu2ghnb7NkyBkURQ3Bu8PDDIqiULautIlRVjPwOh0QGnz4t25OTizZNh4FCQ7FkAACsAH4D8FI0+xfJCuD0aX06g8ceI3/9VQxrgwfnzXti3TodUU2Bg7Wxk1aks6syI99DPjVzKR1IEU2VLS1vzkWpqeLIH673D2cI/ftH3+fZs4yv3YSA6j9cPk3w0oI0WpHK+/A9+dJLeRhwPqGq4i62cKH8DuRYys5zpnZtctIkSZURwIMPavv07VswY/v1V804n1tDls8ndo7wmItdu8QrIpCALqc4AgOXNIodAwCgAJgE4KNojykyFdAzz4gU9fjjBeMuN2WKjphsQEO6kBQkhur5fAZIjRrF301t2Bf/xw1l7hD3ztwgOVmCpUI9nrJqViu5Z0/UXT+KiX7CrzEBE7zB3xakC2O9SDhwgHzuOb23pmfku1yJW3kKZbOfu8Wi5eUPMIHQrJ7z5kU+aXp67qJwVVVWHv36acb5/GD37sxzueoqsc8UlQrOQKGiODKApgAIYCuAzf52Z3bHFBkDUFXy/vtF6mvVKv/pjH0+nc45DTbWx1aa4WFP02Q5n6qKCuHkydz3f/y4PumXokSvjvB4xH0xAsGbjB68GWv4BZ7UM4Aoi5K8N9pHC9J1xH8g3qYV6cHfN5g25+n6Zpw8y0H1fuLIG6ZTPXtOCOzo0eQbb2gBICkp4pI6fXqQkderJ1o4l93DlXF3kR078k7MoQUZtCCD69GQ9Bvo1XDGd8UVDKoCJ03SBrNnj3CWUEyfTpYrJ/fFZhO70vr1uZ/roUNCrMuW1a88coPhw/X31mbT1H0mU9Ek6jNQqCh2DCAvrcgYQEKC5kJns0mQUD4wfz5ZvrzK29x/MgGxJEAfFMYrZRhMSj9ggKw6YmJ4csUefvYZ+eefuTjJX3/pX/Kc0hkEMHFiRN1/ItxBQm1DGk+6asg1KVEiuhXA00+zJRb7u/PSjQQ+gXFUAb6PfrQgg+VKpTE+Phdz9GPxYrJ6zJngyuKJemvI99+X62ezkc8/Lzv27CmSuctFzhBVW506wnjM8PAFfEC6XEH1GaDyFqzkW3iNJnhZF3/xAmLkmqSkiL6/fn0JJMvOvfXIERKgFyY9E6lSRZjzCy9EP9k33tCexTvuyP3FIsnz5zW1ZvnysqooXVo82+rVy1ufBoo1DAaQH6iqBEbZ7WSDBvmO2BWiQ7ocPk6o9Za8eC4XWbIkz748gi1uy2At60Guw02ky8W6Fc8F6daRI1GexOcTAmE2S8RrtKqrGTM0L58QYpUOK0viHK1IZ4zTw8Tth8j//ldyyeSE9HQS4BzcScAXogJSWQVHeBSVhVjv3h3l5DQcPRqI89D6bFTxmLg/2u2ipunVS3Zu104IrsNBfvIJSXLnTtLuJ/guJHGDchMf9qupFHjZ0zmDJXFO7Cm4wHkVeuVuVfbVV0xACY7Bs7QhjeVxggdQg1vQgDWxjw2whUdQNfqVwMKFWirp/HrreL3yLA8aJCvcQMCM0ylqwBdflCCxtm2jC/ozUGxhMID8wusVl88CyKD4ZMNNdCvJdNk8ItUnJMiy3Onkh6aXaFdE0r5dWUHWrMkYty+oaQjNBFAoUFUJBBo4UDyBAhGjZjN3t3qao589ILFTV18tTMLpzF69NH26+JVbrRyEt0PUPxrRbo/5siE2NtcVrvbtC7VNq4zBeW569VtR+7z4omS9DCwr9u8Xptirl0jwftS0HKYJXjqRzBEYxO6YyhEYxNF4ibWwmyZ4aYKXZXGKJ91XRj+4M2d42lyB5XCSCrxBG8dAvM0mWEXARxM8HOj4IFdz5q5dBfcgjB8vD1a4yi888O/22wvmfAaKBAYDKC7Yt49eu4sL0Ya7zddoq4nvviNdLi5EWzpNqXS7yRdeEKn9559l4TFgQB5t0IcOSerhW2/NVQbO40t3s6syk70xlslwasRgwgRJoRzgStmVnwxQZ7OZ36IblbAVAKDyMYxn0FZhs0nfuYiv+OADsmOlTVxiacsDqMEkRxmJ/osSB5Yd4fPXLmRL0xKa4CHgoxNJ/B0taEdqkLkkBNQ/0VbQSk7mImsHf3Cf9OFEEt1IpBVpfqaTxFlPzIl6rAWO776j1xnDd9CfL+E9nkWpzMwgwJzzikmTyBtu0GxFq1dLKdNvv836mI8/lnN26GCkrS4AGAygGODXX8m2zdP5la2PSM6VK2sU3esNLsXX/nCUs2cXoC0ukNoYIDt2jO6YDRv4oP1HmuGhHal8ByGRsldfLRLoPffkbACuVYt0Ovmd9VE+qkzme/gv3UgIqljuwHxmIKxqGEAOGxbVMN95R3hGiRIqH7tpG+1IZRmc5j/lr4tuniSXLyctyMjEmG7Aer/krrIRQlx37703507T0sju3ZlsLckbsZEmePkiPuBcdKQbiWJLsXi47smxQuyKqi6vqvLLe36mE8m0Ip3d8F1kBvDxx7nv+4kn9H2YTHqXaocja0YfCL6MJh23gRyRZwYAwBphW9mcjiuMdikzAK9Xs606bD4eHDFF79K3bZtEYpYsKZyiVStxxcxvzvyTJ8mqVbWXLtoShA0a8Gl8TjtS6UQyv8B/tD5y46d/6hR3vzWdTruoscrgNLegAXs4fuBXEALxMZ5nFRxlP7yvGUmjLBZTty6DauvYGFEvuZHIWU//Kjukpoo75uHDWfbx3HPh9E6lAg+vwh4q8BDw8mFM0nbo3j3npdjYsbpOvTAxHnFcgDZ8ADNYVjnNcc9s0jIV9ugR7RUtOOzbR1osHIvedCKZFmSwC2ZkJv7r1uWq24RzPlawxRPwsRu+pQ8KB2Ik2+MXbkVIbInDoTk9hKNtW7GNlSxZeMFpZ86QDRtKUFzAkSE+vujzVRUCcs0AALQCcAzAGQALANQI+W9TVscVZiuODODoUfLdd3Mu7OTzqiztSqUCH50On/6ZPnZMr4f1p4ag1Zr7ur6hSEiQiFSXSyhk69ZSWzYa3HMPUxxxfNc8kF88vpK+l1+RlMQzZ8r/+/cL0br+eilwng327NEMteXiPPJCr11LAsyAhWZ4hB4ghbtRS+YeKEifDf76S0wVFot4WA4YIEJmjRoh5XhbtBApMjY2Sx/6VatIh02LRQB8fAafcCvq8yasYwNs4V5zHe3+WCw5R4B/8YWOiF6Am7FIIOCjGR6uQWNJM+FwyHxbtMhxvgWO228nIVHob2Aon7F8ydMooyf+1aoJkYza+4B8q9/pkGupcibu9696fLwW2zXiP39+1p2kp0sek5wS8sXHS0GeNm1yNUaSekeH6tXlflitYvfKp6dfcUNeGMB6APX837sA2Augif/3n1kdV5itODKAGjVIq9lLpymNh2ZkowufO5e7HNdzGIZxZcwdkvsmIGmEp+t1uzVPjw8+kBVBXvSg27drnh0mk0Q1R6tXSkqSBGGRUl6HligMqIQCUFWJKg1NQqSq/P7J+fxP7WXcMi/EyAtQBVgT++hGIkvhLM8jJHdNNlJ2qDPM8OHatFJSwhxqYmO1a5rNqiI5mfypQm82xFpakcrKOMqDqMaRGMhn8ClPoLx+znv2yPUcOzZyOoyUFN3+vfE5Qw3gnfCjpMdo2VIIzpgxWY6tUHDiRGZJP7wFAgFNJglCixIL5qQFbT1xOMNlaEqXNYMWs4/NTCtEKAmkn8gvBg8Wom0y5X4VFTrXEiU09zynU579fxHywgC2hP2uB2A3gM7GCkBDbMBDB0nc7Lgla6IVKLYSuvwNPGQNGmjbS5US38QvvyQ/+0wIV0yMSDm5hc8nCcRKlpQUAmazBBHlN+d7OJFo317776OPZMwul+TiJoUhuN3ykt5wg7Zv8+YkwHjEcRoeFHfIQL81amQ7hLfe0tTJN90kLv6//BJhx6+/lqCp++7L1n03I4O82nmEoTaA67GJLiTRjAx2xFz9nP/+m6xXj6n2klSdrsgqpm7dgsd0wo/BfhV4+QWe5Ax04RT0oBcm6TNaz5577pFr2bix3OORI4WARfGMpKaSZ+NVfaBgNC2X9aEXz07iiGa/8sJd3cn77+dPA1dz5Ns+nj6ZzyDKcIwbp0VjR1ugSVVFgghN8rh4sX4FkJVq6hJFXhjABgAVw7ZV9UfsJmZ1XGG24sgAFkw9xduUVRyFARJMkxUDUFWJTL3ySpFuA/VjSZEmu3cXi+aFC+Qtt3AkBtCNJN5tmstkOLirzG2Z6de2bRKNWqsWV0w/xgoV5L2OqOmI8XuwxMRE78WSFQIeQIpCvv22vtTao4/KdpNJC8CaM0dTdTRsyLXjt7O3/RsuQNhKIrTlYKw+On0la1v3s5ztHO12NchTT52iFkmdFY4flwI1Ia6gn35KhruotsWvdCGJFmTwLszRxmYyUT1wkI1MGwiorKPsYtryLPTkX3xB2u3cY6vHxmUP8DbrGv6B2/kp+tCFJLqQxDcwhMvQjFdhLzuWWsXk09kYhM+d01+nZcv0v3/+OctD9+0T+cJqVTkWvbMn+E6nqCKtVpHYAwVjcuGGNno0eT3+ZHdM4Xo00lSRuU1Nkh1UVVyNJ0yILj5n504RgkwmeWCee4666MP4+KJJRljIyAsDaAvg+gjbSwIYnNVxhdmKIwNgRgY5dKhkXQwUF8kOZ89KlOrMmZFfJn+e9kDUrQuJrGk6QIfVyyZNwg7p0SNIiFtW3u0XTlWOGhXhvL17i5QZsC3UqZOtYTRLJCdLXy1akJs2Zf5/715Z0TRsKOkQ5s4ly5fnKWtlpsRVZsbCpXT78x45kMKTKEcV4DI040qEJGCrXDn7cVSpQgJMd5Sgy+YJMoDTCzZpgXWR9PSnTgkVjInRJVbTMwAxAntKlOZovMy++D+eRDmeRwm+g1c4HV256Mb+wf0VqFyzWn8v9+2TqaenU4h0//46Atsf79CCDJrgYU98w2uxPbiSnPjU8qzn7fFoSx9FER15qGdNNgbbMWM0R4Tr3PsiE32A3pKlufHqB0UdZ7GIoPHxx3KeK68U42kO2LmTdFg9wetpRzL743+ciEfpg0nUX0WBxo31c77lFnG4+P57+X/dOmEKS5YUzfgKCYYbaGHhkUc0nX1OhsGFC4VKxcZGzuugqkGPnWZYFtSLB7JPmz6FeSMAACAASURBVM3k+XOqSFBTpwrV8i9/34z7gG4k0YkULp6bErnvkiX1D/+112r/nz1L9ulDvvpq9hLQ8OFahG0kX3tVFcZkt0v/Dgffxcu0Ip1xiOdeez06kRxkACdQnp/g2aA0PAkPy9i+/DLrMZw6pTPgrfxmN3v18ledDBQ6yYqJrF6tLf0djuDmjAxRIwU8Fad8mcyt7ib8AfcxDVJM5R78TBvS6EISP8Zz/iR2wk9DtWp75++h25xKty2dd9/tZwwPP6y79qdRhndiLltjEY+iCntgCt1IpBPJXDs+B0Fi1y5JHxEg9itWiBouB7313r1ayv/PP1PpWbqCv1juFsN7yNjuxs90IYnlcJLxiBMDacAe4HaT06ZlPz7KKtSppOqYKqDSijR+hL5Sl+AiY/VqspztHK/CXh5CdY2JBm5ifLz2bDidUTG6SwUGAygsBFIm22w5G/I6ddJetJdfzvz/tm3BWgOSjsDHm7GK7bCAgMqubc+KTjug8xw8WJ7qtWupQuEK3MYd9hszr0SmTeNa3MyqOMyGWK9lubwuxFf+8cfl3BYL+fTTWc/h/fc1ibNs2cwpAg4dypRL6GrsCUq3X5v+wz9uH8iHMIWzIUnnHsEkSooIH5/CZ9qxH3wgmVi/+kp/joMHtXNYLPql/9Ch2vEWi6xG/vhD+9/rFXtAyZKZir08+qh2aIUyHjqRTDcSJUU1wFuxkoDPz6ge4W/lH+Er/4nn4YN6vfacSk8Fff0rxPmvz+HDpM3GNNj4PP6PnfGDRoRuvJEZZge/t3Tjxl5jCjXwKTVV03j06JToz3uksgqO8jCqkUCQQbuRyOW4XS80lCoVtbfNusE/sSe+YRUEbCukAi9fwTti37rI6NhRpmCCl0PLfiorwMC8bDbxmgsELtrteUvEWExhMIDCwqJFQhDNZlFLZKeHnD5dHjCnM1Nwi+r18Y+mg7gN1/J7dA4+l3ak8H304zx0kP4HDxbCZjJJ7vkABg6Uh7Zz58x5Za66inGID0phbiRwUc2n9A/4U0/ppaE1ayLPIS1Ne2kcjswlE9PSRGJ0OkWaKlmSH5V+g1aksyxO8WiZ6yVFQwhR+QAvMDxH0A3YyGOoxAcxjVdjN6dc85b+PJ9+KvmZZs/Wb8/IEJdAt1tjEjVrZn1PQtCwoTasK2uodJslCrgmRF2yG7XYEXP5Ij5kBqxi83E4yGbNdCks0q65gXdiLivgBKcP2y4MtWxZ0mYL+tyb4OUd+JUZsHAEXuPLGM1z5WvLva1eXRK2FTICJa6FKHr4NgaRAN/EEJrh4a1YwSOoou103XW5j0vZu5f7HxnKGpajtCOVrZyrGL/xoPw3YUL20cAFjA8+0MwQv83zaHWSLRZNIFu8WJL7/fjjRRvXxUB+AsG6RrPtYrRiyQCOH9fqrVosYqTLaf8ILnD9yk+mA8lBSTiwbLYhhQ6k0IUkfo/7xSDXqpUQnWh9lZ97TpfhMqj9OXdOnODT00WHEcj743JpOtFwbNoknjwmE88pcexkmcuW7nU82PB+TTK8cEEvNd5wA8898RLTuz4sQVkhtZVVwC8th9YIkHEGpFOAtCJNUln06BFdoM4mCbKaZ+nEe23zOPuBCTnaPObO1VRAkyaR91x3iDWUg3pj9RVXkHXq8M+Og9jFPIufoo9sj4nRoloPHhQm9+23siIJme936EYXkmhDGrtjKsfgOTr8Ubg9MUF7jiK536qq2FWijeXIAXPmkDFIoAle2pHKR50zguqg72Meo11JowUZfBNDxF317FlZPsyYEX3Opl9+yZRYkL1761ViucmGmk+sXy/2Cb7xhggIZrPYs0ITEaqqCDKvv042aZKPuqrFB/lhAJlcPg03UHlG3n+fvK9DMrfE3h5dfdwscHL7KX8qAjJcZxqQjC3IEE8jQAy40Q5y+nQmDHib/4dnWRJnCXjoQBJ7PZwuNQncbpGkSZH6b7pJEqhFWsns2EGPI4aP+Y2Wj2E8rUinAi8fwPfyYpPB7J+6FogVGDFCt308Hgsj/KFz165FWZyUP53O7O0DIUic8H0wl48ZaTwbU02OzUJX/scfpMsl56zkPi8rpPCqaP5WBUeD6qB1uElWAmFM88gRMnHQ27rjVIAT8ShH4VVeQAw/Ql86kEIr0vlIINpYUSK76vbvL0QrLk6CByPhtddkzNl4A+nw5Zf8xdSRdqRQgZcV8A//RmXd9a+KI1rARfnyWqH6aFYDzZplvn633qrVUgBkvBcbL74YVHlesJZmE2U14xzJnDU5SZ7VgLNEQNV5iSMvXkAdAYwBcBLAxyFtAoB1WR1XmK04MYA//iBdTpHWa+CAXMq2bfPU16J3N/ql/0jET/SmN2CTlqjL6RRf8JxewJkzuc9aly5cIODVqVleNI/R4hJMJs29yOcjN2yIuEpZ+dxUlsfx4NhKIj5ovO2Pd/WeHTVqZH7xly2ToLaQbaPwasi4IhN/Mzycc9MwLfLr11+znbY3NYMx/nxDoaupzqFpDm69VXZOTRWJXVUlbs6cTivS2UDZKvmIKleOyABqYi8VfwbRLWggfvghkvmQIX5aXUrl4TrtIvZBSJrtIXiDz+NjMbgCEtZstdJ7f1fOmCEBs6pKbSxut6gTwxF6bU2m7J8NkqpP5fPmz1gS52j2CyBuJHI1Guuu/+t4I9O4U2wlOa7TXM4dlkOaiIcekhWyxSKfpUqJjWrWLJG+LZb8uyXnBgcPikfTmjVi9LntNk6xPk6X32ZTs/wFvQ3LZtM7S1yiyAsDuB7AYwAO+z8D7X4AcVkdV5itqBlAWkIap72wkj8NWs036n5HO5Jpgof1lO3yMFeqlKci8SnxKWxi3xSSKVMjhCZ4+HineHrW/6l/CW22zPngV68WXWbAO+TTT1kqqPtnSL9ikA0m0qlSRfOJ79kzWJsg3NhXz75H18+V2MdZ6MwJ6EnPlbUyu7WePKkfc6BK1mefkTfeSAL8DL0Zrv8PbbE4x724SojruHE5En+SvLX09rDr6L9kSOFCtNHyDX35peYW07MnSXJx54/5jnkQ/7FfIZXRAlJgWNuHmnwJo/mjpYswplKlRCqeOJFs1YpXxJwJ0urJ75/MlHI5AxbOQifegTlUkEE3ErgLVzMZDnpgYirsfBYf02n30uXyZ8YIGOCrV4/sofLVV9o5FCXH63Rkzd+0IS0oZLTFr1yCFiTAVlhEM9L5QKTcQG43HynxE51IpgtJ2TOBpCRRvk+bJt5b9eqJF1y0K5SChMcjKtRABbRdu8gTJ7ijfEu6kES3JZW9YkLm27SpXPOc0lEUNlRVbEL5KEebr2RwAJwA6uS0b2G3omYA3auv9Nfu1QhUJ2U2Dzd/RHOTiyAtTJkitVN09HTcOJH0evQQz5Q332QHzNURLDPSuRHXcwHasDmW8CP0ZSLcfB4fsbGyjh8/8af2TFy4oBEZl4tMSeGKr/6KIFkHiPdezS00JsbvQ0lZ4gcoV1jATteqK4MlHS3I4Pe4jymw86WyE/j+kLORa5pMniyeOGPHyu8FC8gff6SanMJHMYHwu1KGq3+sSOaV2M0laCp/dO8e9X1ymNIyEX/x7U+nC0kcg2f1RuLANSOFYL3+ugS4vf66Zt/JqgU8p8K2f4QXaFa8rFzZn8ssMTF4bX1QeBdmZ2J8MTjPQKRweZwI1kq2mHwcMsQ/udTUrIvHDBigjaFhwxyvU0p8CisoJxmDC7wS+8U/PxArEjqfEiW07yYTeeIEm5X4kwE7zdgeUWbrDE15UhSVx5KT9feqbFl591SVBz6ZyyX29hKVDYhK7yIaqLOEqmqR382b57kcbX4YwD3+FBAH/b9vADA7p+MKoxU1A9AkYI2w3GFboickYdk2ly/XCmzdeGPIHwF3M7dbpKGrr+Yu1KYVacH+W2AxPVCCKYkBlV3xnY5w/O9//v7OndPGYbfTez6RJRxpzIoB/DnrgHg72GwiwQaknE8/lW033pjJ2Ji8bD07m39mDBLYBCuYAgevN28N9vvGGzlcwClT5GK43fz4xq/DpP3MTAAg4xAvGytWjO4mqSpvqXki03yr4wBN8FKBl73wtURda5VkImc43bqVbNeO35V6mg4kMxbnuQN1GDbYLFsynPSO+0qT3EaNIgE+iGnUVFOhc/YGvwdsQgp8bIOFPH0kQmxHOH74garDycPWq+gZ8qa2PUA0vF6J3bjqqqCXy+mH+nKu5V6et5UTifftt8U7JyBMhDIDqzWYT+mvOfvYtMRm9rhiOVPioxgbqb8+7dpFd0xB4/33tTlZLNrKd9Ikbc4lSkiuruJQh+D8eX052jymqMgPA9joj/79M2TbtpyOK4xW1AxgxWdbWNuynxVNJ6jAR4vJx5+/OqURXrNZPCVWrAgmCVu0SIst0cW+NG4sfwRyuPgfyhI4L3wBF7gQbegDdITsZqzW/e7WLaTPn3+WvubN40MPUbdf+8p/BoliLezmf/AFR2AQT6O0DK59e1FfZKVi8XjY1/SJX00l7qlvYXBIwRSpKkhSvGFatxapPZSJDBoUlDA/LvtGBMIfThB9rIDjsiE2VlRHXq+kAc0if376I09wIh6hHcnB+VqQwvfQjzdgIxtgM/f/lSYv/pNPil69Zs3MieL8KRZ+RTsdo7oC+6NmAMFnImCv8ecGCsQI6Jmzly2xMHie6jhIE7x8CaOlnyjdL+/CbJqRQRO8bF77H6aUraaNZdQo7WGMi5MD2rfXiMvo0bJUzWouiiKG6LwUqvD5tHQkgDgyXHed5tF1773yTlyM0pMjR8r5v/hCfnu9cn0HDBChKKCqLA5QVZH8A0JZHsvR5ocBrPF/hjKArTkdVxitqBlAKNLSQlLgfPGFSE+zZumLr9jtVDdv4fvvCy3cuTOkg6QkIdiHD/PQg6/wPvzA5/F/HI2XaEcKm+IPpkIk1BrYRzGGZvA7dGM7/EIzPCyPEzy2PbPb6dmzpN2qGT/NSOOSvt9z6+yDbG9d7Df4CaFxIFkbb6DEY6h+NpADpnNnNsfSEAk13e+mKauTkrYUHhrvN+Z17CjEwm4XiSuAEyfkYW7YkGrn+9gF0/z59vXSeoD4d8YM/oW6eiJ03XUyxmrVIrpDNsEKHcHW+vXSijSWwWnumuaPwv7hB40ghscK9OnDZWjmrxesja0+wmwxoa1tWyEs4SoUi4Xs21cC0IAQVZ/W7EjmHNzJj/Es9+JKqk6XVoUtGi+Z1FTurNFed1oHUlgFR1keJ7gCt4nKIxBEeMstctzu3RIQ1amTpHnIiaG53dHVgQ7F4sWy2vKrHM+hJOfgLi0gMZxh/vFH/hMWRot//hHbncVy8TOyRgufTyT/fNQizw8D+BpADwBbAdTyewaNzem4wmiFxgD69BHi16lTnnVsJMVzJvRhttmEyOSAO0qvpwle0ae2+k50z2fPcldMIzbAFjbEej6CifwA/TQDJhC5SHByMmtdkZaJwAAq62AnH8bkkOpXQtQ8CPHTtlq1Kl9//y0GYv8SdCNuZF3sYANs4YwSTwZjC+xIFf1xIHFYtWpC/F0uKXUZjq5dg+fbinq0B+0q1I1XgZclcJafog+341oOwCguRXONUDz6qC5txfczQ43oDOlP+63AyyHNlsgBmzZpnkUdOujHuGcPW2BJpjGNQ1iVq4Bk3LmzSLOJiWLXqVVLVi1ms1yL7t1Jp5MZMNOCtAjjlDoBbiSyEdbSggw2sm1lUqu7o4p7OHDPCyF2hQDjz/CXuCRbY5EEDm7fLnaZ8BXF2rV6lVh4i4sTIuly5b5C1113kQBTYecY9GEsztOOVJbDCaYgwjndbnETDV8NLFsmQXVz5+bu/NkhVN2VU/6pSxj5YQAuAG/76wNs8H935HRcYbRCYQApKZrE5nBIwpS8wuvVV9+6886olrTdq68M5piZ1suvgpk+nV0wg4HAsDicCXpoBNtdd+mlgoQEqpWrBAushBNABT4eMV/BgRgRjAxui9/0fVasKCqLM2dE5RKJKLhcpKJwDJ5lc+UPzrfdm3mfbt04/7XlfP89VSvOQgqD9e+zGjf7xxrusaP/VOBjKZyl1OpN5nFUlD7sdvEw8UMuffZ2BSeS+Ye5pRCZX34Rwjdhgrac++ILkZS7dOGbQz10KKk0ISOo+lLg4xm/u+YfuJ2f42km2+P092HdOo2olCol0r//Oj6AbyPMM/xeiS0gxpScbc2UANLTyTjz+aCB3ooUxuEU7Uq6f85JHFnmveyFm+RkvW8+IEFQ1auLZDxwoKiH8uK9M2kSabXycYwPWX2SZmQE009kajabPtbh5Ze1/7KrJJZbHDyoGYYtFnLjxoLpt5gh36kgAJQAEBvt/oXRCoUBqCrZyJ+qtkaN/JeDS00VyTIX/awZt5ldlRn8HL2pQpFoWSBExy4vTC3s0l6CyZMzd7RqFfc4r+P1CHUp1STCJspqUlGYBhtbYRHdSOSneEYLemnfXl4Ci0WiM3fsEELpJ16HUZV/4HaqLVpq4yhTRv43mXSqj+VDF9Dlkve4deuQMV64wHiUZg9MpjMYAawn0uEqITMygnESDqTw0A33asbrkJwyegag7+8abOVkdOco9Oc0dKEHZk3679ZNI47+jJgjlUF+Lxxvpv7qYwsXoC0Dxvg6CLNJbNgQZJKsWVMS//lXUeXwTwTCrx+rlOFMYqmYDB4/TvGTz8b99eBBPbN8By8HE9XZzen8bvA2qr6QOI+skJERVFMRkDG3aiWrGJNJVDgnT+ZNPfPhh2yB33XzrIIjPIuwBIXt28vz9NhjerfH0ESGNlv+9fTjx4sA0aCBMLnAyiOKFfuliPysABoD2AbgkL9tAdAop+MKoxWaCigtTVQpoXnto4TP4+NPA1Zxwc2D5WFauDC6A8ePJ2+7jQeGfhMMpmoMKZPIlkJgVYg7oR2pdCGR9+EH+T+rFBBpabzFtYWAjxaksVJMAquWOMfa1VLYLe43pkKM1QvRxh8cJoTjZ6WT5AUYMUIIoNOplaI8fZrcto1LrW2DTKV8qTT2sX3JU+4awaIuwVa5Mvnaazr1us4zdvRoPoqJYTEPGgH8HU2DQUmhxKIjZvNOzOHEci+J+mLwYLEv+CXvVavIzIRfvr+L//INDNH11wQr2AKL2BAbuAPXSDqA22+XFRDAcjgZJpVrfZbFKfbD+8HfJmRkvieTJ4u6YvduId5+BvAfjAsbXzgTIK1I57foJoFh5UMqkT37bMTbvniBlzazlnrZjAzeiuWMwxk+hm+o3na7MJFWrYQp9e0b8dnh118H5x+xWSwiKLhcsnKKBqoqBl6AO1GHFXEsOH8TPGyCVfpzZJUl9MUXhRE5HJmTA+YF1appRH/YMLGzPPpo8fD8KQTkhwFsBdAs5HdTwwisYXjr3+lGIl1I4hT0EINSTjh1KuhfvszcMugVEod4ecnC8uVsQEN+h25MjSlLLlvGs2ezTsXSvvzGTIbVqjii6+9vVNIZgq1IEx1v6dKSC/3rr0WdtXixbKtThy/cuVdHtCzI4H2lFpOffKJ/gd1uknL4Cy8IL9Nlvp46lT0xUUeMQ4lsWZwI+sPria+P76EfgxKgyUT26hXstmWzDOrplXZsB8xjZLdLaQE12AmU59uWIZxvuYd9TZ/4g6R8tCA9eE0V+Dgbd3Em7g8eXx0HRd+fVZbMiRODJ/bAzAEYETY/n39loLn71sNWdsdkVsQxTkF3Ob5q1YjSd8rw0WykbMx0TQfibd6EdXwLrwnhDHirKUrmlUCPHtnHPJjN+mx5Awfm/JyTTL/rPq7DTVyJJvwF7fkWXmNo5HfFgJdXoPXvn3VnJ07kz0to0iR5P7t1k+SHLpfct0AOp38x8sMAMtX/LahcQADGAzgFYHs0+xdHBtC16kpxCUUGByr/0xUZyRIXLohLnMlEHxQ+FTOVtZQ9/Kn2K5Jsa+5cWQIPHy71BgKRixUqcPOAb4MxTJky6iYn82zVBmyJxcEgokALj0Rt7U8xDUiErO5FD3ju3Oov0GK1cveAr2gxef2Sm5cWpLMLZmZ2G8ypLKGqctsrE6kn7uHEPvRTI9jtMT8zYfKjpC1Zt2/o96o4HIHwa78fh0TQNsJ6WpBBlzWD6x8Yyf2oyWOoxFm4N8iky+MEabVyYvXX6VLECH4dNst1mzNHqruFR+muXh2MKH4bA+kKpqmQ8/8fnudLGJ1pXKH7BI3/Dkfmoi9t2jANNuqD6lS/TYB0Iplb0UBb3bVsmfm++COzs2yVK4u6z2wWJh+Frlw9d543Y43fWcBHFxJZB1qUtgXpnIqH6IOS6X4WCgIBbYH60Bs3+qP0/Dh+XFyMW7SIqrzmkiXyirz+er6CdC8K8pIKoqG/fQRgHICWAFoA+AzAB1kdl5sGoLn/HJcsA9iz4CBvdP7FZu6NPPHG59Gn8d20SZPKTSaNmm/ZIi+pySSpGlRVwuj9L8i76E+rRaTZgCcfT5wga9eWfeLimKCU1OlbFfgkRXLIC70J17MUTtOOFA7FEB5DSM6bSpWESQ0bpuXPXbGC60b8xiXWdvyfMoD9lI8Y76wiCdD8qqPVjZ5j/Xoqu3TJwgQybhxptdIHhY5gsRAfY3E2AiPQE0Ir0rkRN+iJUkxMsOsGlU9lwTxU/oB7/dK8yoo4yk/Q26+CEuN6crP2pNXKK5WDQfqwcNqZ4P1RAQ7FMDaN3cw/rn2ahHi0dMF0XofNXIvGsnIbNEgIdEyMzuc3qU9/Pq98omO6oeqeTbghJPNpoOkN+Trvr1DpOyWFNJmYBKdfraet6gLeVRZkyP2127UU1n366EshrlunJUELJ/42m7bd7c7RvnXwIHn99WT9+pkZu6jWNFuHDWlsh5AcRoWJpk21GtvhS+jVqzMJSTSZmL5yPXeOXUpPYuY5lyunXZKVKwt36PlFXhjAkmza71kdl9sGoMalzADyhTphUaXXXCPh54EHsXRpqav7yy88jxL8Ft25CK1Y2p5Iq1XlzJn+fp5+WuvDYgmm2u2DMbwS+/it5VHN0OUnaGVxiqEEx4IMTkF3jkVv/oo75KUPeMns3cvPPvPzAruXc4atl/8CT72qknPmsF6powRIl1PNbKMOYWIjMSCEAKbyZ9ztj9L1sRmWsBKOMQbnaEY67Ujml3icf6K+RgTj4iStwHJ/6cQzZ9jjlj1hxCagZ07jDtThXfiJQAYBLxthNW3+1NtWk4drbZJuYoNyE9tdf5JDB6SKRHdDGMPJqimKiIGB3EpOpy7j6LCyn9CC0OpYGmFvhYX8HS3p8JfJlBZICS5ziEECJyk9NWIcKn37fGSVKhxpHUI70qjAx9qm3bylyhFdf6mw8yiqcBiG8hf4g78aN9bfo9RUvRdb4Hwhz04wZUY2CHhVA6TdlE4TvIxDPEvjFJtgJWtiN+NwOsRbTQ3ap3xvva2tCEym/DtlhCI5WSKg9+/P/N+IEfrSmhB1XQ0coBke1le2UfXq1Wb16mm+CKHZpIsj8u0FVFgtJwYAoLff/XRD9erVC+ny5BMXLkiN30gPVlZITBSf7HC96333SURmy5ZUrTbuRi3GI46VcIRmpNGFCzymVGXaAz20voYP1+c4cTrFha9bN3qe6sO04/Hi4eQv1q4CtAbrAwQIJlkJx4IJvhaijZRX9CPgHKIoZP8X0/XBOt99R1qt7IIZdCGJTnM6V86N18+3e/fg+EQPLMTNhaSIRNUHhQMxnCa/ayOgshoOMgNmjsAgTkEPqmUkl8t7riEhHkUBb5gMAj6a4KENaTqbR/gK4QO8KFKx2UyvuwTnl+vJbaN/EeabE/E3mSStQUqKGH4tFiGigdQaqsq3rp3KyBlPfXTjAm/HkrBxaZ8BNY7V7KO6dFkwdcGu+fvZrvQGPn3NMqbvP8pHbt1Lk+Kjxezjw908tEPLh1QfW6gCrIsdNMPjVwnVF2Lev7/YpEhRi8TGZj1Xq5Xs1y94S/cvOcxl3+zntm16FciXX+q9h2ORwJ74xp/mROJGHEgKOgJUw0FuwnVchSb+KmwXuAENuRdXsUXVvezWLcQ/Y9MmIdY7dkT/rkWD/fslPXrIe7QaTXT3In7TId0hJ06ItjQghxRn5IsBALgLwKsAhgZaNMdF2felvwJo2FDWgbGx0WUO/O03WY47nUJkA8trp1NysZBMWv8X++F9v/tjqCuij+vRSK+j9HqldOI11+gYyo6uwyTZpcXLObb7ZXvp0n4GEC6RagTIiSSOtzwlxmg/1q+X9+PKGir3XdFa5nvlleI18dZbpEkyWA7GW6yLHbwCB/h82e/oPXpcOvCnQciAmeVwgoFVx+fozQrKCdaMPckDipSoOo6KtOlUItoYQw3cI82DufH+tyIQz4D7aySiykzbF6K1RBibTPwPxmm1ec0hRepDVx9+IuGFib3xORsqm7jC1kqM5v7kYkFs2sSFtjuDxE/Oq7fPZDU2CzKowEMr0ljbqS9mc1useHs5kcyPe6ymVREjuAkeTus5T8fwumMSa2G3P6eUZIJdiVu1RHbt24v9IqyUJwMSf8D75tdfg3Nb/N4mOvzPkMXkzWS7nTpVurPZ1DB7lNRQsPvtAqHXIA6ngqetiL95B36hoqi02/2ZxhMS5LkzmUSfX9BpI1RVu3fbt3PffO2amZFBnycfQaJFjPwYgccCmATgKIBhfpfQr3M6Ltr2r2AAft9xut1Zl1IMRUgkrE7n2rMnqar8c72HTmdWxItMh0mvBti5UxtDSH/DzMOpKHJsaywiAfoa38IhGBaB+GifzbGEaf/NwsvjzBlNSrJaheHFx5O33spzKEETNG8cB5I53dqDqV9NCeaB+Qn36M5l9ecSUuDjoAZzSICVQmrI6pt+vLdWPMAVVzwUYV812Kfo3b0hbqd6BmFFGn9HCx7EFSTABtjqZ4LJ/NL8tH4A33wjhkL/tZ6Nu4PG4Ro4IFHTIVgwagObxWziYLzFCvib19kStAAAIABJREFUNqQEbRGZU2Drmdd9mEmb/9pYkc41Ve7T9X1/5VV+9+Akzhy4gTGmZJrhYQkksJrrdKbrEPgehzP8H16h+sijWpbCVq3Id9/NpAKh0ykqydDndepUkuTQ5r8HiSMggc/h2LhRTFtOh9isbEjjkxjHDbiR9bElS8YXYGTd8W0wVGPGDEoMQoBJ2WxZ5oPKE6ZMEebetKnYQ/r2JRcs4JQ7JvA+53yuHPJLwZ2rCJAvN9CwzxgAy3M6Ltr2r2AA48eLl8Tjj0eXSmL+/IB4pM83v3UrOX06h5nezIKoSeuPd/WBQX/9pTGAkICZtU1eoMuaQTtSOQE96YXJv6LIivAI8amEwyyHE7SYvBGLZ516rD832prQ1+tJTWJq0YJdMU3Xpw1p/qCmZP6BpqTJxP+hf5aMJwYXOAqvZDN3vRpl+aJUcv589sTXfpWP+JY3wlrWxQ4OwzD2UsbrchaFM4lQabw2dnBRzL2s6jzN5o41TJj0kz4AqXVrub9ffUXabNyEG+hEMh1IZkv8Ll4yIQhN+lYOJ/kZeoesBMhqOMhOmBU8fyD+wYo09sR4lkI8TfDQiST+/YAQpAAS/0nkh52X8odXpP7Elslb+HbZD7i96dMsjTMR7qv8/q91jDafBx8Ub7O9e4UA3nKL2J3C01uHCikOB0lR/1Q2/UMTvDSbfJw0KfNzcugQeW/rBHY2/cTO+IHf4UERQqDo9P+iCkxk6RC7VHUcZAJi+MW75/jjjyGLqgkTJPYkq5KleUXonAP6K4tFe69stoI930VGfhjAWv/nGgCVAdgB7MvpuGgagO8A/APAA+AYgCez27/YMoC84Nw5ydr2xBOin582TSQah4PT8UDYy6tvfU1jdDlwSIqk9thj5ObN8ub5FZPxDdvyb1QiAT6IqVkQwcjEAsicHuXgQdF0uVySMTeA3fYGfruCdnzTYPI4shfGk04nm2KZ7nw2pOly42SOlI18DaxI58ofT/CpTseD28qYznKOcg99DhcXoB3duODPhRNJ1cII21R+Y+utZ+JHjujT8R4+LAzXn9Z6ee0n+LnpWSZYSoukvHlz8NBq5r+D/bvMqXwVI4PePgo8nAPJkTPQHxdgRgYr4DjL4gRN/mtihofNsJS7UJvX40/eVumAaBmz8Tvsbf7Kv3pQaUI6LUinE0ksi5M8aQ9JvRAI0W7TRlavgejlzEsvrUXIl5PVUDp0IE2Kjw6k8FM8wwxYSLebXpjCVGI+XostjEcp/ohOXILmWiK8rVuznGeBItQWF8oMzGa5LlkFqF0iyA8DGAKgFIAHAJzwE+y3cjquMNq/igFEwtKl3Gmpr0v/EOnz9N5z5OuvM7laHW5qP5BTx6fS5RKvhKBn35EjmerZlkSoq2SkpmcANlMGn3pKP8SZM7UI31BaILUStGPN8PB7U1c6/EXtFzruJmvV4o+m+2hDGkvhLNfiJh6IvY4j6k6m0ymZSUtnGmP4KiV7hlUfW/k8Pg4p3BPO6MLnq+9vOZrqg61UVSKE7XbxbVy7VojBzTeLtXPECB6Na8Djzpqk3c7V1mZsZ17ESvibN8Yd4G2VD9CqZNBtS6fTIhK+E0lcdcfQ4EAcQYO8GlLKUpubDWmsib0MJIwb2HiRZnz2eCTAqXJlqedctSo9zlh+gj7Bal9mZHACHmGsJZk2k4cz8ICc+9FHxTZTsmRWD4S0wCpVUTILHtlAYssk/1JVHKECnxSYB7gAbdk8aPyWua9AE/15TSYJSrwYGDVK5mc2a95cgKTymDyZ+oRWlx4KxAvIL/2XzM0xBdn+9Qzg3DkOiXmf+qhV7QUxwcOedVZT7fMs02DjVdjrN6iJrtjlEm0USdFHh7xM51GCsTiXBTEMbPOxNE5T8X+uGLstk3SXkCD2UptNS6e+Y0dmQm1FCpeiOeMRJ7WMA0ZySB3cYOUlp5Pp67dwYa+ptOjUU9nTpEirAylr+Buvxu6IDC37frwchVfEhzEcXq94nyxfLswgsBq4915ORxc6kEInkjkWT9EdkmLDjAy+iSF8Fy/TiSQqUGlSxHC7116PBHgMlf31aCOtUgKqoYAnlI8uawanokfw2nHLFn2e/ZA2HINZD9s4GT34NgYF1S51K57TCwcVKmQv+Q8ZIoFid90l9XSjjHpKTBTTwiv9VbrdMq8SOB/sNw1W3f2Zg47685rN4il3MTB2rKb6qVNHW/WFV/jLyJAx5YIRFgfk1wvoNkhK6J6BFs1xBd3+FQzg5MlsnYaXL0ylSYlUI9fnf4F9bGFbyT24KqQYi0qXU6XbHeIdF2LQO4yqISmgIzMAM9J5FfYE9zHBw7n1Xua+jTkHtj37bGSGcg22aRtbt5Z8+aE7KgrTq13FupY9IZXQciL+ekZj8vv2B66RE4lhHk6RVwqRvruQxMTZv0tlm1Djx9mzEvXjdouLp9sdVBNIxlY53hLmampBKj/C8ywbDH7y8TrbX1yO25noKMvmMRv8doJIeZF8rIctfAUjg/3FIoGLF/kLhLhcwuSTkkR373ZnIuIemPkhXmA5/M3wZwnwshoO8oDf+E2zWSvSHnJ/2KOHLC0Dz5PbTc6eHdWjfvasOIiNHUuWLe2lU0llT0wI9q8CrI6DfhXVBa5FI/pCb3ZBZfyMBocOSd6l2FhxM3W55Fq0bavto6qy8nM6ZZVwCeUNyo8KaDKAVf4I4DH+9nFOxxVGu+QZwObN8mA5HFKSMAK+7LkszGedmQiEDanchVq8Gz9TgY9Pxkzj+q8267Lnsnt3kWIcDg5/4mAWBFD7bcp0To1gXFPq7yyn5PUK/YkkcTcIpEgoXVqWCwMHapIVQPbty33zd+sYWbRSvx1JHI7XOdn+BMuXCq8BHImBRiL6mZnASfjDO51OTZ+/Zo2m97LZpCBKq1YkwPswMwsGo7ID5mZS5218+H3SauWkMv3oMqUw/FoHViNuJPJhTAiuKEzI4NC2/sC7jAxxsQmoqk6dEuPo6tU6B/w+GJMFA9S+xwYkcrtdXJLnzZP+nn5a8kKF1vEF5PnNJiV08ulkjumylD8NWsM779Ryx82o8Dx3mq4NBvOpAFfiVm5BfT6PD4OprONwhmtwE/+HV3kANeTgfBRCiRoDBsizqihiy5k8WVY+p09r+yQkFEh5xqJAfhjATgBKTvtdjHbJM4APP9SMTVkU7X6m/rJsX1iRVC/QjUTei1laMRe7XSJTRoyQJf1LL5G7dpHjxnFjTHNmDkTKTFSzI5KbNkmckKpSgmZWrSJVlT166Pez+I2cJni5rtydkiwlNMPkqFGi1923j/R46Pt9KduYf8+CUGXNAIQo+vjiMynBCH9FIc1IZ2OsoqwKwmMBsmMq8hnc6HCIsZcUgtu2rdy7N9+UJDA1anA3agXrBASODySycyORVpO+1oEJHiZv2kX6fCILmCQVgt5Pnsx870WPfgA1Mucc2LVLfOKtVvFOuvZaEuBYPB0SMxF+7bRzWZCuzTk8x1C3bnrXUEWRZzgbFVC3aivpQApdSGLdqheoKELDp7ieIgFuV+qzKo4wBgl0ItnPIPXPZiBK24J0XoBLbC2FjYkTg4Z9Dh+ubVdVyURaty75008MPvAdO+aveNRFRn4YwEwAlXLa72K0S54BHDkieXayqRT23adnQohKJClW1QX0bEBIhsatW7UX1m4PerCMwXNZEP2sfuvPF6Mk0unw0eUiR/Y5or0or77KShUjMQ6Vz+NjpsHGT/E0Y3GOZXGS21CP/7TszqZ1TrLhjSr3tu7Nvc4GbGtazGvMuyLMNaux6cdps6ls04Z+1RlDDKnZMZLMfVmRykVoLRsaN9a5XQaRkUGWLMm/UEfnthhQrXwQ9yanx/Ri/3t3B/s2I4NxOMMZpq5y7bZtI0nu+vx3zsGdbIS12Vx/+d0mULinZEn9eD74QDPSNmokBMrtZlP8kQUzCTDqdLqQyPHm/2gMb9Mmfd+BEpGKIuqQKVNk1ZENA2haYjMDAWpvd17He++VYGPPoqVkixbsVeXXMIIfKWBPe/4fwAy5ZhUrFn6+hd9+EyLv80n1sZdekmSMocXiyYuzIilg5CUX0BwAs/25f84B+M3/ezaA2VkdV5jtkmcApLw82UgOfftmJgA2pLANfqMbCayDnWyM1XTjAsvgtBhYAXlB09Ikb0tMjFS1Sk4OVu6KnIogJz256LWbY0nQgNjx2kPBYJwTdVuwFE5HYFQq+1i+YDMs1QVgdcU0vmoeTbO/wtaDlh9YM8TukLOUHlmlY4LHnzZf/quMYyGql+gZgDBXD2thJ8fgWaaZnKIKCL1fXbqQJhMbYLOO8Ae+m+ChC0n8r/n/6HLqVVFOJAkxmTBB3KnmzCHdblbA8RzmK7/fQz8m/j971xkmRbG135o8s4FlWXLOOSkGVBAlqggGMGPGAKKYUAxXv2tGUUQBkasgKIhXREW5ShITKhIlI1EyS9q8O6Hf70f1hJ7pCZvYBfp9nnpmuruq+lRPzzlVp06oonU2i1gBFBWREyfyBctzUX5fRRVKRcyFS74rfusX/66+H/PmyQlLz55S7dO9e9BxLPwd9nrJl1/mpv6P8ZLUFbyj+U8szNJulP7zD/mgdSKdyKNVdWILOpOFPsvgc7sJM4IPZfjwkvzjio8//tD65wBSJdi+/cm5fzmgJALg4lglWrvyLKeFAIiCP/+Ue2116lD9U3hpQT7tqoXJj6lXcNzNy1nfeoD98RUXoCczUU3+hCaTjAbq80k70DlzAiEpNl96P53IC9HxhzKY0JWGllEIeFXVhI/98C07YC0b1Pdx5aJjZNu29KRlsJFtT6CtC1m0qnpcF3J4DFVUd//g/R6rPZ0fmW+nC7lMMufzie6/UmvxFM6UozFtrTA7B7/y/PN8NJlIi/BwuuNutk7bV4z+9Bnv9ZglmeOttwZ/KDVtYg9NdqvgcxTqZv19mMRvhs+nxRIcnwleqVN/6ikpCFRBWhP7YtChPbZbPPx9aViAtPx8jZmi+6zzeD5+jtOnwgPhSdltNm1K1MLCYEjw8Hq7ZWiKrCy5oPFN+UAySYtFBo4Kw5EjcvGShGw2xVauRXt6YOaDeDPi9wRIAQ/tKOBAfCH9BwDp63IycM45kWOeMkUbPfUUQ2mtgGoBGADgSgC1EmlTHuV0FgD+jVS7nWzf1iuZJHJYV+zluhfmsH374J8jCTn8EgO0L6jJJD2RJ0/WLNE/n3gojNEHl95m3c3XIAPsgNVsj7XchJaBDWW2akUeOsSB/cNz+foCKhgn8vgnzuZYjKQDebSikG3wFx/GG9x41s1ckDaYX7V4jCe+/z0sQbq+yiuc7pohWaUAhS/Vn8h9+2ScssmTyXHXLI3QzccWAvr3boxt8qLq/UpSzorr1OGJS6/m4P75HGD5hnbkBRLYX2L5kZdiEfc7m/D4ml2sm3KCUPNFTGr6uuzjhhtIIWT6T0DjMRx7FSBLE/xN33ldZVgKt1tmR7vrLnLHDv56zRvsiJVRxqz9nVehg7aCEDIEuB+//BLc/A4tnTqRHg8PHyYzMhQ6neQN524LCoCrrop4v/32D1IQejga/w5x0gsfq0+1qJICY2HSADUY0ElCaPRTQI7nrbdkvC2NpcWpg9LsAdwN4B8A0wB8BJkW8s547cqjnM4C4K67gmkAGtb1qLNp0gIPd6w6TrMp+EdxIo8b0UpWrllT/oxpafLY5ZLp/0jS7eac81+j3kw/XQ3IFmsmfBU+D5zIhYs/ohuz7Rn0jJ+o88eVWgSbTaHDVMgu+IPJyKIZRbwSc+lUN/aq4zAVi1xef1V/OB1WD6Nb7ShE1I3M4PeXXwjqZOcNnx92PRYT1GO2weM38aA80Mny9tGw39jCvpPPml/ir+jKsebHucjSh//BHbwf79AMN5ORxS8HTAkwusubb5WNd+zgp42foBWFYUxQj+lHqtcAhcPwLnfc/RI/H/odcxwZpMXCyTWfpVPjAKc/Lj+T3YLmkQ/C/+6QclVRo4bWvPSqqwI68AUPzmOyaqWUmuKTgQyHDw9GFw3Brl3S9SAlhWHe2fGEsU++6yHZ38oVX3yh3fh2uWR0Xn9QvO7dTw4dZYzSCIAtAKqFHFcDsCVeu/Iop7MAcLuln420MlPYNOUQXcjliA4/kCNG8FZMo4CXXfEr10ON+tm8uVwyVK8u/11+k4uFC6WVTePG7IjVOoxPoT0iIXsk82mAHfTCRK/Jyhb2XUxCDhtiFwt/W0VEzLAVXtU7m2/Vfo0D8YWqxgoyMX9Y42RkS1NAk4kzHXeoDEtvf0KP+UWOA/BRWR0Mv3BXm2VxBEA0YRDJZG/CdHlx6tRA/xu++pv1EFR9meHhZktbLrRdHrKa0a6M/McW4SUp+WebxvlhNMSfsYd+vwSLmeIoYpLdzbPFCtJq5e1Vv4zzLEP79nEY3mEenPwSV3IK7pS5kQEZ3nv7drmSzMqSJqeTJ0vmGLK6zKvdlJ2xkiZ4+eL1sUM2tG8ffLcTF8ok4OEvuEA3BEW5INxXxR8J1W4/YwXAMgC2kGMbgGXx2pVHOZ0FQFaWVA07nXLVPWZMyMU6MluXEu1fYrHIcAB33ilz9PrD5gIcgC9DHJSCf7gmph0xmJUsAj6+jeEcivcC9Wwo5Or2Q9gVv+gwZx/ft9zP1lgX4r3q1+n6eKX4mt+LvlwrOtLb8Sx6YGEKToQJi3DGpXdOfregkLNxLTfX6xlwgFvwdX6cGaa+WkXvvMOfKvPiiwM/RXXT4Qha/kFdXoXPo4whWO/ixrs4d67cX7RbvQHbd31aogkmeW7ev1cEtDNOq5scNYqbfz2ihoWOJkTDBY6ibkDL707kcj1aB5KzsGZN3dl8APffTzqd9KWmxbWJb9o0GrOPTWst7OM/qMdsc5pkwHqWWWWJWbMig+H5VwL33ntGqoCmA1gN4Hk1HPQqVR30CIBH4rUvy3JaCYBNm+RmU9++zN9/nHVqeZnk8LBpEx+//TbM0u7uuzUv4x7U5bX4L4fhHflnFYKsX19aYpAy3Z26jM2Fi22wTvPnugafsTU26DCXyD+gA7mayI1nYTldyNGEGA5drocKD3tAty0FwIDLCmmFmw5TIa/sU8AZA//LSMYUjenJ83bk8Sp8zt8sF5AOB+fgGjpFPp0OH2fMkCupext/T0QEgUtkVaC9V2AFoCZBefrCJWH9eNkWf1EBeBM+1lxzalZY8v7K8RMh6mXZXl/FpX0OtkC2MLk5P7beWPp8cm+6dm1tuJzGDcN9CmKNObyeDAFigZvvQk3rpRfm0w9FkTvACWyOPvNM6AogtBTpnAvWew93BowL0nCUq9GxBH+2YiLc+Q2Q8U9OYZRGADwXq8RrX5bltBIAl6lxT6xWbh0+Tk2uIRnlsWPS5Hr5cgZzBIe8jP3xNU3w0oF8voPh8rzFEsxHrCgyiJW6amiI7SGM2Ms8OHQYYqxPydxdgbAF+sxKaI69IYIn/I8vUwW++a/jOtE6ZWmMLaqJaGiic8mk/FnL1jboz/tMk+k3Hbz5Bi8v73qMkYwtkRLJDAlIVdvx4yTJhqbdDI7VR6tZJjf5HedyG5oE/AKScILz0E/j4DUJd5N16jDZpbchr8/89H8HhQ2xI/AabZy3jVu+U4/nzOGG2j1V66tYq5Fo53yqSo5sjO1yFVmcLHfRsGkTp3Z5VyfIoZ4PQPg7pTUTvQ1TS09PPEib4mC5886yzT1QASiTYHAVXU4rAfDII3JZ6XJRefU1DjZ9TjsK+AjGcsCAwCUu//AvzrNcxSXoIXfQMjI4xPYpbSigy1zIGf0+lnHon3022Pc//5A2Gz0wsyt+Dftj+bgXteMwm3DGGP5dn3kIzXUP22J1VMYzAuOYm1KL0QLf2VTroaRAonNtHRdy+DFu4nq0ZS3sZ3Uc5sSHNsZh/HoMUE8IKLQjnx/jRrJePR7972J+85WXF9TfFbgeGte/Kf7mjZihWWmZ4KYTOXQgl78LaeKVnVRLFQryXik4pnrrFsdLW5ZG9T2cfs/PdCKPTuTzyyd/C2zWdo0wAY0mALT3MUEaH9hRwGfxf6Uze3S7pbf4kSNk27b0QfAuTI7B7BMZu5czz3699P+9ePBnewdkiIjTAKVxBNMt0dqVZzmtBIDHI/WN330nVTcDBsgZ14svslkz+cskWQo4GJ8yCTl0IZef3vw1qSjMeeltvtryA3747A59p0yPh2zblhutHUJCDcs/VG98RwXg4PSFqq1/+BJcnyECXtrMXprN+n9aB/LYEDvi/MGD5wW87ILfNVml/Nc6YmXANLIJ/uYDeCsQ1sCCQlZDJi/GEuaaU6X/Q8OGPH7bQ2wZUGvFYv6x1C2RArEBtjMVx2hBEU3CP2slAQ/NJmnzrx/Kwf9c8rlNNCOtVhZWqcEqOE4ripiEbI0QKT5DVNi/zp/0C99h9b4kAT6A8arKLvjbtcJ6mnU3qIN9mtRNesDLnWgghUmrVtKYoCTo21e+z+npPN6+G3NMqSyAnY2xLe674b9mDtAkz43GC2S3bqX62yWE1NQgUZ9+Wv73OwkwHMFOIfz4I9mqlcLBmM2BmEO/XfRTFyyRAb/8BtWxLCOKilj4yedsKTbTopokjhMjSbudykXdeLHrtziMP/zYx85YQZelQHPNjnzWwAE2xA41j28sZhY5C3UiJ2QTWGErbOSu5j35aOpkdsdSPomXNIwsHyHZxtPSpPfr9u08q/rusNg3se4fa5WgJxj0Z9NNxLaImPah9QS8fAxjqHTqLDcPr7uOW9Ccr+NRrkebQNygUCGrT6f83ti0I8DcneYi/l5rIDOQyZriIF+q+TbHYURAcIaamLoiwoBHFyxmeFhkUp+xEDIUQkmgqi2/sF1Pu8VLl6WIy88dzhNI5QWaFUp0gW1GEZOQxYbYzp+hOqRZLOUfm79btyAhjRpJR7tTHGWmAgJQH8DjxW1XFuVMEQAB9OzJTdb27IC1vDB5DQ+uOyzj/fgdBlq2jNtF0cT/cNuAh+lbu056B6em8pizDovvgRutjla9ZEOBmncgWgpG7acVRdw+/A3OvfojnnDWCsadsdvJBx7goLq/Buq6kKNPmMPBqs6CKDSFM9RYjCeWINTvI2jtFDlOJ45TwMMHze/wiwYPcV21i0mA0zCEKThBW9hGsTOQD0F7vxQc5S/oSjfMPF61Mf878mfm1FdNNpOS+EbnGfSrkcy6K7pYM27t9STk0G1PlqaPSUnSBr4kePVV0m7nZem/B+73+L0nmCEOq17pPvV9yQt7V/R/u//gjiDhLpc2J3ZZ47PPgr4PDsfJ80AuR5TWE7g6gGEAfgawHcAbibQr63LGCQCvV8Z6KQhz/f/2W2lWobrjB3DwoPQOjYb33iNNJioAIzdXozH8+DM1fxHw8SU8wRsxnfEcuKrgEOtiN60oYq8m2+kuUqQTUqjTkRCchiG0wE0BL9/CQ/qECsGvrphMl6mAkcwvnHknKtz0GJF2v8KBPE3oblOcGbwZRbwJH0V5pkEhGt72BkxnE2yjGR5OMI+Qv2WvXlJQCsGe+D4OA402Rq0QqIV9nII7qZgtUg3SpUupNz+//FLuo7tc5BtvaO9XB3sizIWjCd9a2Mf9UKPKmkzS6aw8ceWVkmink1y9unzvdRJQEhVQCoDb1CBwOwGMBbA3Wv2TUc44AVAcLFwoZysOR9BhJzOT9PmoKGT2B58F/Aiuw6w4zMD/vYiAl2YUMTWgRoj1Z1V4Gb6M8afWZ8hJ5nx/CmOZcEWteAA1A3sYycgOZhELLWaz3LSz2TjRNFyXgeqvPmIxHD2mr2XsnfEHD5trsWPSVqaZTnDYJRv4zL2HYvQT+/gi/EAH8nV8A3y8D+8GVDv1xT/yOY0bFwhY9ihe0R1jZDjoaMJNO95UHOMSXMyvLFfzzVtWMiurdK9m9pI/mdf5Ai7oM4ZWUzB5j/47py+4BTwcgo/khbQ0OTEqTxQVkXPnnrycxOWMkgiAAgA/AujmzwcAYEe0+iejnPEC4OOP5abnE09EhuS9//7gP/z662VYYLOZStt2vKyvlyb42Avfq2n4ojG90PNay5T78U5I5E89Rh7aJtasM1yF4mG6OCrzHJPkqFGBpBv7UDsgAJzIowdm7kNtnoC6SdewobR4Use+CS3CaIg10461GogurGRxs5v4ib1sS1W1DZkmTrBv7dU694rFbGU5hHRuQ5OQrGjB9oMwi7vQgFVwnE7kcZiYyDm1hnH/y1MDevYDqBmSBS14j/BgfPFXAuF1PLRZfbz88uK/qsuXywCiXi9lDmWAiiuJ/7njF94hpgZiREUXAPJ7Bg7SiVw6kcdRGR9Iq6JTMBxzRaMkAmAkgN8BrAPwFICmhgCoQChKMESt0xmIKR/An3/K0L5Op9xFVlUph50NaFEtd6wo4j7UZo2A92cijC9cKIRqafRVHdHb+7+HhoieJaOadukiuYbHI9VbN95INmnCOY4beZ3jK/6Ei/ge7qEdhXQgnxfiJ3590yw59rfeYhaSmYajMWgJ3/MobgkXdjJUtn+jVSZg17tHsL4JXjpEITtY1wfO3YXJJMBlOD/M5l0+p5m4jgR4HFW4CS3YCNuZhBymO3KZY0vndNzCtvhLM1Yz3PwCA3hNRLYyvd8l/pg7dy7eq7pkSTBlxLBhlOlAzWb5/v78M3dOnM8FSVfpeJN7iMD+gDxXB3s5FbdyEu5lUTv9JEoG4qM0jmBNVAGwDkAhgCcAtIjXrjzKGS0ASJmgw+mUjP7QocjrXq9koJmZMnm5xUJfy9Y8/1wf7SYZM8YLE6/B7BhMIPp5IRQ6HPJPLXOjh89s42XgCmeiZA0c4C/oykuxkCMsE+hZvirIPQYOlONav57f1L+PVcVRzf1sKOSEsfmcMnI978EkHeYfS+VTPOavb0apqHsA/rHH7iMdh/g3GvNwtVa8BdN5I2awDvYwA4f5u/kC2kJ70x0pAAAgAElEQVSiswp4WAXHuAd1uQ2NuRbtWAB7wCvbbnZznbljIGhgKG01sZ8EmA87++IbnecR73f31/WyiWknB9T8jdsW70r4NX3nnWBmyvPOo8wnIARpt3Pi4CW0o4BO5PJxvEp9j21FQ8MsXC87a9SIpNT+TJwoF38GEkOZWAEBaAfgJQDbitOurMoZLQAKCmTI0HPOkTN8HSgKOe2tY3xePCcTxVSrRno89HrJHcsO0DP4RirVMkLMD2MzrHDGkW7N4tLPDgbIqZHst1rxEfCEMclEVC3ae7iQy9lV7w1yjyZNSEqrP7sojGhjgodmeOhAnjoD16e79DN+Lx2qxY4V+QzOUmVdm8bDNXafVhSxEDIt6MN4IxC2ume1VYEQyP77NsLffB2PBI6HYTxfa/UBa6a7+VDv9SxMra5G49Q+7/swgQT4IkbrqFmi/SaRz82KQjrVUNftHYln4zpxguzRg2zWTKZT5hVXkBYLP7HexmRrUMhdgoU8C8tj0CdpGe/3djebeeKE9Id0OOTWjz/6iYHYMDyBT3WMGyfferNZN+EGSU56MZNOq5tWFHEAvlR/XhXt2zMPTrbTqAvC/2zhzCF4zgQPBXwU8HHUKKmK9UfNtcDDH9C9BExXey8H8qXHs80mo4ctWEDOnMkTz76h2QtYIi5ln9praTYFVxxyJuyjCV7Wwy62wEaepUm1WBqBoGWwX+MynoM/aIKX12MWH8VrEXWij1XhkdrtSIDzcIXqyZvHMaMy6VJDZocy4tQQtVaGI1sG+qtRg0xO5oqm19Fuj1yVfIDbSIBP4QV1b8BH/d81tuBrhG10IZcmeHmWc2Px39m8PLkXdcEFzLlxKK1mrYqrCo7zCNJZX9eBUB6b4eFxBB2z/tnp9efRocUSaSBnQB/RBIAFBk4NpKQAQgBmM5CWFnF5wnU/YuR/u8ILCwQUZCMV6Nw5WOHoUfyO87ELjQAIAFQ/hVpBhPVIzTUF5sD3994DXnkFaN0a2LEDaGA+iME5s8P6YUhf4X2HgxBQ8AqexCXiR6BWfWDbNuCbb4C770aK24spjU/g510NcAc/wPkjzsOGxm3wwyMKfCpdHlhRDUeQjRQcRk3ca5uK9923htESj45IuvTaCQj8gfOQjVSYoKAqjoTViXYfwo4CzLNegz0YiEzUwEL0QqopH/mbBuP2Om3xz34TvsGV8P9GzbAdq9AFAPHQTZnAviIgOxsoLMTB3DxYkoCiIqE+Q8AkiAc4EZmogfF4CB7YdeiI9RyC78ZuNMYU3I3V4myM+r5//MelGSqBK68EfvgBIGGzJcOVbEZuLqH4CCuKcCNmohqOYRtaoB72IBM10BFrcSXmYTEuhQNujMIYpCFb9tmlC+o3MuPf/wamTwceewxwOIpHloEw6EmFk1UA9IPMN7ANwJPx6p+JK4C8PPLyy8kmTRQuefhraf+ckxNRr2vyXwRksLd2NQ9x52ZtPlb++Se/TrslYoaZmKrGf1227VBLppssLJR5xI/cPEKnvd7mY/TZvxmFdCGXf5tbsjC1Oj1HTsh8vE4nm6l5g1OQxWwkk6mpdAsbMwJJ2WU/DtUiJ7gqCA+1UJKZv/a4IXYyz5IqVylOJ2dXGRqjb719CKlS8n9vg794EDXpRB7N8DADh/gcnmV1HGLv5tt55AhZ8PHnzHLUIJOSWPTKWP7R91lmJdeh95nnOGKEJCX8nl3wB4u/8a19F0wokhdstuJ7w/71l5aw227j5s3ka6+Ry78+wB3mZprw5j71czU60ok8JiGHgzBbS+Bp4JBVUUBl8wQGYIZ0KmsCmWNgLYA2sdqciQJgxoxg5IdYjr9zn/yNNhQyQxzR3bDzeEibJZpdeCJMMsjMbCjk/tUHA33/9/HfGMnsowkZPcEgj5OQw5fxJO0oYKoph+tWFPJoRgtN+/nmKwK5dDuL1ZprfrWM/9ipqi9Kp/rR0v2UbYzUOyxaxGOrdvLb0T9FGZfes9N/HttSOgVUXHYU8IRIY0fzOlrg5vjmb0ult2p61S1jA5OSZBjonBwy7833GGpTXw2HWRd7+CDGRTD04go8O/KDeaf/+IO84AJy6FAZ6C0eDh+Wxgo2m1RZhetq3n03cKPOWKHSnskZuEmNOks2w9YgMUJIR0YDJUKpBADKwRMYQFcA34ccjwYwOlabM1EA/PlnMEnMDTfErusp8NDn8eleK9q5TzVZjGS8erO/WDPZNBxnXqb0EHW7yeqO41HaB1cN8RhPKo7xDnzAPvhOPefjU3cdpOeKgUxGFv16/uz1u2UgfIeDm9EiJC+BvFdoXCGrZmO4JAIg8jm5kE1OmsRNm2T6y5LvewQ/3bBwIu5jdyzl/NTr+VXd+5msMsHqOBRo7GvQiEKounHVqnIIPgqYoiYjK5BAfRU6JfTc4wluKwq4DY25FzIpES2WxAOkbdlCTpsmrdLCkZdHCsEiWDTPYhYG8zJ8ywbYyQUIyc7ldJIrViR2XwMRKLYAQDl7AgMYBOA/IcdDALyrU+8eACsArGjQoEE5P6bKidWryTlzpHNiibF1K8fhIZ0/eSzmFMoUvAR87J+8hGvr9KX33Ulk06a8pcb/YjCaWGogrVB5Hs9wBm7mk/Wm044CukQ+ly3Op1JYxMNX38O5rUczc5Vq96co5JYtzM+oH0g1GU57M2xmNWSGRDstqQDQMkYBH/nhhxw+PNbziiZ0IsffGcu1la6/nnuW72cVHKcLubxNTAtE1aTDwScsb9BsUgK+GAL+5DNqDP9q1WRScyHYXJP0p2QCz7+SsqOAb2KkPPnTT6V/qY8dC1gRpKpWaWZ4eAgZVCAtpGphH8fgUbni+/vv0t/zDEZJBEC5egInKgBCy5m4AigruN1ksyqHiIjQxZHMODrj8rGLWE4L3KyPf3gUVdkX86O0SYTRuOlAHu3IY2M11o3T7uNHD63kkZ3ZPHRIJjozm8l//UtOPAOahGXLOPfS8bSopqACblpRyKo4ws5YwbtNH/Db5ME6SddLKgCoCgCF9Pl4yy3xnlV8AbAdanowiyXg/UwhyHnzeHTbMa7+dDN923ZI3ffZZ8vrVivzXhjLOnXIpCSFnVoX8ubu/7Bt03z+OH6NNI6HNAEt/bglnRZVPdUBa6RKp6wwaRLzajdlJ6wk4ONovEgC3Nn6MjV5jEwgn1+nqdxwMlBilEQAlKsnsKECOrn4+bkFjB4ALpoAiK6/Txa5/Mo+mEdQlZGRP8PbRmOOCqviCM/C74H6Jnj5tuURsl8/fvgh6bAH9dt2q5eX9fawT9U/2Bjb2Q5rCZBm4eVjpjc4U9zEPd1vZJIp1D+hrNQ/8vvAbkf400/kkCuO6I4n/mog+Bx74Xv9G4e85ydOkAcWbwj6RqSmktu388TaXfwluS8LrClyZ5VyUn12/YN0ILcMxh5J7yTcI/VOWxL3CQhAUcg335RmoZs3yyxrjzzCBeil+jKQqThBAsxp1onpOMIkZLMhdtJnc5A7dsS/h4GoKPEeAMrJExiABcAOAI1DNoHbxmpjCICS4bPPSKsmMXw0Jq/HuCJVRgI+1kjK4V09d7A6DujWia1qis5k62G3tPQRgn9vVSI2cTOS8+lS0xbWxl6a4aYJXi7o+zo5ZAgnjMkNC1VRWgYY2o+Xj9ScTrs1POpnccYb2qeXv+E8novfOAzvBoPdvfUWSXLNGqn9sZs9nII75bVeveSP+p//BFOFNm9Okhw/noGZc3QBVNwxBz9n4Ca5WlmypPgv4aJF0ppBCOngp2a0P4wMVkMmXcjl9ZgVIGAfavO/uJZHUZUcNCgy9pWBYqFMrIBQxp7AAC4HsFXdWH46Xn1DAJQM116bCLOP9l3L2Ezw8iPcwhm4hfoz/1j3is90GmAnU3GC87u/QpIaq55Uax7feXRnwEywHdYE7lO/ZiH3/nWUdpsar0j1Tk5s3NGYn3YsqThKpygoQfvoz7sO/iEgLaC+Rn+ySpXA7/bKK5LfAuQFpmVSF/7JJ/Linj1kRoasMH48N2zwW4v5x1oWqx/t+Fehg6TBp29oEBM//SQJNJlkIiN/XCuAJ5DKtWhPH0JCgVetSrZvL8dpoNQoiQqoGYALdc5fBKBZtHblWQwBUDIsXUo67EqIxUwsJq/HBILXWmM9CfAVjNJpk8j3aP3LT3+sm86dJJO5o+lSmuFhE2xj/uyvSZKrZ27kFxe9wevNnwVoSLIWcg3aBzxXU3EixOqpJMw6/rMoHhPVa6vwIiyhA/l0IJe/4Vy52fv88+RFF3HzXWNYpYpCq9nLz7q+yWDMbBVuN/2xmseMCQqLVFNW2OZ3SYSxdgwWFHAd2kqTzpJixgx6briFN1f9lo2wg3NN15AjR5L9+5NPPimj3VatKs1NT4MsXJUJJREA3wBor3O+PYB50dqVZzEEQAmxahWn1H6GFl0z0HhMQp6TIRZ2cym6kwAvxaIofcXTgcdjjjIz1lPOsVLv+/zz3IvaHI9hfA2j+Fuj67gcXUiAh1GNtbBXXSV4aIKHY/EQn8XzbK9JSJ/I/RNh5ImoVGI9U71rXqbjAIUqtDaiVTDcalIS3S+/zjynaovftCkzM/VztXfooO2zdOof7XjSsZ/DME46bk2eXKpXcWHd2wK5DTJSC2WK00GDyOnTZYWjR+Vq4csv5aAee8xQ/5QBSiIA/oxxbV20a+VZDAFQQnTvzkXokQDTjzz2hznunrqK3tbt5BLe5eKHYzJVm3S/usGt009JZsoKx2MYFQhy9Gjytdd4F6Zo7PudyOU4jKAXIiSAmmxbBUf5Nh4IBFlLjAnGWrnIIuBjFRyjiGlWGk8Qxhcwj2FMMN+B00lOmRLQnX9TZ2gg50+oGt7tDu9Lb+zF/U1UEgJpKhUOx3gZjLAU2NZ5EF3IZRJy2L3DseCyBaD30//KCG9JScFAU0lJUkgYKBVKIgD+jnHNiAZ6KuGBB9gP34YxA58OcyCBokBWKjOKmIwsbkZzOfsTgvktOvLzlzZz00+HOaHHbNbCPpUJhzIWvaxciTIduc/wH9wuE4msW8dLsTCir/74mt+jp07fobH59e4fby8gnIn7r/toRQEdlvCop4kIgMTHbjW5mSJyuLnHveS8efL3mz6dvP9+3nBFdqDuffdpf+LatUOFcbQVQLyx6wkSbfv3XCPZL2M5P77/58j3LBFkZnLDw1P46ROrmLv3OCkEFYBXYw4BcrDp82CICIdDCj/DAqjUKIkAmAVgqM75uwHMjtauPIshAIqHQ4fIZcvIXyauYTV7tuYP7kAuO2J1GKNQKFDENlgXOJ+EHC5BDx5BOvPhYG/TIiYhhy7k0RYRi15vhZE48wttWwMH+G7dl/nd5F3cam3DethNAR/NcDMFx7kcXfgUXozC4BQOwYdRzCHj0RZrNeD/Hh5jp6S6dm3/JpMS4HsTJ0b+nkuWBHli+KT4kktIs8kXsvKJlnKxOHSFj8cXWHHZUcB9Kw+U/iV95RXudzRWrdRIi/DwsKO+zAw3aVL5Jn8/g1ASAVATwDIAS1Uv4LGqY9hvAGpFa1eexRAAiWPvXmlQYrN4aYInJC68LKmqp6lZaNUFZhQFdLQCPl6FORyHEbQJN6uIE6yGwwHBYDVpN1kTzzMQTwAoTMNRWkxeOp0Kf3hqAb033kKX3RNgPp+KG7j+sQ91BI+cud6MGWyBTVHuV1L9v575Z2mKltHaRBFNwscaNaInOykokD5RR4+SQ4aQt98ufQUOHiRHjlRY3XY8jNbw+8SiJdqqQZYUkc0UZBHw0Yk8HtmqsxlRAni9ZLt2UuPVsWPJjIwMxEZp/AAuATBCLZfGq1+exRAAiWP+fBmLy/8n9s+eo//Z1dkncpiKLDrNRQG77HZYp17L51C8x/rYzUH4jIvnHOdtt8kc3VJ1U6TR1ZecMQZn7QI+fvIJ+fnnDNlzkCUjOV/nPgozcIif4VpGT9GYqIBSKE1JfTwXy9gieU8Upqm38inZ6udzXEXfD/oJf0IxbJi0pLRa5T4pSc4atTIGDYmuePSKh92sy/jP5jz+9fkWju66hL++V/Jk6T6fVGG1aCHzrpNSqK1bV8pwJwaiIpoAMCEKhBAOIcRIANcCcAOYRHJJtPoGKhd69AA6dgQcdgU1xBE0wXbUxz8AFATzAFDTRsADxeRAkybEF8+sxMwmzwJt22K4eRKscMOBAmxCGzhRiEEXHkDrFj5MmwZ07SpbK7DiJsxEZ/wZ0qv2HpEIv86Io8GDgYwMwCQUDbVHch1qDQZq21GAHZfcjfrYC8CnueZvp20DnTrB60/hVdyY9A2m2YfhuryPIJ+f/3poPoXQvsPPJwJZfxkugik3O27tqlVlagiLRX4HgKwV28LGoR1L9Psy7DicdjOWebpg3OiDaH9tC7y87BJccG/7uDRGw7JlwIwZwNatwO23y3N2O9CuHWCzlbhbAyWBnlSQAgOzAXwM4F4AXwIYF63uySrGCqCE2LSJdDo5HOPVDFHhs0Q5y7seH9Nslrrom8RMjsVIDjHN4DZXex5HFb6D4XSY/Xp/OUs3mxWedx5psypMQg7HYiS7IDw8dKKz4MhZqNVUxE3Ls/jBB3qz1cj67bGao/EiPTBxWCAkslRtRVeFaPuaj958Gv/H93Gn6nXsow351IbSiDbLL8nqJ6i6mtDsTRm7Ow6KiqTD8DvvBKMzF3Y4h3diClthA2fgBvbEd1H8ARKhL7JO80YJhIFOADt3yn0Ml4s8y8jzflKAEuwBrAv5bgGwKlrdk1UMAVAKvPIK+1f5kUIoNJvJLl3IVEehhjmY4JWhp10KbxXTmIQcCnjZqcZe0mrl0vSr6bS6VTWPlln0vLCAWy2tQkIRlEQFpK+OaOLaH2ItqKdy0btGNsQ2rkVbvomRfBWPRmkXfs6nWkyRvfFdQADEbhudaSY+bh+TcIK3YCp3J7Uu2W/8+uv0wsQNaMW1aM9cSxWNJ3Vi+yF6qiyZEvSTJ9aU2ev411/k1KkyJJCB8kdJBMCqWMcVUQwBUDps3y6tRQYNImcHki1pGdiE17L4RZ9J/B696EIOTfCwS+pmcv9+sqiIf1z7Gh/C2Agmce655J83vxl2PhHGF18HXdd2MGAWHp0R6zOzzviTScjRMVUNb+dndF41v7DMRzsZd7GzJnF5vD5KIgC0NDTADs6b6+HOncX7fXdP+jYQWhnw8SG8ybRATuFE9z4in78VBVxnah8IOlcmmD1bWimcey6ZnV12/RrQRUkEgA9AtlpyAHhDvmdHa1eexRAAZYfXXotkPK2wkYrTRVavTgXgf3AHR2Ac/3G2IJcvlw2feILzxeVqmAWZgKWLfS2HDpXZKmXWsVgCIB4jDR5bUUABHy3mWEw22saltl878hhpxRPZnwletsZ6OpHHjq4t9D0xmt9ZrwhLPBM+htIwf712PiZZCpicHN0aSA/vnv1hyIyfrIojrI5DOs8pnGYpMIK5FXwcgDlMR2bgem+xgDxQBmaffjRqJIlMSjJSPZ4ERBMAUTeBSZpJpqolhaQl5HtqWe9FGDh5OLQ+E/UOr0Tnjl5UqSLw1KNueM7uik1og788rfFDg9sAYcJdmIrxtlGon3wcaN5cNn7uOTxU9SMosMAONyaYHkRBWm188AHw4ovAVddYAE1C+XBES0IfeeyBHYSA1ycghF4d6vSpd18FaTihcz00gT1hM/uQbsvFNZiD39EVy7uOhOnVl/H7099A/lXCaS/uRq8eIsdkgRd5XpntfOvWxHvqeU42bHDDP56B5vm4CTNhRZHuvYS6oW2CgkH4HEnIg/+3m4eBaG7ZBf8zPp/LgKFDizu46LjkEiApSX7v3Lns+jVQPOhJhcpajBVA6ZG9L5vVxBEmI4fNrTuo+NQ4K3PncqGln4y06fLxmaEHZdz2ZctkoPkQXN/3GJOQQyfy2My6QzOjTLbomWaWtARnrcFIl3oezNFWBsFjmWc22uxf4d9oojmejz7S62r9ek6YEJ++shmv7NOibtSbTGSzZsUwjfR4eGLiJ1z49A/8calCxeGkAvAd3Bdl9h/c9+iOJbRAuyfUCuv4Ap4KhqLu3LnsXkSfT8b82b277Po0EBUoq6TwFVkMAVB6bF2wM5CA3AQvi3KC3OW1V320WiUD6NEjeh/uh0fxSwzgIlwSojYgE2OGsa5HY1BKmA4/1j5AOB0KrcgPaxPZTy/M17R/Bs9LAbBtW8h+SWQRItZYE2f62u9ezfn+/eP8qJs3kzVryuTxoekaZYQ4NsT2GM89XJ0VFLAWuKkIU7DymrLbBCYpmf+UKYYQOAkwBIABkqTiU3hP6x9ZTRzhK31/0Fw7eFBO8urVixN/a9s20mymArB3tRWqMPExMgZQSRmhlllXwXFGZ/bRBI//nI8X44fAsSUimJvCVthAW1gilaNIk7HvGzfmAzdmMhbdZZN2Uo922ec118T5UZ98MiiJVGmhKOSy+z7iFtFSNQWN/5zNKKQLubwJM1gb+zgJ98p+Z82ScUXKEoWFMvS10yk/jZSP5QpDABiQKCoizz9f/rFHjix5Px4PefQoFYU8coS0WUvL/PWYuY/v4W7mw85qIlNnthpNGGi/21DABtjBKjjGORjINESmcwyuMBTeY5rCELMjru79OFNS5KHdTr7xhiaIpY5QKem4w8/5ePbZMlRCTCxcKBmpP3ooyWefJV12D53IowvZOkxfJsyxqoHzbMhnO6zlSnSOJK5r15K/J9Fw9GgwKYzVqh/j2kCZIZoAiLoJbOA0xdq1wF9/yb/2O++UvB+LBUhPhxDAzJmA2+PfYC3txmjoxqzAo3gT83EFjjMt5Lz/Uwk5Dm8fhBt2PIFXkIocvIh/YY35HNyCjwAQNhTBDC+s8GDZf/dj/Tpg8hfV5Yan0wm4XOh0ZX2cOAF4vcATTwBPPSUfn0Pu08ILaynHHD5uWdo2LcKKFdLjNyZ69QI2bgRWrwbuvhsAsHgxkF9kQQGcyEcyIje/BQALHChCEvLhhhNHkYGzsDqyf4ullGPTQXo6MGaMNC64+mrgq68An6/s72MgNvSkQmUtxgqgDJCdLXU8Tid5+eWl7u7BB2PNYEuzApD9RcYWimfOqN9naHawC/ALCdANC3dZm/HFFh/xx/FruHixTFd75ZUy6BqXLyd/+UUzXr/1YtmX0HH5aEYRb8THZL9+JfpdfviBIQ572mdkRhFbYQO7YDnnox9743tm4DA/xWB94my2Ur8nUfF//xdcvYwZU373OcMBQwVkIIC8PHLDhgR0C/GhddDSY2rh16Ix62gbu3qMPl7/sWiR3/+NZ+SB0ylVKJQWN4A0TZ81S3+8b78dVLfLz3jCp3TC4DU8WuLf5qKUVbrP8GWMkvl3rdagLksIrkIn3o4P+RmuDRJjtcoUjeWFe++VNFgs5MMPl999znAYAsBAmePYsbJigtGZfvsa+4shHGIxf21bAR/H2J4if/89MJ5BgyTzdzpjh6Ffu1aGL27UqCz2PGI9D4WNsa3Ev4/7aDZbYqNm/AJubkRLyXhD9jlYtSrTcVTKROTx79tfJMePJ7//Xl0OlRMOHyavuko+fGMfoNwQTQCUg3LPwJmC77+XevCCgnDHKCL2XkD49dB20FzbdLg6tBErw9sy7LreNYTVAQgT3q7yLzx+nj1w7pNPgK+/Bpo1Azp1ik796NHAhg3qDKpMnMFCQc3n0HPWAmhaop5ykIJtplaA4h+7glo4jIb4B/juO7nHkZsrK6enAycgxQEEkJMD/H1Ihuv0b3aUB6pXB+bOLb/+DcSEsQlsoMQ47zzA6/aHXGbIlXhMMXzDMxTac9XSPHCiAFZ4kIJsnb6jMf/wa/LzKbwAAR/M8ODCHtrNW5sNGDQoNvMHgJo1VT5ZYuavN+ZIuJCLJ8UY4H//A5KTgSZNgP374/Z+4AAwbBgwdSrQsJGAHUVwIR/TcSs22zvDhQIgMxNF//cq2Ks3cNFFwBdfYPGsTNzZ/BfMOO9dNPtmHDB5MvD00yUco4FTAnrLgspaDBVQ5YI7L1qAtUTORVPl+NgUmwk1gc11mEk78inUpCzQTTgTTf0TrppSQpyiFN56s35447w8csIE8uuv9cedm6un7ioLVVDkuGZjMFm3rjxhs5Fvvhn3d7nkEqndcbnIGTPIRbfP4Al7DanbUtU+94jJBBR27RoMJ01SeuiOHi2d4Ox28pFHSvBmGKhsgGEGaqAskXswF8s/2qRzJdQU1D+bT8Q81D8LNuEQagEAfLDiS1yNIjhAmLAc56MVtqANNsS4Z/h57ex6NxoH6k//xIKvv45sddddwKOPAjfcAHz7beT1pCSgabXjYX2XVhUU/pykyuYY0gGrFXC5pDnmRRcVq1eHA+g59RZU2bAMeP995NnTcQ3+i/c5FIDA2tU+bKzWDahWDfjtN5mV5bXXgA4dgMcfB154oZTjMlCZYewBGCg2jmw5ijatFeSwGSzwhNjBRwvwFk1XLhmoDQVwwwW/wMhFCqR2knDDrunjbzSHD+awPsL1/OFqIX0VCwAsWAAMGKA9t2cPUFgoVeTRNC73PZmOxx+P3m/xod27EFBwB6biDkwFJswF6taVevr69eP29PHHwL//LU3sr7lGPdm0KbByJabZ7sH/8vsDAEwmoKblKJrnrARQALz8MvDPP4CiSH+RP/4ow/EZqIyokBWAEGKwEGKDEEIRQnSpCBoMlByrvtiFAjpQCJfK/PVSTDKkREdTbIUbToQy8pbYCisKQ2oF+2+A3Yi9otCLDBqeqjFIV716kT1MmQJ07w5cdx0wZIj2Wna2LNOmuMNalZUwkP2kIBvv4X7Y4QaefVbm90yA+QNAnTrAe+/JVYzJ/w/fuBG4/no0OrEaJvjgdApcfTWw8b2f4XJCLhVuuEHm3rTZ5OANnP7Q0wuVdwHQGkBLAEsBdEm0nbEHUDmQeyiXjcy7Kc0p9eLgxNP5B9VU+b8AABonSURBVOsEHb381308ijTWwd4w3b7C6tjP5/EUEbhnpI4/cV27TBiflJT4uBctkqpxh4OsWiWchuLq+mPXEfBxK5oFdf+lgaKQkyYFOv8evTn1A18wyujff8v4TqQMFbJ7t2xj4LQBKpMZKMlNACBEWZvQGTgZyFGSYGnsArYJBLeRYun7Q9UyWvWMdhuKuA0fIRU52I86Ef1kohaex4thfUHnM1z9E1011bFjgu+gouCjaUBhoaTXYQmf8YePTe9+ocehdIV/B9pjHZpghzzVt29iNEbD0KHAhx8GDvtcVxW4M+S5N2sW/G6zAQ0alO5+Bk4ZVPpNYCHEPUKIFUKIFZmZmRVNzhmPw4eBli2B7dvD1SpAW6wPqamnAtIz2RQQIJrgb0zDbZiKO2EyCTiQH9Y2MZ1+JMPXE0ayfTKysWhRjK78WLcOSE/HHbMvg8Pmg9MJvNV/Cc5DqI48nrAJpyH6uZ9wEdagE8xQgNdfh+5OdXEwZ46c+/sxc2bp+jNw2qDcBIAQYpEQYr1OGVicfki+T7ILyS7Vq1cvL3INJIgNG6SPkOQn/kxdkrlsQHsAhAm+kCxU4YhkyJ2xArvQGM/iBSzGpWhm3Y1CuALX9VcW8WbbWl2/Xh0LPBiYtBBNLbuxZKxOEDQ/pk8HsrJwiWcBMnveiMOHgdvfORu1XSc049D2H8/yKZx+eexELrphmWx5883AY48Bbjewa5eWiRcHo0aptxDA2WcnEF3OwJmCchMAJHuRbKdTviqvexoof+zfr+VDQ4cKnHOOUNV5sigAPLAjumrG/10er0cHKLBgP+qgNxZiZ1Ft6DP7UIaeCDPUY8JBmk6gGhayJ3b4GuK+0WmIiiuuAJxOfGG5DvdlvYb164EPJxbiq/zeOv2Gji8WXaHjCK4evLBiNxpIJt2okXS3btlSmvT06RN7uNEwejRw9CiwbBnw668l68PAaQnDDNRAsdC4sTQYcbvl9wkT5MTy+eeBV18FvF4BwAx99Y3/u4AZRQBMUCCQiixkB0w+TbptzjlbwRWtt+OV2U1R5AnvXw966phwyOtO5KFj8nbA10B/dtyjB/b8vAs3X1gdhcsEvrgUoLsGgnmCw1coevsD2nsGv2tXKWYoEKAMjfzSS9IHwOORFxctAtasie+qrIf0dOD884vfzsBpjYoyA71aCLEXQFcA3wohvq8IOgwUHxdcAHz+uczj3bAhsG+fNDVs0gR48EHJZyT0VDRBBumDHT5YQJiRh2S8iqdxWZPN8DNEGfZBtrfCi3feERj1fgs0amIO6au0oRgIQMHLeAqfHL8iptOTqFkDqr4LJpNMpaLty/89fI8jmiDSW5koaIEtyMCR4Gk/8wekI9iWLVFpNGCguKgQAUByLsl6JO0ka5IspZmDgZMBRQGefFIy+rVrgaVLZeycK6+UsWfGjQOOHdNrGc4MtaohQiDdVQDRqg3MJsmUe4ql+BhDcDW+wDRxO9J4HDVqRON/iVjehKuOgoz6RTyLFtiCxYtl0hc91Ksn91KHDgV+/BGwO2RbK8L9AfToCB13JK0meNXzJmxGa3yFkG0yl0tmoWnYELjsMmBgsbbQDBiIDT3b0MpaDD+AisX//ift5gEZBtpuJy3mWGGYtbbvVhTSHEifKIsJHk5r/gIbZ2TRjgKa4OU5HQq543+byT595I0uv5wvvajQZAr2VbrQ0/qxhExC4TnnyHA48fDUEx6a4WE97NbtS/+esWiRxYZCrm11HXneeUHbfAMGSgkYsYAMlBbVqkm2ZbHIiJgDriQsPr+1j4AQREqKVAfNmCE9UoMQMFmtaIFtcCEPThTgVkzD2pY34Latz+DAURuK4IAdRfjPlV+hYZ+WeKjldzinkwc/jfoG/S4TcDoToTLa3kCoakZffaRQYPVq4MiRiEsReOlVC47ba2Mj2oTdM95msN4mtmxzTo3dWL2S6LBpNvD77zJ8gwED5QhDABhIGOecI9UgffsCWVnAN98CaTiKVGTBAg+SXQr++APYvh245Rbg9deJoK8f4fYQm9AK+UiGCQoG2eej3YAmAICZPT9EB7EOj5nHoX3vWli8GPjgQ4EVq8246WbgrKLfsGf5AVjM4Tr2cCR6PtSqKKii6ttXhqhPBCl9usJlJ2riMPSFgB6tQTWQGYqm3tUnpqFNdcPXxcDJgyEADBQL/frJmX1RkUwEcxB1kI0qaJW8DwcOCLRuDeko0KkTZt/8dcBktGeHTLiQD/nKKaiVVgDbv5+RkScBXH1pFtaKTvh31bcgWrdC7dpyz8HhAOoXbQd690Za58bw+oDoTF5v9h/NGim8jkCaowjz5gEJO6jPnAnz2DG4vl92WP/hq41wyHOhKdC74hc85nkFmD07wZsbMFB6GALAQLHxr39JayBp8SOZ3D5rIySlqK/TkiXA9u0YiXFwoABVqgAvvpWMFuadsMCDZtiOvTlpuPr5jlixUmWQEydKjl9QACxZgnbtgIULgTfeAOZbBgB5echxW6NQFDqTj2Z5FA3B2frlni8TZ/4AkJyMXVcMx/jvWkC7yRvutxAqGEKFg9+MFDgXK2B1mGUUumLC65Xm/YcOFbupgTMchgAwUGzUqwf8/LO0yKlXT4aPefvtkAqqt+klruXIHTIMx44B51/qwqp9NeG5/CrQlYwinxUmk8xeBQC47TbZkd0OdOsGALjwQmD4cKDqa08CTicyUR0W+M10wplptFAModCz01cAEA7ko5fve5kFqxjwm8H6+w1a9AjVy8E/zxdoat+DGjgYQrsXVuFFqtODK94bAGzdCpx7brHuDwCDB0u1XMuWCSUMM2AgCL2d4cpaDCugUwhHj5KrV+ua1Pz2mzRyGT6c9HpDLuzdK9Nx6WD6RwrtyKd+RrBErX/0jhVegkW8G5NZBCt5yy3FGqbPR7ZoEexTQGHjugUEFJrhYXcsYQ3sZzoy+WPy5VQGDORIvEETvKxuPkK7XbZNSip5AM60tGAf331Xsj4MnN6AYQVk4KQiPV16rJoiX7Hzz5dGLu++G+Z4W7eutHsPw9z3DuDW24giOACEOoIlot/XuxZUybTGehAmXIMvYatXEz/3eQEPPCCTY8VDzv4c9MpYia1bgyooAti5Tyax8cGMX3AxCAuOOBqge4uDEP+bj29xJRSYkeNzQVH3ga3RtFsJ4MUXZfKas84CLr645P0YOPMgyFh/nMqFLl26cMWKFRVNhoGTiO3bgebNvCD0ApjFU9iHqojC33MFchvWGrheX+zFIWs9uN0CLheQmakrjwBIK6hzWxzH1sNVoK9JDd7bYiFmvbQDD7xUGx1yl6GzsgLj8RCsDgvGT7Ji7VrgzjuB9u3jDMeAgRJCCLGSZETyLSMWkIFKja++gsr8Y8XZiRZ2IZY5pgnQxPIB9rAeTOoWg88HxJobzZgB7DrmZ/7hJqBawXNuJzcefqcpDmUDP5svxnmdi7B2KlC9gRVVq0a/hwED5Q1DBWSgUqNvX8Akwq1piuMDEKuNdmVgFsTttwsMHgzMmyeTv0dDmzaA2WqCw+5Dn46H8P34zUi1hucwkPf4baUd1aoBNosPhT4LXl99Keb+a6XB/A1UOAwBYKBSo21boFeynkI+UXtNvRg8oWaZwbhEteua8P77wGf3LELvI7OiBwYCcOmlMhbSp7PNmL+yFvqMaI1Hnk4KuV+QPrNZxkvqWXsjALmX8cy8rtiwIcEhGDBQTjBUQAYqPQ76aqjf9Lx540Xd9F+L1iZ4bd8+IO+zb5F6t5oQ/bffgPHjo9J17rnA7t0y9EVWlmTyobjgAhkyIzkZePxx4Hi/qmjdzY0COmCyCCxeLAWcAQMVBWMFYKDSYulS4OGHgb43VVPPxHLyihX9M3wvQDL+JORo+qxaFUjdv1nO/PPzgU2b4tI4Y4YUHFlZ0KSXNJuBjh2BL76QCcVSU4GGF9bDB584YLEAaVVNuOqquN0bMFCuMFYABiolDh+W0Y8LCwHAn2RAL9RC+MzffxyeGjK8DtAKW1AVx/ErLoIHVky46gfg7ruBxYslARrvNn107y791wDg9tuBuXOl7Bg5EhgxIrL+jdUXYVDDETCf1RGmWtMB2OLew4CB8oJhBmqg1DhwABg7Vm6M3nFHMWLpxMD+/TLjmFsTbj+U8fsQndHHUg0F33cTvLgUS/Ed+kKBGVabSQY5KiZ27pThjzp0SKByo0ZSb5SUJJcPV19d7PsZMFBcGGagBsoNgwdLxy67Xfpy9S2D9D45OXJm7Y6ab8WvvYy1B6B3LrghrMCKRbgUuUhBFWQDbTuXiNbGjeXnnDnAkCFAgwbAL78AGRk6ldu2lfGmFQVo1qxE9zNgoKxg7AEYKDWKiiQ/E6JEE+gIFBRIp6jcXAAghFCQ4kw081Z4ULhwE1J/XT9MgM0uFfY//VQqup95RtL+zz9SFaSLzz8HpkyRCdoNzy8DFQxjBWCg1Jg9W0YIbdcu0hKmJCgo0KbCJQVyCmyQap/wOUu4nX/4ZyxvYHn8bNcFGL+gTVCZX0zs2SPNQvfulasgkwno2jVKZacTuPHGEt3HgIGyhrECMFBqNGkCfPyxzBdcFvr/9HTJUAFAaJKmmEK+y6v6MfcTcRwLtnvvxzZ4/8Yf/DvOxcb06XIfoLBQ7gNs3SqFoQEDlR2GADBQKbF4sYzFU6emgkhzz1DnrlhhGPwCRC9BS7CtF2b8+eU+qcPJzy82rRdeKBcPLhcwaFB4KkwDBiovDAFgoNIiIwN48BF/mEy9mb6el68f/uicJjyHf6E1NqIBdsIifEhFVqC9y1SINtiMJ/GqzAWQmhrIUpYoevSQKv0nngD69ClWUwMGKhSGADBQqTFqFHD//bpRpRGZbUsiI6lAU6cx9mDjJ2uwe+bvcOcUYVHS1XAhH3YUYsa/d2H9PePRtFsdGf3N5wPGjSs2nSNHSrlx4YXA5s3Fbm7AQIXAEAAGKj0mTJDm8na7zBEcjJ0vULOmwFsv5KGJ8yCaO/fgm2lHkF7XH8NZrgLew31A9+7IqtIAfVOX4fa8CViAvjiAOrjm10flzH/WLNm53V6iTdqtW6X2SAhg166yGbcBA+UNwwrIQOWBosh8wnXrQmaXlxBC+ky9/77kz6++Kn2pAJkH99FnnVCQBBMUPP7QAWzNAvyqITM8qNNEOih8dNUS/KRchyI48CRexc/WS4MhP2vXlp0eOSIN+YuJDz8EHnpIJrvp1avUT8KAgZMCQwAYqDwYMQL46CMpCJYu1eTHdTpllIamTWXcnVAoMAMgLPAiJYkIqPgh0OcyG+jojPenAK0GtoR5pYIk5KJDewIDnwAefTTYUVJS7BjQMdCvn8yRbMDAqYQKEQBCiNcBXAnADWA7gDtInqgIWgxUIvz6K5CXJ7n9mjURCdL9QdektSZRI7UQh7MdAIA6VfMxuv96dLyrC7r3kPUtFmDlShnW57vvgGXLzsOCzsdxMNOMgUO6AZZuJ3V4BgxUNlTUHsBCAO1IdgCwFcDoCqLDQGXC2LHYnHERHq/yPr6vdlPE5ebNgWuvleGVB1+4X2X+0jLImZ6EETPOw+tjzQFfBJ8POHZMflcUmV7ywv5Vce0dqbAYa18DBipGAJBcQNKfbeN3APUqgg4DlQw9e6KH+WeMPXQLrh6SjB07tJeFkA5nOTlA62qZmmv+PYF584AaavoAUkZ2Npvl5623AmPGxKEhN1fewICBMwCVwQroTgD/i3ZRCHGPEGKFEGJFZmZmtGoGThMUFQVz8UYPBAfcO96fSUVWrlFDao4uuCA46/fDZJJ95ucD334b4+Y//yw7qlEDWLCgxGMwYOBUQbkJACHEIiHEep0yMKTO0wC8AD6J1g/J90l2IdmlevXq5UWugUqC+fOBgQOBd94BWrWKXm/DViv8jmBCCLz/PrBqFTBtmtZnID1dhmpo1AhISwOefz7GzWfOlIGICguBqVPLYjgGDFRqlJsmlGRMYzghxO0A+gPoyVMpKYGBckXXrsCXX8av53ZLfwCPR5qG9ugRNOD5+GPggw9ktOVRo4D69YEbbkjg5jfdJK2QSJnYwICB0xwVZQXUD8AoABeTLH7wFQNnPC7rR5xfYzvWH6yOF+7aB5utTeDaoEGy5OcDX30lXQo6dUqg027dpMkQCaSklB/xBgxUElTUHsC7AFIALBRCrBFCvFdBdBg4RbFs8jqs2lcbx31V8MCE1mjUSPpwheKaa4ChQ+W+wPr1CXacnGwwfwNnDCrKCqgZyfokO6nlvoqgw8Cpi1otq4Ah6R+zs2UWrlBs3CjdCkwmaQJqwIABLSqDFZABA8VGs54NsXTaLvRutQdWC5GcLAOxheKDD4CGDeXm78KFsa2KDBg4E2EIAAOnLM65rQ0WbGqAvfsEdu8Gwo3EevcGWraUSes//NAw7DFgIByGP6SBUx5+xy89VK0qQ0IIAVSpcvJoMmDgVIAhAAyc1pgyReZ7r1MHuP76iqbGgIHKBUMAGDitkZICjDYiTRkwoAtjD8CAAQMGzlAYAsCAAQMGzlAYAsCAAQMGzlAYAsCAAQMGzlAYAsCAAQMGzlAYAsCAAQMGzlAYAsCAAQMGzlCIUykUvxAiE8Duk3jLDABH4taqPDjV6AVOPZoNessfpxrNpwK9DUlGZNQ6pQTAyYYQYgXJLhVNR6I41egFTj2aDXrLH6cazacavaEwVEAGDBgwcIbCEAAGDBgwcIbCEACx8X5FE1BMnGr0AqcezQa95Y9TjeZTjd4AjD0AAwYMGDhDYawADBgwYOAMhSEADBgwYOAMhSEA4kAI8YIQ4i8hxBohxAIhRJ2KpikWhBCvCyE2qzTPFUKkVTRNsSCEGCyE2CCEUIQQldaUTgjRTwixRQixTQjxZEXTEw9CiA+FEIeFEOsrmpZEIISoL4T4QQixUX0fHqpomuJBCOEQQiwXQqxVaf6/iqapuDD2AOJACJFKMlv9/iCANiTvq2CyokII0QfAEpJeIcRrAEDyiQomKyqEEK0BKAAmA3iM5IoKJikCQggzgK0AegPYC+BPADeS3FihhMWAEKI7gFwA00m2q2h64kEIURtAbZKrhBApAFYCuKqSP2MBIIlkrhDCCuAXAA+R/L2CSUsYxgogDvzMX0USgEotMUkuIOlVD38HUK8i6YkHkptIbqloOuLgXADbSO4g6QbwKYCBFUxTTJD8CcCxiqYjUZA8QHKV+j0HwCYAdSuWqtigRK56aFVLpeYP4TAEQAIQQrwkhNgD4GYA/6poeoqBOwH8r6KJOA1QF8CekOO9qOTM6VSGEKIRgM4A/qhYSuJDCGEWQqwBcBjAQpKVnuZQGAIAgBBikRBivU4ZCAAknyZZH8AnAB6oWGrj06vWeRqAF5LmCkUi9BowAABCiGQAcwCMDFt9V0qQ9JHsBLnSPlcIUenVbaEwksIDINkrwaqfAJgP4LlyJCcu4tErhLgdQH8APVkJNnmK8XwrK/YBqB9yXE89Z6AMoerR5wD4hOQXFU1PcUDyhBDiBwD9AJwSG++AsQKICyFE85DDgQA2VxQtiUAI0Q/AKAD/397dhVhVhWEc/z8l6pjVhYjhRVlmwlTDSGVEBhpR1EViH4xeBEEUEgRedGFf0+AQXRQSNIQUChGh9sFQhCBBDkyQOYWmSRcVfYkYKChZMxX2drHWscOZ4zlnHGjPmf38YC5mnbU37z7M7Hevvdd+170R8UfR8UwTI8ASSVdKmgmsBT4sOKZpJT9Q3Qp8ExGbi46nFZLmV2bZSeogTRKY0ueHWp4F1ISk94GlpJkqPwHrI2LKXv1J+g6YBZzITXun+KylNcCrwHzgJHAgIu4qNqrxJN0DvAJcCGyLiBcKDqkhSduBlaRSxb8Cz0fE1kKDakDSCmAYOET6XwN4OiJ2FRdVY5K6gDdJfxMXAO9ExKZio5oYJwAzs5LyLSAzs5JyAjAzKyknADOzknICMDMrKScAM7OScgKwtiPpTK7O+rWkdyXNye2XSdoh6XtJX0raJemaqu02SBqTdGmT/Y/rJ2mlpI+abNe0T51thiZSBVVSd56SWu+zebmi5mlJAxOJw8rJCcDa0WhEdOcql38B6/OLRIPAUEQsjogbgKeABVXbrSO91HVfk/232q8I3UDdBACMAc8BT/5/4Vg7cwKwdjcMXA2sAv6OiC2VDyLiq4gYBpC0GJgLPEs6wdfVSj9JfZLekvSZpG8lPVr18VxJ7+U1Gd7OiQlJvZJG8qjl9Up79lDViGZ57n9Rrum/T9J+SavzW8ibgJ7cv6c6roj4PSI+JSUCs6acAKxtSZoB3E16e/Q6Ug35c1lLKuM8DCyVtGCS/bqA24FbgF79t1DQMmAD0AlcBdya2wci4qY8aukg1WqqmJMLij0ObMttz5DWdVhOSm4vkcoN9wI78whoZ4PjNWvKCcDaUUcuwfsF8DOphkwz64AdEfEPqeDYg5Ps90FEjEbEcWAPac0AgH0RcSRvfwBYlNtXSfpc0iFS4ri2al/b4WwN/0tyfZk7gY35OIeA2cDlLRynWctcDdTa0Wi+Yj5L0mHggXqdJV0PLAE+zndeZgI/AAPn0y+rraFS+f3PqrYzwAxJs4HXgBsj4hdJfaQTeqN9Cbi/drEcSTfXO0az8+ERgE0XnwCzJD1WaZDUJek20lV9X0Qsyj8LgYWSrqjZR6v9AFYrrQk7j1R0baRBbJWT/fFc7742UfXkeFcApyLiFLAbeKLqGcKy3Pc34OJGX4RZq5wAbFrI6x6sAe7I00APAy8Cx0j39QdrNhnM7dVa7QdwkHTrZy/QHxFHG8R2EniDVCd+N+OTxZik/cAW4JHc1k+6538wH0t/bt8DdNZ7CAwg6UdgM/CwpCOSOs8Vl5mrgZpNUL6FczoiXi46FrPJ8AjAzKykPAIwMyspjwDMzErKCcDMrKScAMzMSsoJwMyspJwAzMxK6l/Oxr3cyt6n2AAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [], + "needs_background": "light" + } + } + ] + } + ] +} \ No newline at end of file From ac9ab8754555747bd9030bd39d9d2a7e1f485b04 Mon Sep 17 00:00:00 2001 From: Bryan Kwong Date: Fri, 1 May 2020 12:54:54 -0700 Subject: [PATCH 2/2] Added Changes Requested PR #1 --- 07-Unsupervised-Learning/.DS_Store | Bin 0 -> 6148 bytes ...alizing_Higher_Dimensions-checkpoint.ipynb | 738 +++++++ .../7.3 Visualizing_Higher_Dimensions.ipynb | 738 +++++++ .../7_2_Visualizing_Higher_Dimensions.ipynb | 731 ------- 10-Textual-Data/10.5 Sentiment_Analysis.ipynb | 1700 +++++++++++++++++ 5 files changed, 3176 insertions(+), 731 deletions(-) create mode 100644 07-Unsupervised-Learning/.DS_Store create mode 100644 07-Unsupervised-Learning/.ipynb_checkpoints/7.3 Visualizing_Higher_Dimensions-checkpoint.ipynb create mode 100644 07-Unsupervised-Learning/7.3 Visualizing_Higher_Dimensions.ipynb delete mode 100644 07-Unsupervised-Learning/7_2_Visualizing_Higher_Dimensions.ipynb create mode 100644 10-Textual-Data/10.5 Sentiment_Analysis.ipynb diff --git a/07-Unsupervised-Learning/.DS_Store b/07-Unsupervised-Learning/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..8ada7b08d0c364897a146b305050c2a6823d3b28 GIT binary patch literal 6148 zcmeHKu};H441HHR6tQ%Jf#Eh-s8lg9utcRuAjE{!wX}fRpwd)?4lw0+_)2*8Sru*R ziV)b6?>Ua`yS$6y9DvZ7-?xD#fCg2tw$0%ik^7=+QZvsMF=&k|9N`)d7+|^=dg1+5XrTD^}hubya68JnX}+&tU+%4|($jXA@N75cObt~jl` zg-3cv%c~8$EBUfw7iYj3a0dPd1MJx%jh%!(Is?vtGq7Mlz7LTqm>E_H_0z#Yj{w9v z-72)@S5R`IVP;q*^_D6kaaF=a3Jq<= fjFncrM^&Leib2c_tAzAW{6|1(@WC1QQ3gH%DT_#? literal 0 HcmV?d00001 diff --git a/07-Unsupervised-Learning/.ipynb_checkpoints/7.3 Visualizing_Higher_Dimensions-checkpoint.ipynb b/07-Unsupervised-Learning/.ipynb_checkpoints/7.3 Visualizing_Higher_Dimensions-checkpoint.ipynb new file mode 100644 index 0000000..13d4466 --- /dev/null +++ b/07-Unsupervised-Learning/.ipynb_checkpoints/7.3 Visualizing_Higher_Dimensions-checkpoint.ipynb @@ -0,0 +1,738 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "XTPLBCWKzfWS" + }, + "source": [ + "# 7.3 Visualizing Higher Dimensions" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "f1mm_fCWm9dO" + }, + "source": [ + ">\"I am a Tralfamadorian, seeing all time as you might see a stretch of the Rocky Mountains. All time is all time. It does not change. It does not lend itself to warnings or explanations. It simply is.\" \n", + ">\n", + ">-Kurt Vonnegate in \"Slaughterhour-Five\"\n", + "\n", + "We unfortunately are not Tralfamadorians. We instead we are three dimensional beings who can't visually see a fourth dimension like its a location on the Rocky Mountains. However, this doesn't mean that the fourth dimension is meaingless to us. We can derive a lot of understand from understanding the higher dimensions. The problem is, we can't see it and thus we can't plot it. \n", + "\n", + "Fortunately, very clever mathematicians throughout history has invented techniques to allow us to simulate what the higher dimension would look like. The rest of the section will discuss these techniques." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "XR-JkcQuwozv" + }, + "source": [ + "## Using Size and Color\n", + "\n", + "This is more of a review from previous sections but one way to visualize more dimensions is by using the size and color attributes of your scatter plots. This is rather intuitive. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 368 + }, + "colab_type": "code", + "id": "kgChDoxMwyKa", + "outputId": "cc578404-1e67-476a-f736-e070bc9c233e" + }, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "
\n", + "" + ], + "text/plain": [ + "alt.Chart(...)" + ] + }, + "execution_count": 30, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "from scipy import stats\n", + "from sklearn.linear_model import LinearRegression\n", + "from sklearn.preprocessing import LabelEncoder, StandardScaler\n", + "import altair as alt\n", + "\n", + "df_bordeaux = pd.read_csv(\"http://dlsun.github.io/pods/data/bordeaux.csv\")\n", + "\n", + "alt.Chart(df_bordeaux).mark_circle().encode(\n", + " alt.X('age',\n", + " scale=alt.Scale(zero=False)\n", + " ),\n", + " alt.Y('sep',\n", + " scale=alt.Scale(zero=False)\n", + " ),\n", + " color=\"summer\",\n", + " size=\"win\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "gxV83MG_y2LJ" + }, + "source": [ + "I am sure you can see the limitations of this method: you can only go up for 4 dimensions (5 if you use a 3-D scatter plot). This is still worth mentioning as sometimes, this may be all you need. \n", + "\n", + "For higher dimensions, we should consider either feature selection or feature reduction. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "tlWDTyDsr5Zv" + }, + "source": [ + "## Feature Selection \n", + "\n", + "If we want to compress 10 dimensions worth of data into 2 dimensions, we're bound to lose some detail during that compression. We can measure how much detail we kept at the end with the explained variance ratio which gives the percentage of variance/detail we kept after the compression. The higher the ratio, the more variance and detail we kept. \n", + "\n", + "One way very simple, almost trivial, way to only visualize higher dimensional data is to only plot the two dimensions that explains the most variation in the data. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 136 + }, + "colab_type": "code", + "id": "g9kxLPD30dxa", + "outputId": "c1aedb62-573b-406c-cfa6-844ab4054122" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "summer 0.343538\n", + "sep 0.323582\n", + "age 0.206936\n", + "year 0.206936\n", + "har 0.199621\n", + "win 0.053456\n", + "Name: R^2 Values, dtype: float64" + ] + }, + "execution_count": 31, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "sclr = StandardScaler()\n", + "df_bordeaux = pd.DataFrame(sclr.fit_transform(df_bordeaux.dropna()), columns=df_bordeaux.columns)\n", + "\n", + "X = df_bordeaux.drop(\"price\", axis=1)\n", + "y = df_bordeaux[\"price\"]\n", + "\n", + "reg = LinearRegression()\n", + "reg.fit(X, y)\n", + "\n", + "scores = pd.Series(dtype=float, name=\"R^2 Values\")\n", + "for column in X.columns: \n", + " reg = LinearRegression()\n", + " reg.fit(X[[column]], y)\n", + " scores[column] = reg.score(X[[column]], y)\n", + "\n", + "scores.sort_values(ascending=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "qHprVgDLxKyE" + }, + "source": [ + "As we can see above, average summer temperature **summer** and average september temperature **sep** are the two variables that explain the most variance in the quality of the wine **price**. Thus, if we want to get the best representation of the dataset with only two dimensions, we can make a scatterplot of **summer** vs **sep**. However, even with the two variables that explain the most variation, we can only capture 33% of the variation of the original data. The other 66% is lost to the other features we chose to ignore. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 330 + }, + "colab_type": "code", + "id": "FcUCeYGOx2nP", + "outputId": "16dc19dd-f2ac-44b7-8ac6-0236cac283d4" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "% Variance Explained: 0.3333333333333333\n", + "\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 32, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAUNklEQVR4nO3dfYxcV33G8efZeLu26rQYrwvEDpiQqLwaJ2zTgCmF8NKQVqZgKFDRgqByU4RU1Ao7CLX0TY1spP5RUVqshAKCBigmtVsCIWAoL2rSrINf8kJCQEmzbkrM1glZsLfrzK9/zF083njt2ezcOefO+X6kVWbvzO793ZvxPHvOPedcR4QAAOUZSl0AACANAgAACkUAAEChCAAAKBQBAACFWpK6gIUYHR2NtWvXpi4DABpl7969P4yIVXO3NyoA1q5dq/Hx8dRlAECj2L7vVNvpAgKAQhEAAFAoAgAACkUAAEChCAAAKBQBAABzTE5Na//9D2lyajp1KbVq1DBQAKjbrn2HtHXnAQ0PDWmm1dL2Teu0cf3q1GXVghYAAFQmp6a1decBHZtp6ZHp4zo209KWnQcGtiVAAABAZeLIUQ0PnfyxODw0pIkjRxNVVC8CAAAqa1Ys00yrddK2mVZLa1YsS1RRvQgAAKisXD6i7ZvWaenwkM4eWaKlw0PavmmdVi4fSV1aLbgIDAAdNq5frQ3nj2riyFGtWbFsYD/8JQIAAB5j5fKRgf7gn0UXEAAUigAAgEIRAABQKAIAAApFAABAoQgAACgUAQAAhSIAAKBQBAAAFIoAAIBCEQAAUCgCAAAKlSwAbJ9r+6u277B9u+0/TFULAJQo5WqgxyX9cUTcavtsSXtt3xgRdySsCQCKkawFEBEPRMSt1eNHJN0paTDvvAwAGcriGoDttZIulHTzKZ7bbHvc9vjhw4f7XRoADKzkAWB7uaSdkt4dET+a+3xE7IiIsYgYW7VqVf8LBIABlTQAbA+r/eH/yYj4XMpaAKA0KUcBWdI1ku6MiL9JVQcAlCplC2CDpN+RdKntfdXX5QnrAYCiJBsGGhHflORU+weA0iW/CAwASIMAAIBCEQAAUCgCAAAKRQAAQKEIAAAoFAEAAIUiAACgUAQAABSKAACAQhEAAFAoAgAACkUAAEChCACgzyanprX//oc0OTWduhQULtly0ECJdu07pK07D2h4aEgzrZa2b1qnjetXpy4LhaIFAPTJ5NS0tu48oGMzLT0yfVzHZlrasvMALQEkQwAAfTJx5KiGh07+Jzc8NKSJI0cTVYTSEQBAn6xZsUwzrdZJ22ZaLa1ZsSxRRSgdAQD0ycrlI9q+aZ2WDg/p7JElWjo8pO2b1mnl8pHUpaFQXAQG+mjj+tXacP6oJo4c1ZoVy/jwR1IEANBnK5eP8MGPLNAFBACFIgAAoFAEAADMUcpsba4BAECHkmZr0wIAgEpps7UJAACo9Gu2di5dTHQBAUClH7O1c+piogUAAJW6Z2vn1sVECwAAOtQ5W3u2i+mYTrQyZruYUkwOJAAAYI66ZmvntiAgXUAA0Ce5LQhICwAA+iinBQEJAADos1wWBKQLCAAKlTQAbH/E9oO2b0tZBwCUKHUL4KOSLktcAxool5mU/VDSsaK/kl4DiIiv216bsgY0T04zKetW0rGi/1K3AM7I9mbb47bHDx8+nLocJJbbTMo6lXSsSCP7AIiIHRExFhFjq1atSl0OEuvXYl05KOlYkUb2AQB0ym0mZZ1KOlakQQCgUXKbSVmnko4VaTgi0u3cvlbSSyWNSvqBpPdHxDXzvX5sbCzGx8f7VB1yNjk1ncVMyn4o6VhRD9t7I2Js7vbUo4DenHL/aK5cZlL2Q0nHiv6iCwgACkUAAEChCAAAKBQBAACFIgAAoFAEAFAwFporGzeEAQrFQnOgBQAUiIXmIBEAQJEGYaE5uq8Wjy4goEBNX2iO7qveoAUAFKjJC83RfdU7tACAQm1cv1obzh9t3EJzs91Xx3SiBTPbfdWUY8gFAQAUrIkLzTW9+yondAEBaJQmd1/lhhYAgMZpavdVbggAAI3UxO6r3NAFBACFIgAAoFAEAAAUigAAgEIRAABQKAIAAApFAABAoRYUALZ/zvbZdRUDAOifrgLA9i/ZPijpgKTbbO+3/YJ6S0PTsV47kLduZwJfI+mdEfENSbL9Ykn/KGldXYWh2VivHchft11Aj85++EtSRHxT0vF6SkLTsV470AzdtgD+3faHJV0rKSS9UdLXbF8kSRFxa031oYFYrx1ohm4D4PnVf98/Z/uFagfCpT2rCI03COu1T05NF7HSZCnHiVPrKgAi4mV1F4LBMbte+5Y51wCa8gFTyvWLUo4T83NEnPlF9pMk/bWkcyLi1bafLemFEXFN3QV2Ghsbi/Hx8X7uEovQxL8uJ6emtWHbHh2bOdGCWTo8pG9tvbQxx9CNUo4Tbbb3RsTY3O3dXgT+qKQbJJ1TfX+3pHf3pjQMqpXLR/T8c5/QqA+U2esXnWavX8ynicNdH89xPh5NPDcl6fYawGhEfMb2eyUpIo7bfrTGuoAkFnr9oqndKP24TtPUc1OSblsAP7a9Uu0LvrJ9iaSHa6sKSGQh95tt8nDXuu+r2+RzU5JuWwB/JGm3pGfY/pakVZJeX1tVQELd3m+26cNd67yvbtPPTSm6DYBnSHq1pHMlbZL0ywv4WaBxurnf7CAMd63rvrqDcG5K0G0X0J9ExI8krZD0MkkfkvT3i9257cts32X7HttXLvb3Af1UdzdKk3FumqHbYaDfjogLbV8l6WBE/NPstse9Y/sstUcTvVLShKRbJL05Iu6Y72cYBoocNXG4a79wbvIw3zDQbrtxDlVLQbxS0jbbI1r8vQQulnRPRHy/KvBTkl4jad4AAHJUVzfKIODc5K3bD/HfUnsewK9FxEOSnijpPYvc92pJ93d8P1FtO4ntzbbHbY8fPnx4kbsEAMzqdimIn0j6XMf3D0h6oK6i5ux7h6QdUrsLqB/7BIASpLwl5CG1RxXNWlNtA5ApZvYOlpRDOW+RdIHtp6v9wf8mSb+dsB4Ap8HM3sGTrAUQEcclvUvtawt3SvpMRNyeqh4A82Nm72BKOpkrIq6XdH3KGkrGED10K8eZvbx/F4/ZvIWiOY+FyG1mL+/f3kh5ERiJ0JzHQuU0s5f3b+/QAihQjs155K/OxeMWgvdv7xAABcqtOY/myGFmL+/f3qELqEA5NeeBheL92ztdLQaXCxaD6y1GUTQD/59OjfPSvcUuBocBlENzHqfHaJf58f5dPLqAgEwx2gV1IwCATM2Oduk0O9oF6AUCADiFHBY9Y7QL6sY1AGCOXPrdZ0e7bJlTC/3e6BUCAOjQ2e8+O9Foy84D2nD+aJIP3lwmX2EwEQBAhxxnmTLaBXXhGgDQYc2KZTo6c/ykbUdnjtPvjoFEAABz2D7t98CgIACADhNHjmrpkrNO2rZ0yVkMvcRAIgCADgy9REkIAKADC42hJIwCQteavPjWQmpf6NDLJp8XlI0AQFdymRz1eDye2rsdetnk8wLQBYQzavKiZHXW3uTzAkgEALrQ5EXJ6qy9yecFkAgAdKHJI2PqrL3J5wXNUtfihAQAzqjJI2PqrL3J5wXNsWvfIW3Ytkdvufpmbdi2R7v3HerZ7+aWkOhak0e71Fl7k88L8jY5Na0N2/bo2MyJlubS4SF9a+ulC3qvcUtILFqTFyWrs/Ymnxfkre7FCekCAoBM1X2diQAAgEzVfZ2JLiAAyFidNwUiAAAgc3VdZ6ILCAAKRQAAfVbXpB5goegCAvqIxeOQE1oAQJ+weBxyQwAAfcLicchNkgCw/Qbbt9tu2X7M9GRgELF4HHKTqgVwm6TXSfp6ov0DfcficchNkovAEXGnJNlOsXsgmTon9QALlf0oINubJW2WpKc+9amJqwEWj8XjkIvaAsD2lyU9+RRPvS8idnX7eyJih6QdUns56B6VBwDFqy0AIuIVdf1uAMDiMQwUAAqVahjoa21PSHqhpM/bviFFHQBQslSjgK6TdF2KfQMA2ugCAoBCEQAAUCgCAAAKRQAAQKEIAAAoFAEAAIUiAACgUAQAABSKAACAQhEAAFAoAgAACkUAAEChCAAAKBQBAACFIgAAoFAEAAAUigAAgEIRAABQKAIAAApFAABAoQiAzE1OTWv//Q9pcmo6dSkABsyS1AVgfrv2HdLWnQc0PDSkmVZL2zet08b1q1OXBWBA0ALI1OTUtLbuPKBjMy09Mn1cx2Za2rLzAC0BAD1DAGRq4shRDQ+d/L9neGhIE0eOJqoIwKAhADK1ZsUyzbRaJ22babW0ZsWyRBUBGDQEQKZWLh/R9k3rtHR4SGePLNHS4SFt37ROK5ePpC4NwIAo4iLw5NS0Jo4c1ZoVyxr1Abpx/WptOH+0kbUDyN/AB0DTR9KsXD7CBz+AWgx0FxAjaQBgfgMdAIykAYD5DXQAMJIGAOY30AHASBoAmN/AXwRmJA0AnNrAB4DESBoAOJWB7gICAMyPAACAQiUJANsfsP0d2wdsX2f7CSnqANBc3Ctj8VK1AG6U9NyIWCfpbknvTVQHgAbate+QNmzbo7dcfbM2bNuj3fsOpS6pkZIEQER8KSKOV9/eJGlNijoANA8z/Hsnh2sAb5f0hfmetL3Z9rjt8cOHD/exLAA5YoZ/79Q2DNT2lyU9+RRPvS8idlWveZ+k45I+Od/viYgdknZI0tjYWNRQKoAGYYZ/79QWABHxitM9b/ttkn5D0ssjgg92AF2ZneG/Zc4qv8z1WbgkE8FsXyZpi6RfjYifpKgBQHMxw783Us0E/qCkEUk32pakmyLiikS1AGggZvgvXpIAiIjzU+wXAHBCDqOAAAAJEAAAUCgCAAAKRQAAQKHcpCH4th+RdFfqOuYYlfTD1EWcQo515ViTlGddOdYk5VlXjjVJedX1tIhYNXdj024Ic1dEjKUuopPt8dxqkvKsK8eapDzryrEmKc+6cqxJyreuTnQBAUChCAAAKFTTAmBH6gJOIceapDzryrEmKc+6cqxJyrOuHGuS8q3rpxp1ERgA0DtNawEAAHqEAACAQmUdAN3ePN72vbYP2t5nezyTmi6zfZfte2xfWWdN1f7eYPt22y3b8w496/O56ramfp+rJ9q+0fZ3q/+umOd1j1bnaZ/t3TXVctpjtz1i+9PV8zfbXltHHQus6W22D3ecm9/rQ00fsf2g7dvmed62/7aq+YDti+quqcu6Xmr74Y5z9af9qKtrEZHtl6RXSVpSPd4mads8r7tX0mguNUk6S9L3JJ0n6Wck7Zf07JrrepakX5T0NUljp3ldP8/VGWtKdK62S7qyenzlad5XUzXXccZjl/ROSf9QPX6TpE9nUNPbJH2wH++hjn2+RNJFkm6b5/nL1b61rCVdIunmTOp6qaR/6+e5WshX1i2AyPDm8V3WdLGkeyLi+xHxf5I+Jek1Ndd1Z0RkNUu6y5r6fq6q3/+x6vHHJP1mzfubTzfH3lnrZyW93NVNNBLW1HcR8XVJ/3ual7xG0sej7SZJT7D9lAzqylrWATDH6W4eH5K+ZHuv7c0Z1LRa0v0d309U23KQ6lzNJ8W5elJEPFA9/h9JT5rndUttj9u+yXYdIdHNsf/0NdUfHg9LWllDLQupSZI2VV0tn7V9bo31dCvnf3MvtL3f9hdsPyd1MZ2SLwXRo5vHvzgiDtn+BbXvMvadKplT1tRz3dTVhb6fqxROV1fnNxERtucbC/206lydJ2mP7YMR8b1e19pA/yrp2oiYtv37ardQLk1cU65uVft9NGX7ckn/IumCxDX9VPIAiB7cPD4iDlX/fdD2dWo3Yx/3h1oPajokqfOvojXVtkU5U11d/o6+nqsu9P1c2f6B7adExANVN8GD8/yO2XP1fdtfk3Sh2v3jvdLNsc++ZsL2Ekk/L2myhzUsuKaI6Nz/1WpfU0mtlvfRYkXEjzoeX2/7Q7ZHIyKLReKy7gLyiZvHb4x5bh5v+2dtnz37WO2LtKe8It+vmiTdIukC20+3/TNqX7yrZRTJQvT7XHUpxbnaLemt1eO3SnpMS8X2Ctsj1eNRSRsk3dHjOro59s5aXy9pz3x/CPWrpjl96xsl3VljPd3aLel3q9FAl0h6uKObLxnbT569ZmP7YrU/c+sM8IVJfRX6dF+S7lG7X29f9TU7GuIcSddXj89Te6TCfkm3q931kLSm6vvLJd2t9l+MtdZU7e+1avd7Tkv6gaQbMjhXZ6wp0blaKekrkr4r6cuSnlhtH5N0dfX4RZIOVufqoKR31FTLY45d0l+o/QeGJC2V9M/V++4/JZ3Xh/Nzppquqt4/+yV9VdIz+1DTtZIekDRTvafeIekKSVdUz1vS31U1H9RpRsL1ua53dZyrmyS9qB91dfvFUhAAUKisu4AAAPUhAACgUAQAABSKAACAQhEAAFAoAgAACkUAAJmpJjPxbxO1402GIlWzoj9fLdJ1m+03un2vhNHq+bFq+QfZ/jPbH7P9Ddv32X6d7e1u31fhi7aHq9fda/uqat33cdsX2b7B9vdsX9Gx7/fYvqVaTO3Pq21r3V6D/+Nqz87OYYE1DDgCAKW6TNJ/R8TzI+K5kr54htc/Q+0FzzZK+oSkr0bE8yQdlfTrHa/7r4hYL+kbkj6q9vINl0ia/aB/ldqLgV0sab2kF9h+SfWzF0j6UEQ8JyLuW/whAqdHAKBUByW90vY2278SEQ+f4fVfiIiZ6ufO0onAOChpbcfrdndsvzkiHomIw5Km3b573Kuqr2+rvVLkM3Vidcj7or2WPdAXyVcDBVKIiLvdvm3g5ZL+yvZX1F7ee/aPoqVzfmS6+rmW7Zk4sYZKSyf/O5ru2D7dsX32dZZ0VUR8uPOXu32rxx8v5piAhaIFgCLZPkfSTyLiE5I+oPZt/e6V9ILqJZtq2vUNkt5ue3lVx+rq3gxA39ECQKmeJ+kDtltqr+T4B5KWSbrG9l+qfR/jnouIL9l+lqT/qFYJnpL0FkmP1rE/4HRYDRQACkUXEAAUigAAgEIRAABQKAIAAApFAABAoQgAACgUAQAAhfp/rOG9WPsCRBEAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light", + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "explained_var = X[[\"summer\", \"sep\"]].var(axis=0).sum() / X.var(axis=0).sum() \n", + "\n", + "print(\"% Variance Explained:\", explained_var, end=\"\\n\\n\")\n", + "df_bordeaux.plot.scatter(x=\"summer\", y=\"sep\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Y6gWdO1CINp-" + }, + "source": [ + "Additionally, if we look below, using only two features has hindered our predictive accuracy. This sucks! Fortunately, some very clever mathematicians came up with ways to get around this. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + }, + "colab_type": "code", + "id": "l7re2E6KHUbs", + "outputId": "24c2f5c2-ff71-420e-874c-2bbdee6f9a8f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "All Features:\t\t 0.7526018827767169\n", + "With PCA Features:\t 0.4633153344681292\n", + "\n" + ] + } + ], + "source": [ + "reg = LinearRegression()\n", + "\n", + "reg.fit(X, y)\n", + "print(\"All Features:\\t\\t\", reg.score(X, y))\n", + "reg.fit(X[[\"summer\", \"sep\"]], y)\n", + "print(\"With PCA Features:\\t\", reg.score(X[[\"summer\", \"sep\"]], y), end=\"\\n\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "tflTZWS688xW" + }, + "source": [ + "## Dimensionality Reduction\n", + "\n", + "With feature selection, we were only able to capture 33% of the original variance, which isn't great. To capture more variation while still remaining in two variables, have to utilize some clever math.\n", + "\n", + "These clever mathematical techniques are known as feature creation, where we try to create new variables that helps us visualize higher dimensional data. There are many different techniques for dimensionality reduction all of which attempts to accomplish different things. Let's start off with the simplest and most popular one: **Principle Component Analysis**." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "cZ52OQlGAm6Q" + }, + "source": [ + "### Principle Component Analysis (PCA)\n", + "\n", + "Simply put, Principle Component Analysis create new features that maximizes variation. What PCA is **not** doing is feature selection, rather it is creating an entirely new arbitrary feature that is a combination of all the features. \n", + "\n", + "PCA involves some simple linear algebra, but SciKit-Learn has a PCA implementation. Note that all dimensionality reduction algorithms in SciKit-Learn is operated very similarity to machine learning algorithms you learned in the previous chapters. Create the object and then run `fit()` or `fit_transform()`." + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 255 + }, + "colab_type": "code", + "id": "FESc4NoI8-3v", + "outputId": "1d1b00d1-65f5-40dd-ca5f-93f9ba84ef7c" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "% Variance Explained: 0.6462543462926849\n", + "\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
PCA1PCA2
02.620822-1.437859
12.1664460.732196
22.359265-0.067934
31.518218-0.792417
41.5199200.451721
\n", + "
" + ], + "text/plain": [ + " PCA1 PCA2\n", + "0 2.620822 -1.437859\n", + "1 2.166446 0.732196\n", + "2 2.359265 -0.067934\n", + "3 1.518218 -0.792417\n", + "4 1.519920 0.451721" + ] + }, + "execution_count": 36, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "from sklearn.decomposition import PCA\n", + "\n", + "# We want to be able to plot this on a 2-D scatter plot so we choose 2 Dimensions\n", + "dimension_we_want = 2\n", + "\n", + "pca = PCA(n_components=dimension_we_want)\n", + "X_2d = pca.fit_transform(X) \n", + "X_2d = pd.DataFrame(X_2d, columns=[\"PCA1\", \"PCA2\"])\n", + "\n", + "print(\"\\n% Variance Explained:\", pca.explained_variance_ratio_.sum(), end=\"\\n\\n\")\n", + "X_2d.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 296 + }, + "colab_type": "code", + "id": "Gj87JXmmBdNu", + "outputId": "4574186e-9f94-4b91-9cf9-ba4a68b9f866" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 37, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAATe0lEQVR4nO3dcWzcZ33H8c/nEtfxcLZGjpWucUMqUtCqYIKwOjZv0lrYKIgFtVk1OsHEYIuQhkQlpoSqbGOaJtGwMaSViUWUwaSKqpvbBdFWNFVgUAQFB7mmaQrqGKyOGA1eSmOwXaf33R93Xm3X8Tm5+93zu3veL8mS73eX+30vd/597nme3/P8HBECAOSnkroAAEAaBAAAZIoAAIBMEQAAkCkCAAAytTF1ARdi69atsXPnztRlAEBHOX78+E8iYnDl9o4KgJ07d2p8fDx1GQDQUWz/cLXtdAEBQKYIAADIFAEAAJkiAAAgUwQAAGSKAADQEtMz83rs6Wc1PTOfuhSsU0edBgqgnI5MnNLBsUn1VCpaqFZ1aN+w9u7ZnrosNEALAEBTpmfmdXBsUnMLVZ2dP6e5haoOjE3SEugABACApkydmVVPZfmhpKdS0dSZ2UQVYb0IAABNGdrSp4Vqddm2hWpVQ1v6ElWE9SIAADRloL9Xh/YNa1NPRZt7N2pTT0WH9g1roL83dWlogEFgAE3bu2e7Rndt1dSZWQ1t6ePg3yEIAAAtMdDfy4G/w9AFBACZIgAAIFPJAsD2JtvftP2Y7RO2/ypVLQCQo5RjAPOSrouIGds9kh6x/WBEfCNhTQCQjWQBEBEhaaZ+s6f+E6nqAYDcJB0DsL3B9oSkZyQdjYhHV3nMftvjtsdPnz7d/iIBoEslDYCIeCEi9kgaknSN7d2rPOZwRIxExMjg4EuuaQwAuEilOAsoIp6V9CVJ16euBQBykfIsoEHbl9Z/75P025KeTFUPAOQm5VlAvyzps7Y3qBZE90TEFxLWAwBZSXkW0KSk16baPwDkrhRjAACA9iMAACBTBAAAZIoAAIBMEQAAkCkCAAAyRQAAQKYIAADIFAEAAJkiAAAgUwQAAGSKAACATBEAAJApAgAAMkUAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMhUsgCwfYXtL9l+wvYJ2+9PVQsA5Ghjwn2fk/SBiPi27c2Sjts+GhFPJKwJALKRrAUQET+KiG/Xfz8r6aSk7anqAYDclGIMwPZOSa+V9Ogq9+23PW57/PTp0+0uDQC6VvIAsN0vaUzSLRHx3Mr7I+JwRIxExMjg4GD7CwSALpU0AGz3qHbwvysi7k1ZCwDkJuVZQJZ0p6STEfGxVHUAQK5StgBGJb1T0nW2J+o/b0lYDwBkJdlpoBHxiCSn2j8A5C75IDAAIA0CAAAyRQAAQKYIAADIFAEAAJkiAIAVpmfm9djTz2p6Zj51KUChUq4GCpTOkYlTOjg2qZ5KRQvVqg7tG9bePaxRiO5ECwCom56Z18GxSc0tVHV2/pzmFqo6MDZJSwBdiwAA6qbOzKqnsvxPoqdS0dSZ2UQVAcUiAIC6oS19WqhWl21bqFY1tKUvUUVAsQgAoG6gv1eH9g1rU09Fm3s3alNPRYf2DWugvzd1aUAhGAQGlti7Z7tGd23V1JlZDW3p4+CPrkYAACsM9Pdy4EcW6AICgEwRAACQKQIAQCGYUV1+jAEAaDlmVHcGWgB1fFsBWoMZ1Z2DFoD4tgK00uKM6jm9OKlucUY1Z1eVS/YtAL6tdB9ac2kxo7pzZBUAqx0YWP+luxyZOKXR24/pHZ96VKO3H9PnJ06lLik7zKjuHNl0AZ2vm4dvK91jaWtusfvhwNikRndt5eDTZsyo7gxZtADW6ubh20r3oDVXLgP9vXrNFZfyt1RiWbQAGg1K8W2lO9CaAy5M0haA7U/bfsb240XuZz0HBr6tdD5ac8CFSd0C+IykOyT9S5E7WTwwHFgxBsCBofvQmgPWL2kARMRXbO9sx744MOSD1TyB9UndAmjI9n5J+yVpx44dTT0XBwYAeFHpzwKKiMMRMRIRI4ODg6nLAdqOiW0oSulbAEDOWKYERSp9CwDIFcuUoGipTwP9nKSvS3qV7Snb70lZD1AmTGxD0VKfBXRzyv0DZcbENhSNLiCgpJjYVl7dMjDPIDBQYsxfKZ9uGpgnAICSY/5KeXTbirN0AQHAOnXbwDwBAADr9LJLNmj+3AvLtnXywDwBsIZuGejBi3hPcbGOTJzSW+94RJWKJUm9G9zxA/OMAZxHNw30oIb3FBdrad//orB1//t+Q7u2bU5YWXPW1QKw3bPKtq2tL6cYF/qtjxmY3Yf3FM1Yre+/d0NFP3v+hfP8i86wZgDYvtb2lKQf2X5oxdLNDxVZWKtczEXCu22gB7ynaE63Tspr1AI4JOlNEbFV0mFJR22/vn6fC62sBS72W1+3vtk54z3tTGUZs+nWSXmNxgAuiYgTkhQR/2b7pKR7bR+UFIVX16RG1wI+H64g1n14TztP2cZsunFSXqMAWLB9WUT8jyRFxAnbb5D0BUmvKLy6JjXzra8b3+zc8Z52jrJOuOq2SXmNuoA+KGnb0g0RMSXptyR9pKCaWqbZZhsXiu8+vKedgTGb9lizBRARD5/nrs2Snm99Oa3Ht758TM/M8z53CcZs2mPd8wBsD0q6SdLNki6XdF9RRbVatzXb8FJl6y9GcxizaY81A8D2Zkk3SvoDSa+UdK+kKyNiqA21AetS1v5iNIfWe/EatQCekfRNSR+S9EhEhO0bii8LWL+LPdsL5UfrvViNBoFvldQr6R8l3Wq79Gf+ID/0FwMXZ80AiIiPR8TrJb2tvunfJV1u+6DtVxZeHbAO3TpJByiaIy5sPpft3aoNBP9+ROwqpKrzGBkZifHx8XbuEh2Es4CA1dk+HhEjK7c3GgTeJWlbRHxtcVtEPG77QUn/3Poy0clSH4DpLwYuTKNB4I+rNg6w0k8l/b2k3215RehInIYJdJ5Gg8DbIuI7KzfWt+0spCJ0HJZaBjpTowC4dI37OMUCkso5bb8sq0gCZdYoAMZt/8nKjbb/WNLxZndu+3rb37X9lO0PNvt8SKNsp2FezDUggBw1CoBbJP2R7S/b/rv6z39Ieo+k9zezY9sbJH1C0pslXS3pZttXN/OcSKNMp2HSHQWsX6PF4H4s6ddtXytpd33z/RFxrAX7vkbSUxHxfUmyfbdq8w2eaMFzo83KMm2fWcEoo9RnyJ1Po9NAN0l6r6Rdkr4j6c6IONeifW+X9PSS21OSfnWVGvZL2i9JO3bsaNGuUYQynIZZtu4ooMxnyDXqAvqspBHVDv5vlvS3hVe0QkQcjoiRiBgZHBxs9+7RYcrUHQWUvUuy0TyAqyPi1ZJk+07VFoZrlVOSrlhye6i+DWhKWbqjgLJ3STa8JOTiLxFxzm7pdeC/Jekq21eqduB/u2rLTgNNK0N3FFD2LslGXUCvsf1c/eespOHF320/18yO62MJ75P0RUknJd2zeAF6AOgGZe+SbHQW0IYidx4RD0h6oMh9AEBKZe6SXPclIQEAF6esXZKNuoAAAF2KAACATBEAAJApAgAAMkUAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMgUAQAAmSIAACBTBABWNT0zr8eeflbTM/OpS0HG+BwWiyuC4SWOTJzSwbFJ9VQqWqhWdWjfsPbu2Z66LGSGz2HxaAFgmemZeR0cm9TcQlVn589pbqGqA2OTfANDW/E5bA8CAMtMnZlVT2X5x6KnUtHUmdlEFSFHfA7bgwDAMkNb+rRQrS7btlCtamhLX6KKOgt91q3B57A9CAAsM9Dfq0P7hrWpp6LNvRu1qaeiQ/uGNdDfm7q00jsycUqjtx/TOz71qEZvP6bPT5xKXVLH4nPYHo6I9u/UvknShyX9iqRrImJ8Pf9uZGQkxsfX9VA0aXpmXlNnZjW0pY8/unWYnpnX6O3HNLfw4rfWTT0Vfe3gdfz/NYHPYWvYPh4RIyu3pzoL6HFJN0r6p0T7RwMD/b0d8QdXlgPEYp/1nF4MgMU+6074fyyrTvkcdqokARARJyXJdordo0uU6TRB+qzRiRgDQEdq1WmCrRq0pc8anaiwFoDthyVdtspdt0XEkQt4nv2S9kvSjh07WlQdOl0rulxa3YLYu2e7RndtLUWXFLAehQVARLyxRc9zWNJhqTYI3IrnROdrtstlaQtiMUQOjE1qdNfWpg7c9Fmjk9AFhI7UbJcLE42ARIPAtm+Q9A+SBiXdb3siIt6UohZ0rma6XBi0BRK1ACLivogYiojeiNjGwR8Xa6C/V6+54tIL7nZh0BZgNVBkjEFb5I4AQNYYtEXOGAQGgEwRAACQKQIAADJFAABApggAACi5oi40xFlAAFBiRa56SwsAAEqqVaveng8BAAAlVfSaVQQAAJRU0WtWEQAAUFJFr1nFIDAAlFiRa1YRAABQckWtWUUXELJU1HnVQCehBYDsFHledSebnplnaezMEADISlHXAu50hGKe6AJCVrgW8EsVPdkI5UUAICtcC/ilCMV8EQDICtcCfilCMV+MASA7XAt4ucVQPLBiDCD3/5ccEADIEtcCXo5QzBMBAEASoZgjxgAAIFNJAsD2R20/aXvS9n22L01RBwDkLFUL4Kik3RExLOl7km5NVAcAZCtJAETEQxFxrn7zG5KGUtQBADkrwxjAuyU9eL47be+3PW57/PTp020sqxxYtAxAUQo7C8j2w5IuW+Wu2yLiSP0xt0k6J+mu8z1PRByWdFiSRkZGooBSS4v1WQAUqbAAiIg3rnW/7XdJequkN0REVgf29WDRMgBFS3UW0PWSDkjaGxE/T1FD2bE+C4CipRoDuEPSZklHbU/Y/mSiOkqL9VkAFC3JTOCI2JViv52E9VkAFI2lIEqM9VkAFIkAKDnWZwFQlDLMAwAAJEAAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMgUAQAAmSIAACBTBACyMD0zr8eeflbTM/OpSwFKg+Wg0fWOTJzSwRUX1tm7Z3vqsoDkaAGgq03PzOvg2KTmFqo6O39OcwtVHRibpCUAiABAl5s6M6ueyvKPeU+loqkzs4kqAsqDAEBXG9rSp4Vqddm2hWpVQ1v6ElUElAcBgK420N+rQ/uGtamnos29G7Wpp6JD+4a5zCYgBoGRgb17tmt011ZNnZnV0JY+Dv5AHQGALAz093LgB1agCwgAMpUkAGz/te1J2xO2H7J9eYo6ACBnqVoAH42I4YjYI+kLkv4iUR0AkK0kARARzy25+TJJkaIOAMhZskFg238j6Q8l/VTStWs8br+k/ZK0Y8eO9hQHABlwRDFfvm0/LOmyVe66LSKOLHncrZI2RcRfruM5T0v6YeuqLNRWST9JXUQb8Xq7W06vtxtf68sjYnDlxsICYL1s75D0QETsTlpIi9kej4iR1HW0C6+3u+X0enN6ranOArpqyc23SXoyRR0AkLNUYwAfsf0qSVXVunTem6gOAMhWkgCIiH0p9ttmh1MX0Ga83u6W0+vN5rUmHwMAAKTBUhAAkCkCAAAyRQAUyPZHbT9ZX/foPtuXpq6pSLZvsn3CdtV2V55GZ/t629+1/ZTtD6aup2i2P237GduPp66laLavsP0l20/UP8fvT11T0QiAYh2VtDsihiV9T9Ktiesp2uOSbpT0ldSFFMH2BkmfkPRmSVdLutn21WmrKtxnJF2fuog2OSfpAxFxtaTXS/rTbn9/CYACRcRDEXGufvMbkoZS1lO0iDgZEd9NXUeBrpH0VER8PyKel3S3avNYulZEfEXS/6auox0i4kcR8e3672clnZS0PW1VxSIA2ufdkh5MXQSasl3S00tuT6nLDxC5sr1T0mslPZq2kmJxRbAmrWfNI9u3qda8vKudtRVhvWs8AZ3Kdr+kMUm3rFi5uOsQAE2KiDeudb/td0l6q6Q3RBdMumj0ervcKUlXLLk9VN+GLmG7R7WD/10RcW/qeopGF1CBbF8v6YCkvRHx89T1oGnfknSV7SttXyLp7ZI+n7gmtIhtS7pT0smI+FjqetqBACjWHZI2Szpav/zlJ1MXVCTbN9iekvRrku63/cXUNbVSfUD/fZK+qNoA4T0RcSJtVcWy/TlJX5f0KttTtt+TuqYCjUp6p6Tr6n+vE7bfkrqoIrEUBABkihYAAGSKAACATBEAAJApAgAAMkUAAECmCABgFbZfqJ8G+Ljtf7X9C/Xtl9m+2/Z/2j5u+wHbr1zy726xPWf7l5ZsG6ivMjlj+44UrwdYDQEArG42IvZExG5Jz0t6b32i0H2SvhwRr4iI16m2wuu2Jf/uZtUmjN24ZNucpD+X9GftKR1YHwIAaOyrknZJulbSQkT8/4S+iHgsIr4qSbZfIalf0odUC4LFx/wsIh5RLQiA0iAAgDXY3qja+v/fkbRb0vE1Hv521ZaI/qpqM2e3rfFYIDkCAFhdn+0JSeOS/lu1NWIauVnS3RFRVW1BsZsKrA9oGquBAqubjYg9SzfYPiHp91Z7sO1XS7pKtXWfJOkSSf+l2npQQCnRAgDW75ikXtv7FzfYHrb9m6p9+/9wROys/1wu6XLbL09VLNAIi8EBq7A9ExH9q2y/XNLHJb1OtUHdH0i6RbUVQt8SEU8ueezHJP04Im63/QNJv6hay+BZSb8TEU8U/TqAtRAAAJApuoAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMjU/wFur9UWE4oSkQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light", + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "X_2d.plot.scatter(x=\"PCA1\", y=\"PCA2\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "cT7Rv9lEDSji" + }, + "source": [ + "As we can see from above, the two new features (known as Principle Components) are not like any of our input features. Additionally, these two new components explains 64.6% of all the original variation. Let's see how well this performs in explaining our dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "eBHVGhd-CJF8" + }, + "outputs": [], + "source": [ + "reg = LinearRegression()\n", + "\n", + "reg.fit(X, y)\n", + "print(\"All Features:\\t\\t\", reg.score(X, y))\n", + "reg.fit(X_2d, y)\n", + "print(\"With PCA Features:\\t\", reg.score(X_2d, y), end=\"\\n\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "bsoN_a8XI4p2" + }, + "source": [ + "As expected, with a higher explained variance ratio, we perform better in predicting the quality of the wine. With Dimensionality Reduction, we were able to capture most of the variation in the dataset while still being able to view it in two dimensions. In the most basic of terms, PCA creates a variable projected along the axis of maximum variable.\n", + "\n", + "![](https://raw.githubusercontent.com/bfkwong/data/master/IMG_0187%202.jpg)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "vwDiZRX3JYLN" + }, + "source": [ + "### Optional: Linear Algebra Behind PCA\n", + "\n", + "Principle Component Analysis chooses principle components along te **axis of greatest variance**. In Linear Algebra terms, the axis of greatest variance is the **Eigenvector with the largest Eigenvalue of the covariance matrix ($\\Sigma$)**\n", + "\n", + "The following are the steps in order to do PCA manually. \n", + "\n", + "1. Given data matrix $M$, generate the covariance matrix of $M$ denoted as $\\Sigma$\n", + "2. We then compute the Eigenvector and Eigenvalue of covariance matrix $\\Sigma$\n", + "3. Project the features to the Eigenvector with the largest Eigenvalue using the dot product (cross product for more than 1 dimensions)" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "J26dEAoAD0q7" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "# Step 1: Calculate covariance matrix \n", + "cov_mtrx = np.cov(X.T)\n", + "\n", + "# Step 2: Calculate Eigenvector and Eigenvalue\n", + "W,v = np.linalg.eig(cov_mtrx)\n", + "\n", + "# Step 3: Find the largest Eigenvalue and project our data onto the corresponding Eigenvector\n", + "idx_largest_eigenval = np.argmax(W)\n", + "eigenvec = v[:,idx_largest_eigenval]\n", + "\n", + "total = []\n", + "for row in X.index: \n", + " total.append(np.dot(X.loc[row], eigenvec))\n", + "\n", + "pd.DataFrame(pd.Series(total), columns=[\"PCA1\"]).head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "_G7NWh_1nf_k" + }, + "source": [ + "One interesting thing you may see here is that the eigenvalue corresponds to the variance explained by its corresponding eigenvalue. The eigenvalue of PCA1 is 2.26 and the variance of PCA1 is also 2.26" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "9QCALn43iTYd" + }, + "outputs": [], + "source": [ + "idx_largest_eigenval = np.argmax(W)\n", + "variance = pd.Series(total).var()\n", + "\n", + "print(\"Largest Eigenvalue:\\t\", W[idx_largest_eigenval])\n", + "print(\"Variance of Eigenvector:\", variance)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "bmLOaYzvEeG7" + }, + "source": [ + "### Multidimensional Scaling (MDS)\n", + "\n", + "We now pay a visit to our good friend, the Euclidean distance. One incredibly useful aspect of Euclidean distance is that it works in higher dimensions. The formula $$\\sqrt{x_1^2 + x_2^2}$$ is for two dimensional Euclidean distance, but to move it to a third dimension, it is as easy as adding a $x^3$ variable. One thing to recognize is that Euclidean distance in all dimension is still a number. \n", + "\n", + "Why am I rambling on about something you learned in middle school? Well the realization that Euclidean distance is scalar in all dimensions means that we can preserve the variance of n-th dimensional data in two dimensions as long as we try to ensure that the Euclidean distances in the n-th dimensional is proportion to the Euclidean distance in 2 dimensions. That was a lot to take in, the following image explains the concept.\n", + "\n", + "![](https://raw.githubusercontent.com/bfkwong/data/master/IMG_0188.jpg)\n", + "\n", + "Notice how when we reduced our dimensions from 2 to 1, the distances between points A, B, and C remained the same. Meaning that x, y, and z remained the same between the two dimensions. While distances may not always be preserved perfectly between dimensions, MDS attempts to preserve it as well as possible. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "L4AjVFrwn78I" + }, + "outputs": [], + "source": [ + "from sklearn.manifold import MDS\n", + "# We want to be able to plot this on a 2-D scatter plot so we choose 2 Dimensions\n", + "dimension_we_want = 2\n", + "\n", + "mds = MDS(n_components=dimension_we_want)\n", + "X_2d = mds.fit_transform(X) \n", + "X_2d = pd.DataFrame(X_2d, columns=[\"Dimension 1\", \"Dimension 2\"])\n", + "\n", + "display(X_2d.head())\n", + "print(\"Percentage variance explained:\", X_2d.var().sum()/X.var().sum())\n", + "X_2d.plot.scatter(x=\"Dimension 1\", y=\"Dimension 2\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "mlIA-o_cEnSD" + }, + "source": [ + "By using MDS, we were actually able to preserve over 95% of the variance from the original datasets. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "88kI9JxzFO6n" + }, + "source": [ + "### Linear vs Nonlinear Dimensionality Reduction \n", + "\n", + "Thus far, we have explored PCA (a linear reduction technique) and MDS (a nonlinear reduction technique). While there are a lot of differences between the two methods. The key differences could be boiled down to just the following statement: **linear dimensionality reduction technique only stretch and shift the data while nonlinear techniques make more drastic changes to the data**.\n", + "\n", + "This sometimes leads to nonlinear techniques being better at capturing variance but losing the overall shape of the data whereas linear techniques are better at keeping the general shape of the original data but loses more variance along the way. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "7uprQWlUGC37" + }, + "source": [ + "# Exercises" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "UnMJKgsTkDoZ" + }, + "source": [ + "1. Consider the Iris dataset (https://raw.githubusercontent.com/dlsun/pods/master/data/iris.csv). Drop the \"SepalWidth\" and \"PedalWidth\" columns and then apply PCA on \"SepalLength\" and \"PedalLength\" with `n_components = 2`. How many percent of the variance was PCA able to capture in this case? What happens when we use PCA to compress 2D data into 2D data?" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "HCXKtA8AlSQq" + }, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "7.2 Visualizing Higher Dimensions.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.4" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/07-Unsupervised-Learning/7.3 Visualizing_Higher_Dimensions.ipynb b/07-Unsupervised-Learning/7.3 Visualizing_Higher_Dimensions.ipynb new file mode 100644 index 0000000..13d4466 --- /dev/null +++ b/07-Unsupervised-Learning/7.3 Visualizing_Higher_Dimensions.ipynb @@ -0,0 +1,738 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "XTPLBCWKzfWS" + }, + "source": [ + "# 7.3 Visualizing Higher Dimensions" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "f1mm_fCWm9dO" + }, + "source": [ + ">\"I am a Tralfamadorian, seeing all time as you might see a stretch of the Rocky Mountains. All time is all time. It does not change. It does not lend itself to warnings or explanations. It simply is.\" \n", + ">\n", + ">-Kurt Vonnegate in \"Slaughterhour-Five\"\n", + "\n", + "We unfortunately are not Tralfamadorians. We instead we are three dimensional beings who can't visually see a fourth dimension like its a location on the Rocky Mountains. However, this doesn't mean that the fourth dimension is meaingless to us. We can derive a lot of understand from understanding the higher dimensions. The problem is, we can't see it and thus we can't plot it. \n", + "\n", + "Fortunately, very clever mathematicians throughout history has invented techniques to allow us to simulate what the higher dimension would look like. The rest of the section will discuss these techniques." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "XR-JkcQuwozv" + }, + "source": [ + "## Using Size and Color\n", + "\n", + "This is more of a review from previous sections but one way to visualize more dimensions is by using the size and color attributes of your scatter plots. This is rather intuitive. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 368 + }, + "colab_type": "code", + "id": "kgChDoxMwyKa", + "outputId": "cc578404-1e67-476a-f736-e070bc9c233e" + }, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "
\n", + "" + ], + "text/plain": [ + "alt.Chart(...)" + ] + }, + "execution_count": 30, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "import pandas as pd\n", + "from scipy import stats\n", + "from sklearn.linear_model import LinearRegression\n", + "from sklearn.preprocessing import LabelEncoder, StandardScaler\n", + "import altair as alt\n", + "\n", + "df_bordeaux = pd.read_csv(\"http://dlsun.github.io/pods/data/bordeaux.csv\")\n", + "\n", + "alt.Chart(df_bordeaux).mark_circle().encode(\n", + " alt.X('age',\n", + " scale=alt.Scale(zero=False)\n", + " ),\n", + " alt.Y('sep',\n", + " scale=alt.Scale(zero=False)\n", + " ),\n", + " color=\"summer\",\n", + " size=\"win\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "gxV83MG_y2LJ" + }, + "source": [ + "I am sure you can see the limitations of this method: you can only go up for 4 dimensions (5 if you use a 3-D scatter plot). This is still worth mentioning as sometimes, this may be all you need. \n", + "\n", + "For higher dimensions, we should consider either feature selection or feature reduction. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "tlWDTyDsr5Zv" + }, + "source": [ + "## Feature Selection \n", + "\n", + "If we want to compress 10 dimensions worth of data into 2 dimensions, we're bound to lose some detail during that compression. We can measure how much detail we kept at the end with the explained variance ratio which gives the percentage of variance/detail we kept after the compression. The higher the ratio, the more variance and detail we kept. \n", + "\n", + "One way very simple, almost trivial, way to only visualize higher dimensional data is to only plot the two dimensions that explains the most variation in the data. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 136 + }, + "colab_type": "code", + "id": "g9kxLPD30dxa", + "outputId": "c1aedb62-573b-406c-cfa6-844ab4054122" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "summer 0.343538\n", + "sep 0.323582\n", + "age 0.206936\n", + "year 0.206936\n", + "har 0.199621\n", + "win 0.053456\n", + "Name: R^2 Values, dtype: float64" + ] + }, + "execution_count": 31, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "sclr = StandardScaler()\n", + "df_bordeaux = pd.DataFrame(sclr.fit_transform(df_bordeaux.dropna()), columns=df_bordeaux.columns)\n", + "\n", + "X = df_bordeaux.drop(\"price\", axis=1)\n", + "y = df_bordeaux[\"price\"]\n", + "\n", + "reg = LinearRegression()\n", + "reg.fit(X, y)\n", + "\n", + "scores = pd.Series(dtype=float, name=\"R^2 Values\")\n", + "for column in X.columns: \n", + " reg = LinearRegression()\n", + " reg.fit(X[[column]], y)\n", + " scores[column] = reg.score(X[[column]], y)\n", + "\n", + "scores.sort_values(ascending=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "qHprVgDLxKyE" + }, + "source": [ + "As we can see above, average summer temperature **summer** and average september temperature **sep** are the two variables that explain the most variance in the quality of the wine **price**. Thus, if we want to get the best representation of the dataset with only two dimensions, we can make a scatterplot of **summer** vs **sep**. However, even with the two variables that explain the most variation, we can only capture 33% of the variation of the original data. The other 66% is lost to the other features we chose to ignore. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 330 + }, + "colab_type": "code", + "id": "FcUCeYGOx2nP", + "outputId": "16dc19dd-f2ac-44b7-8ac6-0236cac283d4" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "% Variance Explained: 0.3333333333333333\n", + "\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 32, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAUNklEQVR4nO3dfYxcV33G8efZeLu26rQYrwvEDpiQqLwaJ2zTgCmF8NKQVqZgKFDRgqByU4RU1Ao7CLX0TY1spP5RUVqshAKCBigmtVsCIWAoL2rSrINf8kJCQEmzbkrM1glZsLfrzK9/zF083njt2ezcOefO+X6kVWbvzO793ZvxPHvOPedcR4QAAOUZSl0AACANAgAACkUAAEChCAAAKBQBAACFWpK6gIUYHR2NtWvXpi4DABpl7969P4yIVXO3NyoA1q5dq/Hx8dRlAECj2L7vVNvpAgKAQhEAAFAoAgAACkUAAEChCAAAKBQBAABzTE5Na//9D2lyajp1KbVq1DBQAKjbrn2HtHXnAQ0PDWmm1dL2Teu0cf3q1GXVghYAAFQmp6a1decBHZtp6ZHp4zo209KWnQcGtiVAAABAZeLIUQ0PnfyxODw0pIkjRxNVVC8CAAAqa1Ys00yrddK2mVZLa1YsS1RRvQgAAKisXD6i7ZvWaenwkM4eWaKlw0PavmmdVi4fSV1aLbgIDAAdNq5frQ3nj2riyFGtWbFsYD/8JQIAAB5j5fKRgf7gn0UXEAAUigAAgEIRAABQKAIAAApFAABAoQgAACgUAQAAhSIAAKBQBAAAFIoAAIBCEQAAUCgCAAAKlSwAbJ9r+6u277B9u+0/TFULAJQo5WqgxyX9cUTcavtsSXtt3xgRdySsCQCKkawFEBEPRMSt1eNHJN0paTDvvAwAGcriGoDttZIulHTzKZ7bbHvc9vjhw4f7XRoADKzkAWB7uaSdkt4dET+a+3xE7IiIsYgYW7VqVf8LBIABlTQAbA+r/eH/yYj4XMpaAKA0KUcBWdI1ku6MiL9JVQcAlCplC2CDpN+RdKntfdXX5QnrAYCiJBsGGhHflORU+weA0iW/CAwASIMAAIBCEQAAUCgCAAAKRQAAQKEIAAAoFAEAAIUiAACgUAQAABSKAACAQhEAAFAoAgAACkUAAEChCACgzyanprX//oc0OTWduhQULtly0ECJdu07pK07D2h4aEgzrZa2b1qnjetXpy4LhaIFAPTJ5NS0tu48oGMzLT0yfVzHZlrasvMALQEkQwAAfTJx5KiGh07+Jzc8NKSJI0cTVYTSEQBAn6xZsUwzrdZJ22ZaLa1ZsSxRRSgdAQD0ycrlI9q+aZ2WDg/p7JElWjo8pO2b1mnl8pHUpaFQXAQG+mjj+tXacP6oJo4c1ZoVy/jwR1IEANBnK5eP8MGPLNAFBACFIgAAoFAEAADMUcpsba4BAECHkmZr0wIAgEpps7UJAACo9Gu2di5dTHQBAUClH7O1c+piogUAAJW6Z2vn1sVECwAAOtQ5W3u2i+mYTrQyZruYUkwOJAAAYI66ZmvntiAgXUAA0Ce5LQhICwAA+iinBQEJAADos1wWBKQLCAAKlTQAbH/E9oO2b0tZBwCUKHUL4KOSLktcAxool5mU/VDSsaK/kl4DiIiv216bsgY0T04zKetW0rGi/1K3AM7I9mbb47bHDx8+nLocJJbbTMo6lXSsSCP7AIiIHRExFhFjq1atSl0OEuvXYl05KOlYkUb2AQB0ym0mZZ1KOlakQQCgUXKbSVmnko4VaTgi0u3cvlbSSyWNSvqBpPdHxDXzvX5sbCzGx8f7VB1yNjk1ncVMyn4o6VhRD9t7I2Js7vbUo4DenHL/aK5cZlL2Q0nHiv6iCwgACkUAAEChCAAAKBQBAACFIgAAoFAEAFAwFporGzeEAQrFQnOgBQAUiIXmIBEAQJEGYaE5uq8Wjy4goEBNX2iO7qveoAUAFKjJC83RfdU7tACAQm1cv1obzh9t3EJzs91Xx3SiBTPbfdWUY8gFAQAUrIkLzTW9+yondAEBaJQmd1/lhhYAgMZpavdVbggAAI3UxO6r3NAFBACFIgAAoFAEAAAUigAAgEIRAABQKAIAAApFAABAoRYUALZ/zvbZdRUDAOifrgLA9i/ZPijpgKTbbO+3/YJ6S0PTsV47kLduZwJfI+mdEfENSbL9Ykn/KGldXYWh2VivHchft11Aj85++EtSRHxT0vF6SkLTsV470AzdtgD+3faHJV0rKSS9UdLXbF8kSRFxa031oYFYrx1ohm4D4PnVf98/Z/uFagfCpT2rCI03COu1T05NF7HSZCnHiVPrKgAi4mV1F4LBMbte+5Y51wCa8gFTyvWLUo4T83NEnPlF9pMk/bWkcyLi1bafLemFEXFN3QV2Ghsbi/Hx8X7uEovQxL8uJ6emtWHbHh2bOdGCWTo8pG9tvbQxx9CNUo4Tbbb3RsTY3O3dXgT+qKQbJJ1TfX+3pHf3pjQMqpXLR/T8c5/QqA+U2esXnWavX8ynicNdH89xPh5NPDcl6fYawGhEfMb2eyUpIo7bfrTGuoAkFnr9oqndKP24TtPUc1OSblsAP7a9Uu0LvrJ9iaSHa6sKSGQh95tt8nDXuu+r2+RzU5JuWwB/JGm3pGfY/pakVZJeX1tVQELd3m+26cNd67yvbtPPTSm6DYBnSHq1pHMlbZL0ywv4WaBxurnf7CAMd63rvrqDcG5K0G0X0J9ExI8krZD0MkkfkvT3i9257cts32X7HttXLvb3Af1UdzdKk3FumqHbYaDfjogLbV8l6WBE/NPstse9Y/sstUcTvVLShKRbJL05Iu6Y72cYBoocNXG4a79wbvIw3zDQbrtxDlVLQbxS0jbbI1r8vQQulnRPRHy/KvBTkl4jad4AAHJUVzfKIODc5K3bD/HfUnsewK9FxEOSnijpPYvc92pJ93d8P1FtO4ntzbbHbY8fPnx4kbsEAMzqdimIn0j6XMf3D0h6oK6i5ux7h6QdUrsLqB/7BIASpLwl5CG1RxXNWlNtA5ApZvYOlpRDOW+RdIHtp6v9wf8mSb+dsB4Ap8HM3sGTrAUQEcclvUvtawt3SvpMRNyeqh4A82Nm72BKOpkrIq6XdH3KGkrGED10K8eZvbx/F4/ZvIWiOY+FyG1mL+/f3kh5ERiJ0JzHQuU0s5f3b+/QAihQjs155K/OxeMWgvdv7xAABcqtOY/myGFmL+/f3qELqEA5NeeBheL92ztdLQaXCxaD6y1GUTQD/59OjfPSvcUuBocBlENzHqfHaJf58f5dPLqAgEwx2gV1IwCATM2Oduk0O9oF6AUCADiFHBY9Y7QL6sY1AGCOXPrdZ0e7bJlTC/3e6BUCAOjQ2e8+O9Foy84D2nD+aJIP3lwmX2EwEQBAhxxnmTLaBXXhGgDQYc2KZTo6c/ykbUdnjtPvjoFEAABz2D7t98CgIACADhNHjmrpkrNO2rZ0yVkMvcRAIgCADgy9REkIAKADC42hJIwCQteavPjWQmpf6NDLJp8XlI0AQFdymRz1eDye2rsdetnk8wLQBYQzavKiZHXW3uTzAkgEALrQ5EXJ6qy9yecFkAgAdKHJI2PqrL3J5wXNUtfihAQAzqjJI2PqrL3J5wXNsWvfIW3Ytkdvufpmbdi2R7v3HerZ7+aWkOhak0e71Fl7k88L8jY5Na0N2/bo2MyJlubS4SF9a+ulC3qvcUtILFqTFyWrs/Ymnxfkre7FCekCAoBM1X2diQAAgEzVfZ2JLiAAyFidNwUiAAAgc3VdZ6ILCAAKRQAAfVbXpB5goegCAvqIxeOQE1oAQJ+weBxyQwAAfcLicchNkgCw/Qbbt9tu2X7M9GRgELF4HHKTqgVwm6TXSfp6ov0DfcficchNkovAEXGnJNlOsXsgmTon9QALlf0oINubJW2WpKc+9amJqwEWj8XjkIvaAsD2lyU9+RRPvS8idnX7eyJih6QdUns56B6VBwDFqy0AIuIVdf1uAMDiMQwUAAqVahjoa21PSHqhpM/bviFFHQBQslSjgK6TdF2KfQMA2ugCAoBCEQAAUCgCAAAKRQAAQKEIAAAoFAEAAIUiAACgUAQAABSKAACAQhEAAFAoAgAACkUAAEChCAAAKBQBAACFIgAAoFAEAAAUigAAgEIRAABQKAIAAApFAABAoQiAzE1OTWv//Q9pcmo6dSkABsyS1AVgfrv2HdLWnQc0PDSkmVZL2zet08b1q1OXBWBA0ALI1OTUtLbuPKBjMy09Mn1cx2Za2rLzAC0BAD1DAGRq4shRDQ+d/L9neGhIE0eOJqoIwKAhADK1ZsUyzbRaJ22babW0ZsWyRBUBGDQEQKZWLh/R9k3rtHR4SGePLNHS4SFt37ROK5ePpC4NwIAo4iLw5NS0Jo4c1ZoVyxr1Abpx/WptOH+0kbUDyN/AB0DTR9KsXD7CBz+AWgx0FxAjaQBgfgMdAIykAYD5DXQAMJIGAOY30AHASBoAmN/AXwRmJA0AnNrAB4DESBoAOJWB7gICAMyPAACAQiUJANsfsP0d2wdsX2f7CSnqANBc3Ctj8VK1AG6U9NyIWCfpbknvTVQHgAbate+QNmzbo7dcfbM2bNuj3fsOpS6pkZIEQER8KSKOV9/eJGlNijoANA8z/Hsnh2sAb5f0hfmetL3Z9rjt8cOHD/exLAA5YoZ/79Q2DNT2lyU9+RRPvS8idlWveZ+k45I+Od/viYgdknZI0tjYWNRQKoAGYYZ/79QWABHxitM9b/ttkn5D0ssjgg92AF2ZneG/Zc4qv8z1WbgkE8FsXyZpi6RfjYifpKgBQHMxw783Us0E/qCkEUk32pakmyLiikS1AGggZvgvXpIAiIjzU+wXAHBCDqOAAAAJEAAAUCgCAAAKRQAAQKHcpCH4th+RdFfqOuYYlfTD1EWcQo515ViTlGddOdYk5VlXjjVJedX1tIhYNXdj024Ic1dEjKUuopPt8dxqkvKsK8eapDzryrEmKc+6cqxJyreuTnQBAUChCAAAKFTTAmBH6gJOIceapDzryrEmKc+6cqxJyrOuHGuS8q3rpxp1ERgA0DtNawEAAHqEAACAQmUdAN3ePN72vbYP2t5nezyTmi6zfZfte2xfWWdN1f7eYPt22y3b8w496/O56ramfp+rJ9q+0fZ3q/+umOd1j1bnaZ/t3TXVctpjtz1i+9PV8zfbXltHHQus6W22D3ecm9/rQ00fsf2g7dvmed62/7aq+YDti+quqcu6Xmr74Y5z9af9qKtrEZHtl6RXSVpSPd4mads8r7tX0mguNUk6S9L3JJ0n6Wck7Zf07JrrepakX5T0NUljp3ldP8/VGWtKdK62S7qyenzlad5XUzXXccZjl/ROSf9QPX6TpE9nUNPbJH2wH++hjn2+RNJFkm6b5/nL1b61rCVdIunmTOp6qaR/6+e5WshX1i2AyPDm8V3WdLGkeyLi+xHxf5I+Jek1Ndd1Z0RkNUu6y5r6fq6q3/+x6vHHJP1mzfubTzfH3lnrZyW93NVNNBLW1HcR8XVJ/3ual7xG0sej7SZJT7D9lAzqylrWATDH6W4eH5K+ZHuv7c0Z1LRa0v0d309U23KQ6lzNJ8W5elJEPFA9/h9JT5rndUttj9u+yXYdIdHNsf/0NdUfHg9LWllDLQupSZI2VV0tn7V9bo31dCvnf3MvtL3f9hdsPyd1MZ2SLwXRo5vHvzgiDtn+BbXvMvadKplT1tRz3dTVhb6fqxROV1fnNxERtucbC/206lydJ2mP7YMR8b1e19pA/yrp2oiYtv37ardQLk1cU65uVft9NGX7ckn/IumCxDX9VPIAiB7cPD4iDlX/fdD2dWo3Yx/3h1oPajokqfOvojXVtkU5U11d/o6+nqsu9P1c2f6B7adExANVN8GD8/yO2XP1fdtfk3Sh2v3jvdLNsc++ZsL2Ekk/L2myhzUsuKaI6Nz/1WpfU0mtlvfRYkXEjzoeX2/7Q7ZHIyKLReKy7gLyiZvHb4x5bh5v+2dtnz37WO2LtKe8It+vmiTdIukC20+3/TNqX7yrZRTJQvT7XHUpxbnaLemt1eO3SnpMS8X2Ctsj1eNRSRsk3dHjOro59s5aXy9pz3x/CPWrpjl96xsl3VljPd3aLel3q9FAl0h6uKObLxnbT569ZmP7YrU/c+sM8IVJfRX6dF+S7lG7X29f9TU7GuIcSddXj89Te6TCfkm3q931kLSm6vvLJd2t9l+MtdZU7e+1avd7Tkv6gaQbMjhXZ6wp0blaKekrkr4r6cuSnlhtH5N0dfX4RZIOVufqoKR31FTLY45d0l+o/QeGJC2V9M/V++4/JZ3Xh/Nzppquqt4/+yV9VdIz+1DTtZIekDRTvafeIekKSVdUz1vS31U1H9RpRsL1ua53dZyrmyS9qB91dfvFUhAAUKisu4AAAPUhAACgUAQAABSKAACAQhEAAFAoAgAACkUAAJmpJjPxbxO1402GIlWzoj9fLdJ1m+03un2vhNHq+bFq+QfZ/jPbH7P9Ddv32X6d7e1u31fhi7aHq9fda/uqat33cdsX2b7B9vdsX9Gx7/fYvqVaTO3Pq21r3V6D/+Nqz87OYYE1DDgCAKW6TNJ/R8TzI+K5kr54htc/Q+0FzzZK+oSkr0bE8yQdlfTrHa/7r4hYL+kbkj6q9vINl0ia/aB/ldqLgV0sab2kF9h+SfWzF0j6UEQ8JyLuW/whAqdHAKBUByW90vY2278SEQ+f4fVfiIiZ6ufO0onAOChpbcfrdndsvzkiHomIw5Km3b573Kuqr2+rvVLkM3Vidcj7or2WPdAXyVcDBVKIiLvdvm3g5ZL+yvZX1F7ee/aPoqVzfmS6+rmW7Zk4sYZKSyf/O5ru2D7dsX32dZZ0VUR8uPOXu32rxx8v5piAhaIFgCLZPkfSTyLiE5I+oPZt/e6V9ILqJZtq2vUNkt5ue3lVx+rq3gxA39ECQKmeJ+kDtltqr+T4B5KWSbrG9l+qfR/jnouIL9l+lqT/qFYJnpL0FkmP1rE/4HRYDRQACkUXEAAUigAAgEIRAABQKAIAAApFAABAoQgAACgUAQAAhfp/rOG9WPsCRBEAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light", + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "explained_var = X[[\"summer\", \"sep\"]].var(axis=0).sum() / X.var(axis=0).sum() \n", + "\n", + "print(\"% Variance Explained:\", explained_var, end=\"\\n\\n\")\n", + "df_bordeaux.plot.scatter(x=\"summer\", y=\"sep\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Y6gWdO1CINp-" + }, + "source": [ + "Additionally, if we look below, using only two features has hindered our predictive accuracy. This sucks! Fortunately, some very clever mathematicians came up with ways to get around this. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + }, + "colab_type": "code", + "id": "l7re2E6KHUbs", + "outputId": "24c2f5c2-ff71-420e-874c-2bbdee6f9a8f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "All Features:\t\t 0.7526018827767169\n", + "With PCA Features:\t 0.4633153344681292\n", + "\n" + ] + } + ], + "source": [ + "reg = LinearRegression()\n", + "\n", + "reg.fit(X, y)\n", + "print(\"All Features:\\t\\t\", reg.score(X, y))\n", + "reg.fit(X[[\"summer\", \"sep\"]], y)\n", + "print(\"With PCA Features:\\t\", reg.score(X[[\"summer\", \"sep\"]], y), end=\"\\n\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "tflTZWS688xW" + }, + "source": [ + "## Dimensionality Reduction\n", + "\n", + "With feature selection, we were only able to capture 33% of the original variance, which isn't great. To capture more variation while still remaining in two variables, have to utilize some clever math.\n", + "\n", + "These clever mathematical techniques are known as feature creation, where we try to create new variables that helps us visualize higher dimensional data. There are many different techniques for dimensionality reduction all of which attempts to accomplish different things. Let's start off with the simplest and most popular one: **Principle Component Analysis**." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "cZ52OQlGAm6Q" + }, + "source": [ + "### Principle Component Analysis (PCA)\n", + "\n", + "Simply put, Principle Component Analysis create new features that maximizes variation. What PCA is **not** doing is feature selection, rather it is creating an entirely new arbitrary feature that is a combination of all the features. \n", + "\n", + "PCA involves some simple linear algebra, but SciKit-Learn has a PCA implementation. Note that all dimensionality reduction algorithms in SciKit-Learn is operated very similarity to machine learning algorithms you learned in the previous chapters. Create the object and then run `fit()` or `fit_transform()`." + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 255 + }, + "colab_type": "code", + "id": "FESc4NoI8-3v", + "outputId": "1d1b00d1-65f5-40dd-ca5f-93f9ba84ef7c" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "% Variance Explained: 0.6462543462926849\n", + "\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
PCA1PCA2
02.620822-1.437859
12.1664460.732196
22.359265-0.067934
31.518218-0.792417
41.5199200.451721
\n", + "
" + ], + "text/plain": [ + " PCA1 PCA2\n", + "0 2.620822 -1.437859\n", + "1 2.166446 0.732196\n", + "2 2.359265 -0.067934\n", + "3 1.518218 -0.792417\n", + "4 1.519920 0.451721" + ] + }, + "execution_count": 36, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "from sklearn.decomposition import PCA\n", + "\n", + "# We want to be able to plot this on a 2-D scatter plot so we choose 2 Dimensions\n", + "dimension_we_want = 2\n", + "\n", + "pca = PCA(n_components=dimension_we_want)\n", + "X_2d = pca.fit_transform(X) \n", + "X_2d = pd.DataFrame(X_2d, columns=[\"PCA1\", \"PCA2\"])\n", + "\n", + "print(\"\\n% Variance Explained:\", pca.explained_variance_ratio_.sum(), end=\"\\n\\n\")\n", + "X_2d.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 296 + }, + "colab_type": "code", + "id": "Gj87JXmmBdNu", + "outputId": "4574186e-9f94-4b91-9cf9-ba4a68b9f866" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 37, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAATe0lEQVR4nO3dcWzcZ33H8c/nEtfxcLZGjpWucUMqUtCqYIKwOjZv0lrYKIgFtVk1OsHEYIuQhkQlpoSqbGOaJtGwMaSViUWUwaSKqpvbBdFWNFVgUAQFB7mmaQrqGKyOGA1eSmOwXaf33R93Xm3X8Tm5+93zu3veL8mS73eX+30vd/597nme3/P8HBECAOSnkroAAEAaBAAAZIoAAIBMEQAAkCkCAAAytTF1ARdi69atsXPnztRlAEBHOX78+E8iYnDl9o4KgJ07d2p8fDx1GQDQUWz/cLXtdAEBQKYIAADIFAEAAJkiAAAgUwQAAGSKAADQEtMz83rs6Wc1PTOfuhSsU0edBgqgnI5MnNLBsUn1VCpaqFZ1aN+w9u7ZnrosNEALAEBTpmfmdXBsUnMLVZ2dP6e5haoOjE3SEugABACApkydmVVPZfmhpKdS0dSZ2UQVYb0IAABNGdrSp4Vqddm2hWpVQ1v6ElWE9SIAADRloL9Xh/YNa1NPRZt7N2pTT0WH9g1roL83dWlogEFgAE3bu2e7Rndt1dSZWQ1t6ePg3yEIAAAtMdDfy4G/w9AFBACZIgAAIFPJAsD2JtvftP2Y7RO2/ypVLQCQo5RjAPOSrouIGds9kh6x/WBEfCNhTQCQjWQBEBEhaaZ+s6f+E6nqAYDcJB0DsL3B9oSkZyQdjYhHV3nMftvjtsdPnz7d/iIBoEslDYCIeCEi9kgaknSN7d2rPOZwRIxExMjg4EuuaQwAuEilOAsoIp6V9CVJ16euBQBykfIsoEHbl9Z/75P025KeTFUPAOQm5VlAvyzps7Y3qBZE90TEFxLWAwBZSXkW0KSk16baPwDkrhRjAACA9iMAACBTBAAAZIoAAIBMEQAAkCkCAAAyRQAAQKYIAADIFAEAAJkiAAAgUwQAAGSKAACATBEAAJApAgAAMkUAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMhUsgCwfYXtL9l+wvYJ2+9PVQsA5Ghjwn2fk/SBiPi27c2Sjts+GhFPJKwJALKRrAUQET+KiG/Xfz8r6aSk7anqAYDclGIMwPZOSa+V9Ogq9+23PW57/PTp0+0uDQC6VvIAsN0vaUzSLRHx3Mr7I+JwRIxExMjg4GD7CwSALpU0AGz3qHbwvysi7k1ZCwDkJuVZQJZ0p6STEfGxVHUAQK5StgBGJb1T0nW2J+o/b0lYDwBkJdlpoBHxiCSn2j8A5C75IDAAIA0CAAAyRQAAQKYIAADIFAEAAJkiAIAVpmfm9djTz2p6Zj51KUChUq4GCpTOkYlTOjg2qZ5KRQvVqg7tG9bePaxRiO5ECwCom56Z18GxSc0tVHV2/pzmFqo6MDZJSwBdiwAA6qbOzKqnsvxPoqdS0dSZ2UQVAcUiAIC6oS19WqhWl21bqFY1tKUvUUVAsQgAoG6gv1eH9g1rU09Fm3s3alNPRYf2DWugvzd1aUAhGAQGlti7Z7tGd23V1JlZDW3p4+CPrkYAACsM9Pdy4EcW6AICgEwRAACQKQIAQCGYUV1+jAEAaDlmVHcGWgB1fFsBWoMZ1Z2DFoD4tgK00uKM6jm9OKlucUY1Z1eVS/YtAL6tdB9ac2kxo7pzZBUAqx0YWP+luxyZOKXR24/pHZ96VKO3H9PnJ06lLik7zKjuHNl0AZ2vm4dvK91jaWtusfvhwNikRndt5eDTZsyo7gxZtADW6ubh20r3oDVXLgP9vXrNFZfyt1RiWbQAGg1K8W2lO9CaAy5M0haA7U/bfsb240XuZz0HBr6tdD5ac8CFSd0C+IykOyT9S5E7WTwwHFgxBsCBofvQmgPWL2kARMRXbO9sx744MOSD1TyB9UndAmjI9n5J+yVpx44dTT0XBwYAeFHpzwKKiMMRMRIRI4ODg6nLAdqOiW0oSulbAEDOWKYERSp9CwDIFcuUoGipTwP9nKSvS3qV7Snb70lZD1AmTGxD0VKfBXRzyv0DZcbENhSNLiCgpJjYVl7dMjDPIDBQYsxfKZ9uGpgnAICSY/5KeXTbirN0AQHAOnXbwDwBAADr9LJLNmj+3AvLtnXywDwBsIZuGejBi3hPcbGOTJzSW+94RJWKJUm9G9zxA/OMAZxHNw30oIb3FBdrad//orB1//t+Q7u2bU5YWXPW1QKw3bPKtq2tL6cYF/qtjxmY3Yf3FM1Yre+/d0NFP3v+hfP8i86wZgDYvtb2lKQf2X5oxdLNDxVZWKtczEXCu22gB7ynaE63Tspr1AI4JOlNEbFV0mFJR22/vn6fC62sBS72W1+3vtk54z3tTGUZs+nWSXmNxgAuiYgTkhQR/2b7pKR7bR+UFIVX16RG1wI+H64g1n14TztP2cZsunFSXqMAWLB9WUT8jyRFxAnbb5D0BUmvKLy6JjXzra8b3+zc8Z52jrJOuOq2SXmNuoA+KGnb0g0RMSXptyR9pKCaWqbZZhsXiu8+vKedgTGb9lizBRARD5/nrs2Snm99Oa3Ht758TM/M8z53CcZs2mPd8wBsD0q6SdLNki6XdF9RRbVatzXb8FJl6y9GcxizaY81A8D2Zkk3SvoDSa+UdK+kKyNiqA21AetS1v5iNIfWe/EatQCekfRNSR+S9EhEhO0bii8LWL+LPdsL5UfrvViNBoFvldQr6R8l3Wq79Gf+ID/0FwMXZ80AiIiPR8TrJb2tvunfJV1u+6DtVxZeHbAO3TpJByiaIy5sPpft3aoNBP9+ROwqpKrzGBkZifHx8XbuEh2Es4CA1dk+HhEjK7c3GgTeJWlbRHxtcVtEPG77QUn/3Poy0clSH4DpLwYuTKNB4I+rNg6w0k8l/b2k3215RehInIYJdJ5Gg8DbIuI7KzfWt+0spCJ0HJZaBjpTowC4dI37OMUCkso5bb8sq0gCZdYoAMZt/8nKjbb/WNLxZndu+3rb37X9lO0PNvt8SKNsp2FezDUggBw1CoBbJP2R7S/b/rv6z39Ieo+k9zezY9sbJH1C0pslXS3pZttXN/OcSKNMp2HSHQWsX6PF4H4s6ddtXytpd33z/RFxrAX7vkbSUxHxfUmyfbdq8w2eaMFzo83KMm2fWcEoo9RnyJ1Po9NAN0l6r6Rdkr4j6c6IONeifW+X9PSS21OSfnWVGvZL2i9JO3bsaNGuUYQynIZZtu4ooMxnyDXqAvqspBHVDv5vlvS3hVe0QkQcjoiRiBgZHBxs9+7RYcrUHQWUvUuy0TyAqyPi1ZJk+07VFoZrlVOSrlhye6i+DWhKWbqjgLJ3STa8JOTiLxFxzm7pdeC/Jekq21eqduB/u2rLTgNNK0N3FFD2LslGXUCvsf1c/eespOHF320/18yO62MJ75P0RUknJd2zeAF6AOgGZe+SbHQW0IYidx4RD0h6oMh9AEBKZe6SXPclIQEAF6esXZKNuoAAAF2KAACATBEAAJApAgAAMkUAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMgUAQAAmSIAACBTBABWNT0zr8eeflbTM/OpS0HG+BwWiyuC4SWOTJzSwbFJ9VQqWqhWdWjfsPbu2Z66LGSGz2HxaAFgmemZeR0cm9TcQlVn589pbqGqA2OTfANDW/E5bA8CAMtMnZlVT2X5x6KnUtHUmdlEFSFHfA7bgwDAMkNb+rRQrS7btlCtamhLX6KKOgt91q3B57A9CAAsM9Dfq0P7hrWpp6LNvRu1qaeiQ/uGNdDfm7q00jsycUqjtx/TOz71qEZvP6bPT5xKXVLH4nPYHo6I9u/UvknShyX9iqRrImJ8Pf9uZGQkxsfX9VA0aXpmXlNnZjW0pY8/unWYnpnX6O3HNLfw4rfWTT0Vfe3gdfz/NYHPYWvYPh4RIyu3pzoL6HFJN0r6p0T7RwMD/b0d8QdXlgPEYp/1nF4MgMU+6074fyyrTvkcdqokARARJyXJdordo0uU6TRB+qzRiRgDQEdq1WmCrRq0pc8anaiwFoDthyVdtspdt0XEkQt4nv2S9kvSjh07WlQdOl0rulxa3YLYu2e7RndtLUWXFLAehQVARLyxRc9zWNJhqTYI3IrnROdrtstlaQtiMUQOjE1qdNfWpg7c9Fmjk9AFhI7UbJcLE42ARIPAtm+Q9A+SBiXdb3siIt6UohZ0rma6XBi0BRK1ACLivogYiojeiNjGwR8Xa6C/V6+54tIL7nZh0BZgNVBkjEFb5I4AQNYYtEXOGAQGgEwRAACQKQIAADJFAABApggAACi5oi40xFlAAFBiRa56SwsAAEqqVaveng8BAAAlVfSaVQQAAJRU0WtWEQAAUFJFr1nFIDAAlFiRa1YRAABQckWtWUUXELJU1HnVQCehBYDsFHledSebnplnaezMEADISlHXAu50hGKe6AJCVrgW8EsVPdkI5UUAICtcC/ilCMV8EQDICtcCfilCMV+MASA7XAt4ucVQPLBiDCD3/5ccEADIEtcCXo5QzBMBAEASoZgjxgAAIFNJAsD2R20/aXvS9n22L01RBwDkLFUL4Kik3RExLOl7km5NVAcAZCtJAETEQxFxrn7zG5KGUtQBADkrwxjAuyU9eL47be+3PW57/PTp020sqxxYtAxAUQo7C8j2w5IuW+Wu2yLiSP0xt0k6J+mu8z1PRByWdFiSRkZGooBSS4v1WQAUqbAAiIg3rnW/7XdJequkN0REVgf29WDRMgBFS3UW0PWSDkjaGxE/T1FD2bE+C4CipRoDuEPSZklHbU/Y/mSiOkqL9VkAFC3JTOCI2JViv52E9VkAFI2lIEqM9VkAFIkAKDnWZwFQlDLMAwAAJEAAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMgUAQAAmSIAACBTBACyMD0zr8eeflbTM/OpSwFKg+Wg0fWOTJzSwRUX1tm7Z3vqsoDkaAGgq03PzOvg2KTmFqo6O39OcwtVHRibpCUAiABAl5s6M6ueyvKPeU+loqkzs4kqAsqDAEBXG9rSp4Vqddm2hWpVQ1v6ElUElAcBgK420N+rQ/uGtamnos29G7Wpp6JD+4a5zCYgBoGRgb17tmt011ZNnZnV0JY+Dv5AHQGALAz093LgB1agCwgAMpUkAGz/te1J2xO2H7J9eYo6ACBnqVoAH42I4YjYI+kLkv4iUR0AkK0kARARzy25+TJJkaIOAMhZskFg238j6Q8l/VTStWs8br+k/ZK0Y8eO9hQHABlwRDFfvm0/LOmyVe66LSKOLHncrZI2RcRfruM5T0v6YeuqLNRWST9JXUQb8Xq7W06vtxtf68sjYnDlxsICYL1s75D0QETsTlpIi9kej4iR1HW0C6+3u+X0enN6ranOArpqyc23SXoyRR0AkLNUYwAfsf0qSVXVunTem6gOAMhWkgCIiH0p9ttmh1MX0Ga83u6W0+vN5rUmHwMAAKTBUhAAkCkCAAAyRQAUyPZHbT9ZX/foPtuXpq6pSLZvsn3CdtV2V55GZ/t629+1/ZTtD6aup2i2P237GduPp66laLavsP0l20/UP8fvT11T0QiAYh2VtDsihiV9T9Ktiesp2uOSbpT0ldSFFMH2BkmfkPRmSVdLutn21WmrKtxnJF2fuog2OSfpAxFxtaTXS/rTbn9/CYACRcRDEXGufvMbkoZS1lO0iDgZEd9NXUeBrpH0VER8PyKel3S3avNYulZEfEXS/6auox0i4kcR8e3672clnZS0PW1VxSIA2ufdkh5MXQSasl3S00tuT6nLDxC5sr1T0mslPZq2kmJxRbAmrWfNI9u3qda8vKudtRVhvWs8AZ3Kdr+kMUm3rFi5uOsQAE2KiDeudb/td0l6q6Q3RBdMumj0ervcKUlXLLk9VN+GLmG7R7WD/10RcW/qeopGF1CBbF8v6YCkvRHx89T1oGnfknSV7SttXyLp7ZI+n7gmtIhtS7pT0smI+FjqetqBACjWHZI2Szpav/zlJ1MXVCTbN9iekvRrku63/cXUNbVSfUD/fZK+qNoA4T0RcSJtVcWy/TlJX5f0KttTtt+TuqYCjUp6p6Tr6n+vE7bfkrqoIrEUBABkihYAAGSKAACATBEAAJApAgAAMkUAAECmCABgFbZfqJ8G+Ljtf7X9C/Xtl9m+2/Z/2j5u+wHbr1zy726xPWf7l5ZsG6ivMjlj+44UrwdYDQEArG42IvZExG5Jz0t6b32i0H2SvhwRr4iI16m2wuu2Jf/uZtUmjN24ZNucpD+X9GftKR1YHwIAaOyrknZJulbSQkT8/4S+iHgsIr4qSbZfIalf0odUC4LFx/wsIh5RLQiA0iAAgDXY3qja+v/fkbRb0vE1Hv521ZaI/qpqM2e3rfFYIDkCAFhdn+0JSeOS/lu1NWIauVnS3RFRVW1BsZsKrA9oGquBAqubjYg9SzfYPiHp91Z7sO1XS7pKtXWfJOkSSf+l2npQQCnRAgDW75ikXtv7FzfYHrb9m6p9+/9wROys/1wu6XLbL09VLNAIi8EBq7A9ExH9q2y/XNLHJb1OtUHdH0i6RbUVQt8SEU8ueezHJP04Im63/QNJv6hay+BZSb8TEU8U/TqAtRAAAJApuoAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMjU/wFur9UWE4oSkQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light", + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "X_2d.plot.scatter(x=\"PCA1\", y=\"PCA2\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "cT7Rv9lEDSji" + }, + "source": [ + "As we can see from above, the two new features (known as Principle Components) are not like any of our input features. Additionally, these two new components explains 64.6% of all the original variation. Let's see how well this performs in explaining our dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "eBHVGhd-CJF8" + }, + "outputs": [], + "source": [ + "reg = LinearRegression()\n", + "\n", + "reg.fit(X, y)\n", + "print(\"All Features:\\t\\t\", reg.score(X, y))\n", + "reg.fit(X_2d, y)\n", + "print(\"With PCA Features:\\t\", reg.score(X_2d, y), end=\"\\n\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "bsoN_a8XI4p2" + }, + "source": [ + "As expected, with a higher explained variance ratio, we perform better in predicting the quality of the wine. With Dimensionality Reduction, we were able to capture most of the variation in the dataset while still being able to view it in two dimensions. In the most basic of terms, PCA creates a variable projected along the axis of maximum variable.\n", + "\n", + "![](https://raw.githubusercontent.com/bfkwong/data/master/IMG_0187%202.jpg)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "vwDiZRX3JYLN" + }, + "source": [ + "### Optional: Linear Algebra Behind PCA\n", + "\n", + "Principle Component Analysis chooses principle components along te **axis of greatest variance**. In Linear Algebra terms, the axis of greatest variance is the **Eigenvector with the largest Eigenvalue of the covariance matrix ($\\Sigma$)**\n", + "\n", + "The following are the steps in order to do PCA manually. \n", + "\n", + "1. Given data matrix $M$, generate the covariance matrix of $M$ denoted as $\\Sigma$\n", + "2. We then compute the Eigenvector and Eigenvalue of covariance matrix $\\Sigma$\n", + "3. Project the features to the Eigenvector with the largest Eigenvalue using the dot product (cross product for more than 1 dimensions)" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "J26dEAoAD0q7" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "# Step 1: Calculate covariance matrix \n", + "cov_mtrx = np.cov(X.T)\n", + "\n", + "# Step 2: Calculate Eigenvector and Eigenvalue\n", + "W,v = np.linalg.eig(cov_mtrx)\n", + "\n", + "# Step 3: Find the largest Eigenvalue and project our data onto the corresponding Eigenvector\n", + "idx_largest_eigenval = np.argmax(W)\n", + "eigenvec = v[:,idx_largest_eigenval]\n", + "\n", + "total = []\n", + "for row in X.index: \n", + " total.append(np.dot(X.loc[row], eigenvec))\n", + "\n", + "pd.DataFrame(pd.Series(total), columns=[\"PCA1\"]).head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "_G7NWh_1nf_k" + }, + "source": [ + "One interesting thing you may see here is that the eigenvalue corresponds to the variance explained by its corresponding eigenvalue. The eigenvalue of PCA1 is 2.26 and the variance of PCA1 is also 2.26" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "9QCALn43iTYd" + }, + "outputs": [], + "source": [ + "idx_largest_eigenval = np.argmax(W)\n", + "variance = pd.Series(total).var()\n", + "\n", + "print(\"Largest Eigenvalue:\\t\", W[idx_largest_eigenval])\n", + "print(\"Variance of Eigenvector:\", variance)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "bmLOaYzvEeG7" + }, + "source": [ + "### Multidimensional Scaling (MDS)\n", + "\n", + "We now pay a visit to our good friend, the Euclidean distance. One incredibly useful aspect of Euclidean distance is that it works in higher dimensions. The formula $$\\sqrt{x_1^2 + x_2^2}$$ is for two dimensional Euclidean distance, but to move it to a third dimension, it is as easy as adding a $x^3$ variable. One thing to recognize is that Euclidean distance in all dimension is still a number. \n", + "\n", + "Why am I rambling on about something you learned in middle school? Well the realization that Euclidean distance is scalar in all dimensions means that we can preserve the variance of n-th dimensional data in two dimensions as long as we try to ensure that the Euclidean distances in the n-th dimensional is proportion to the Euclidean distance in 2 dimensions. That was a lot to take in, the following image explains the concept.\n", + "\n", + "![](https://raw.githubusercontent.com/bfkwong/data/master/IMG_0188.jpg)\n", + "\n", + "Notice how when we reduced our dimensions from 2 to 1, the distances between points A, B, and C remained the same. Meaning that x, y, and z remained the same between the two dimensions. While distances may not always be preserved perfectly between dimensions, MDS attempts to preserve it as well as possible. " + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "L4AjVFrwn78I" + }, + "outputs": [], + "source": [ + "from sklearn.manifold import MDS\n", + "# We want to be able to plot this on a 2-D scatter plot so we choose 2 Dimensions\n", + "dimension_we_want = 2\n", + "\n", + "mds = MDS(n_components=dimension_we_want)\n", + "X_2d = mds.fit_transform(X) \n", + "X_2d = pd.DataFrame(X_2d, columns=[\"Dimension 1\", \"Dimension 2\"])\n", + "\n", + "display(X_2d.head())\n", + "print(\"Percentage variance explained:\", X_2d.var().sum()/X.var().sum())\n", + "X_2d.plot.scatter(x=\"Dimension 1\", y=\"Dimension 2\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "mlIA-o_cEnSD" + }, + "source": [ + "By using MDS, we were actually able to preserve over 95% of the variance from the original datasets. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "88kI9JxzFO6n" + }, + "source": [ + "### Linear vs Nonlinear Dimensionality Reduction \n", + "\n", + "Thus far, we have explored PCA (a linear reduction technique) and MDS (a nonlinear reduction technique). While there are a lot of differences between the two methods. The key differences could be boiled down to just the following statement: **linear dimensionality reduction technique only stretch and shift the data while nonlinear techniques make more drastic changes to the data**.\n", + "\n", + "This sometimes leads to nonlinear techniques being better at capturing variance but losing the overall shape of the data whereas linear techniques are better at keeping the general shape of the original data but loses more variance along the way. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "7uprQWlUGC37" + }, + "source": [ + "# Exercises" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "UnMJKgsTkDoZ" + }, + "source": [ + "1. Consider the Iris dataset (https://raw.githubusercontent.com/dlsun/pods/master/data/iris.csv). Drop the \"SepalWidth\" and \"PedalWidth\" columns and then apply PCA on \"SepalLength\" and \"PedalLength\" with `n_components = 2`. How many percent of the variance was PCA able to capture in this case? What happens when we use PCA to compress 2D data into 2D data?" + ] + }, + { + "cell_type": "code", + "execution_count": 0, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "HCXKtA8AlSQq" + }, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "7.2 Visualizing Higher Dimensions.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.4" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/07-Unsupervised-Learning/7_2_Visualizing_Higher_Dimensions.ipynb b/07-Unsupervised-Learning/7_2_Visualizing_Higher_Dimensions.ipynb deleted file mode 100644 index c2cb755..0000000 --- a/07-Unsupervised-Learning/7_2_Visualizing_Higher_Dimensions.ipynb +++ /dev/null @@ -1,731 +0,0 @@ -{ - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "name": "7.2 Visualizing Higher Dimensions.ipynb", - "provenance": [], - "collapsed_sections": [] - }, - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - } - }, - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "XTPLBCWKzfWS", - "colab_type": "text" - }, - "source": [ - "# 7.2 Visualizing Higher Dimensions\n", - "\n", - "By this point, we hope we've convinced you how important it is to visualize your data. While summary statistics are helpful, it doesn't provide us with a good grasp of what the entire dataset looks like. In two dimension, we can use a 2-D scatter plot. In three dimension, we can use a 3-D scatter plot. But what if we have more than three dimensions? This chapter talks about how we can visualize data that is beyond 3 dimensions." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "f1mm_fCWm9dO", - "colab_type": "text" - }, - "source": [ - "## Goal of Visualizing Higher Dimensions\n", - "\n", - ">\"I am a Tralfamadorian, seeing all time as you might see a stretch of the Rocky Mountains. All time is all time. It does not change. It does not lend itself to warnings or explanations. It simply is.\" \n", - ">\n", - ">-Kurt Vonnegate in \"Slaughterhour-Five\"\n", - "\n", - "We unfortunately are not Tralfamadorians, instead we are three dimensional beings who can't visually see a fourth dimension like its a location on the Rocky Mountains. However, this doesn't mean that the fourth dimension is meaingless to us. We can derive a lot of understand from understanding the higher dimensions. The problem is, we can't it and thus we can't plot it. \n", - "\n", - "Fortunately, very clever mathematicians throughout history has invented techniques to allow us to simulate what the higher dimension would look like. The rest of the section will discuss these techniques." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "XR-JkcQuwozv", - "colab_type": "text" - }, - "source": [ - "## Using Size and Color\n", - "\n", - "This is more of a review from previous sections but one way to visualize more dimensions is by using the size and color attributes of your scatter plots. This is rather intuitive. " - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "kgChDoxMwyKa", - "colab_type": "code", - "outputId": "cc578404-1e67-476a-f736-e070bc9c233e", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 368 - } - }, - "source": [ - "import pandas as pd\n", - "from scipy import stats\n", - "from sklearn.linear_model import LinearRegression\n", - "from sklearn.preprocessing import LabelEncoder, StandardScaler\n", - "import altair as alt\n", - "\n", - "df_bordeaux = pd.read_csv(\"http://dlsun.github.io/pods/data/bordeaux.csv\")\n", - "\n", - "alt.Chart(df_bordeaux).mark_circle().encode(\n", - " alt.X('age',\n", - " scale=alt.Scale(zero=False)\n", - " ),\n", - " alt.Y('sep',\n", - " scale=alt.Scale(zero=False)\n", - " ),\n", - " color=\"summer\",\n", - " size=\"win\"\n", - ")" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "alt.Chart(...)" - ], - "text/html": [ - "\n", - "
\n", - "" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 30 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "gxV83MG_y2LJ", - "colab_type": "text" - }, - "source": [ - "I am sure you can see the limitations of this method: you can only go up for 4 dimensions (5 if you use a 3-D scatter plot). This is still worth mentioning as sometimes, this may be all you need. \n", - "\n", - "For higher dimensions, we should consider either feature selection or feature reduction. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "tlWDTyDsr5Zv", - "colab_type": "text" - }, - "source": [ - "## Feature Selection \n", - "\n", - "If we want to compress 10 dimensions worth of data into 2 dimensions, we're bound to lose some detail during that compression. We can measure how much detail we kept at the end with the explained variance ratio which gives the percentage of variance/detail we kept after the compression. The higher the ratio, the more variance and detail we kept. \n", - "\n", - "One way very simple, almost trivial, way to only visualize higher dimensional data is to only plot the two dimensions that explains the most variation in the data. " - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "g9kxLPD30dxa", - "colab_type": "code", - "outputId": "c1aedb62-573b-406c-cfa6-844ab4054122", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 136 - } - }, - "source": [ - "sclr = StandardScaler()\n", - "df_bordeaux = pd.DataFrame(sclr.fit_transform(df_bordeaux.dropna()), columns=df_bordeaux.columns)\n", - "\n", - "X = df_bordeaux.drop(\"price\", axis=1)\n", - "y = df_bordeaux[\"price\"]\n", - "\n", - "reg = LinearRegression()\n", - "reg.fit(X, y)\n", - "\n", - "scores = pd.Series(dtype=float, name=\"R^2 Values\")\n", - "for column in X.columns: \n", - " reg = LinearRegression()\n", - " reg.fit(X[[column]], y)\n", - " scores[column] = reg.score(X[[column]], y)\n", - "\n", - "scores.sort_values(ascending=False)" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "summer 0.343538\n", - "sep 0.323582\n", - "age 0.206936\n", - "year 0.206936\n", - "har 0.199621\n", - "win 0.053456\n", - "Name: R^2 Values, dtype: float64" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 31 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "qHprVgDLxKyE", - "colab_type": "text" - }, - "source": [ - "As we can see above, average summer temperature **summer** and average september temperature **sep** are the two variables that explain the most variance in the quality of the wine **price**. Thus, if we want to get the best representation of the dataset with only two dimensions, we can make a scatterplot of **summer** vs **sep**. However, even with the two variables that explain the most variation, we can only capture 33% of the variation of the original data. The other 66% is lost to the other features we chose to ignore. " - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "FcUCeYGOx2nP", - "colab_type": "code", - "outputId": "16dc19dd-f2ac-44b7-8ac6-0236cac283d4", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 330 - } - }, - "source": [ - "explained_var = X[[\"summer\", \"sep\"]].var(axis=0).sum() / X.var(axis=0).sum() \n", - "\n", - "print(\"% Variance Explained:\", explained_var, end=\"\\n\\n\")\n", - "df_bordeaux.plot.scatter(x=\"summer\", y=\"sep\")" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "% Variance Explained: 0.3333333333333333\n", - "\n" - ], - "name": "stdout" - }, - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 32 - }, - { - "output_type": "display_data", - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAUNklEQVR4nO3dfYxcV33G8efZeLu26rQYrwvEDpiQqLwaJ2zTgCmF8NKQVqZgKFDRgqByU4RU1Ao7CLX0TY1spP5RUVqshAKCBigmtVsCIWAoL2rSrINf8kJCQEmzbkrM1glZsLfrzK9/zF083njt2ezcOefO+X6kVWbvzO793ZvxPHvOPedcR4QAAOUZSl0AACANAgAACkUAAEChCAAAKBQBAACFWpK6gIUYHR2NtWvXpi4DABpl7969P4yIVXO3NyoA1q5dq/Hx8dRlAECj2L7vVNvpAgKAQhEAAFAoAgAACkUAAEChCAAAKBQBAABzTE5Na//9D2lyajp1KbVq1DBQAKjbrn2HtHXnAQ0PDWmm1dL2Teu0cf3q1GXVghYAAFQmp6a1decBHZtp6ZHp4zo209KWnQcGtiVAAABAZeLIUQ0PnfyxODw0pIkjRxNVVC8CAAAqa1Ys00yrddK2mVZLa1YsS1RRvQgAAKisXD6i7ZvWaenwkM4eWaKlw0PavmmdVi4fSV1aLbgIDAAdNq5frQ3nj2riyFGtWbFsYD/8JQIAAB5j5fKRgf7gn0UXEAAUigAAgEIRAABQKAIAAApFAABAoQgAACgUAQAAhSIAAKBQBAAAFIoAAIBCEQAAUCgCAAAKlSwAbJ9r+6u277B9u+0/TFULAJQo5WqgxyX9cUTcavtsSXtt3xgRdySsCQCKkawFEBEPRMSt1eNHJN0paTDvvAwAGcriGoDttZIulHTzKZ7bbHvc9vjhw4f7XRoADKzkAWB7uaSdkt4dET+a+3xE7IiIsYgYW7VqVf8LBIABlTQAbA+r/eH/yYj4XMpaAKA0KUcBWdI1ku6MiL9JVQcAlCplC2CDpN+RdKntfdXX5QnrAYCiJBsGGhHflORU+weA0iW/CAwASIMAAIBCEQAAUCgCAAAKRQAAQKEIAAAoFAEAAIUiAACgUAQAABSKAACAQhEAAFAoAgAACkUAAEChCACgzyanprX//oc0OTWduhQULtly0ECJdu07pK07D2h4aEgzrZa2b1qnjetXpy4LhaIFAPTJ5NS0tu48oGMzLT0yfVzHZlrasvMALQEkQwAAfTJx5KiGh07+Jzc8NKSJI0cTVYTSEQBAn6xZsUwzrdZJ22ZaLa1ZsSxRRSgdAQD0ycrlI9q+aZ2WDg/p7JElWjo8pO2b1mnl8pHUpaFQXAQG+mjj+tXacP6oJo4c1ZoVy/jwR1IEANBnK5eP8MGPLNAFBACFIgAAoFAEAADMUcpsba4BAECHkmZr0wIAgEpps7UJAACo9Gu2di5dTHQBAUClH7O1c+piogUAAJW6Z2vn1sVECwAAOtQ5W3u2i+mYTrQyZruYUkwOJAAAYI66ZmvntiAgXUAA0Ce5LQhICwAA+iinBQEJAADos1wWBKQLCAAKlTQAbH/E9oO2b0tZBwCUKHUL4KOSLktcAxool5mU/VDSsaK/kl4DiIiv216bsgY0T04zKetW0rGi/1K3AM7I9mbb47bHDx8+nLocJJbbTMo6lXSsSCP7AIiIHRExFhFjq1atSl0OEuvXYl05KOlYkUb2AQB0ym0mZZ1KOlakQQCgUXKbSVmnko4VaTgi0u3cvlbSSyWNSvqBpPdHxDXzvX5sbCzGx8f7VB1yNjk1ncVMyn4o6VhRD9t7I2Js7vbUo4DenHL/aK5cZlL2Q0nHiv6iCwgACkUAAEChCAAAKBQBAACFIgAAoFAEAFAwFporGzeEAQrFQnOgBQAUiIXmIBEAQJEGYaE5uq8Wjy4goEBNX2iO7qveoAUAFKjJC83RfdU7tACAQm1cv1obzh9t3EJzs91Xx3SiBTPbfdWUY8gFAQAUrIkLzTW9+yondAEBaJQmd1/lhhYAgMZpavdVbggAAI3UxO6r3NAFBACFIgAAoFAEAAAUigAAgEIRAABQKAIAAApFAABAoRYUALZ/zvbZdRUDAOifrgLA9i/ZPijpgKTbbO+3/YJ6S0PTsV47kLduZwJfI+mdEfENSbL9Ykn/KGldXYWh2VivHchft11Aj85++EtSRHxT0vF6SkLTsV470AzdtgD+3faHJV0rKSS9UdLXbF8kSRFxa031oYFYrx1ohm4D4PnVf98/Z/uFagfCpT2rCI03COu1T05NF7HSZCnHiVPrKgAi4mV1F4LBMbte+5Y51wCa8gFTyvWLUo4T83NEnPlF9pMk/bWkcyLi1bafLemFEXFN3QV2Ghsbi/Hx8X7uEovQxL8uJ6emtWHbHh2bOdGCWTo8pG9tvbQxx9CNUo4Tbbb3RsTY3O3dXgT+qKQbJJ1TfX+3pHf3pjQMqpXLR/T8c5/QqA+U2esXnWavX8ynicNdH89xPh5NPDcl6fYawGhEfMb2eyUpIo7bfrTGuoAkFnr9oqndKP24TtPUc1OSblsAP7a9Uu0LvrJ9iaSHa6sKSGQh95tt8nDXuu+r2+RzU5JuWwB/JGm3pGfY/pakVZJeX1tVQELd3m+26cNd67yvbtPPTSm6DYBnSHq1pHMlbZL0ywv4WaBxurnf7CAMd63rvrqDcG5K0G0X0J9ExI8krZD0MkkfkvT3i9257cts32X7HttXLvb3Af1UdzdKk3FumqHbYaDfjogLbV8l6WBE/NPstse9Y/sstUcTvVLShKRbJL05Iu6Y72cYBoocNXG4a79wbvIw3zDQbrtxDlVLQbxS0jbbI1r8vQQulnRPRHy/KvBTkl4jad4AAHJUVzfKIODc5K3bD/HfUnsewK9FxEOSnijpPYvc92pJ93d8P1FtO4ntzbbHbY8fPnx4kbsEAMzqdimIn0j6XMf3D0h6oK6i5ux7h6QdUrsLqB/7BIASpLwl5CG1RxXNWlNtA5ApZvYOlpRDOW+RdIHtp6v9wf8mSb+dsB4Ap8HM3sGTrAUQEcclvUvtawt3SvpMRNyeqh4A82Nm72BKOpkrIq6XdH3KGkrGED10K8eZvbx/F4/ZvIWiOY+FyG1mL+/f3kh5ERiJ0JzHQuU0s5f3b+/QAihQjs155K/OxeMWgvdv7xAABcqtOY/myGFmL+/f3qELqEA5NeeBheL92ztdLQaXCxaD6y1GUTQD/59OjfPSvcUuBocBlENzHqfHaJf58f5dPLqAgEwx2gV1IwCATM2Oduk0O9oF6AUCADiFHBY9Y7QL6sY1AGCOXPrdZ0e7bJlTC/3e6BUCAOjQ2e8+O9Foy84D2nD+aJIP3lwmX2EwEQBAhxxnmTLaBXXhGgDQYc2KZTo6c/ykbUdnjtPvjoFEAABz2D7t98CgIACADhNHjmrpkrNO2rZ0yVkMvcRAIgCADgy9REkIAKADC42hJIwCQteavPjWQmpf6NDLJp8XlI0AQFdymRz1eDye2rsdetnk8wLQBYQzavKiZHXW3uTzAkgEALrQ5EXJ6qy9yecFkAgAdKHJI2PqrL3J5wXNUtfihAQAzqjJI2PqrL3J5wXNsWvfIW3Ytkdvufpmbdi2R7v3HerZ7+aWkOhak0e71Fl7k88L8jY5Na0N2/bo2MyJlubS4SF9a+ulC3qvcUtILFqTFyWrs/Ymnxfkre7FCekCAoBM1X2diQAAgEzVfZ2JLiAAyFidNwUiAAAgc3VdZ6ILCAAKRQAAfVbXpB5goegCAvqIxeOQE1oAQJ+weBxyQwAAfcLicchNkgCw/Qbbt9tu2X7M9GRgELF4HHKTqgVwm6TXSfp6ov0DfcficchNkovAEXGnJNlOsXsgmTon9QALlf0oINubJW2WpKc+9amJqwEWj8XjkIvaAsD2lyU9+RRPvS8idnX7eyJih6QdUns56B6VBwDFqy0AIuIVdf1uAMDiMQwUAAqVahjoa21PSHqhpM/bviFFHQBQslSjgK6TdF2KfQMA2ugCAoBCEQAAUCgCAAAKRQAAQKEIAAAoFAEAAIUiAACgUAQAABSKAACAQhEAAFAoAgAACkUAAEChCAAAKBQBAACFIgAAoFAEAAAUigAAgEIRAABQKAIAAApFAABAoQiAzE1OTWv//Q9pcmo6dSkABsyS1AVgfrv2HdLWnQc0PDSkmVZL2zet08b1q1OXBWBA0ALI1OTUtLbuPKBjMy09Mn1cx2Za2rLzAC0BAD1DAGRq4shRDQ+d/L9neGhIE0eOJqoIwKAhADK1ZsUyzbRaJ22babW0ZsWyRBUBGDQEQKZWLh/R9k3rtHR4SGePLNHS4SFt37ROK5ePpC4NwIAo4iLw5NS0Jo4c1ZoVyxr1Abpx/WptOH+0kbUDyN/AB0DTR9KsXD7CBz+AWgx0FxAjaQBgfgMdAIykAYD5DXQAMJIGAOY30AHASBoAmN/AXwRmJA0AnNrAB4DESBoAOJWB7gICAMyPAACAQiUJANsfsP0d2wdsX2f7CSnqANBc3Ctj8VK1AG6U9NyIWCfpbknvTVQHgAbate+QNmzbo7dcfbM2bNuj3fsOpS6pkZIEQER8KSKOV9/eJGlNijoANA8z/Hsnh2sAb5f0hfmetL3Z9rjt8cOHD/exLAA5YoZ/79Q2DNT2lyU9+RRPvS8idlWveZ+k45I+Od/viYgdknZI0tjYWNRQKoAGYYZ/79QWABHxitM9b/ttkn5D0ssjgg92AF2ZneG/Zc4qv8z1WbgkE8FsXyZpi6RfjYifpKgBQHMxw783Us0E/qCkEUk32pakmyLiikS1AGggZvgvXpIAiIjzU+wXAHBCDqOAAAAJEAAAUCgCAAAKRQAAQKHcpCH4th+RdFfqOuYYlfTD1EWcQo515ViTlGddOdYk5VlXjjVJedX1tIhYNXdj024Ic1dEjKUuopPt8dxqkvKsK8eapDzryrEmKc+6cqxJyreuTnQBAUChCAAAKFTTAmBH6gJOIceapDzryrEmKc+6cqxJyrOuHGuS8q3rpxp1ERgA0DtNawEAAHqEAACAQmUdAN3ePN72vbYP2t5nezyTmi6zfZfte2xfWWdN1f7eYPt22y3b8w496/O56ramfp+rJ9q+0fZ3q/+umOd1j1bnaZ/t3TXVctpjtz1i+9PV8zfbXltHHQus6W22D3ecm9/rQ00fsf2g7dvmed62/7aq+YDti+quqcu6Xmr74Y5z9af9qKtrEZHtl6RXSVpSPd4mads8r7tX0mguNUk6S9L3JJ0n6Wck7Zf07JrrepakX5T0NUljp3ldP8/VGWtKdK62S7qyenzlad5XUzXXccZjl/ROSf9QPX6TpE9nUNPbJH2wH++hjn2+RNJFkm6b5/nL1b61rCVdIunmTOp6qaR/6+e5WshX1i2AyPDm8V3WdLGkeyLi+xHxf5I+Jek1Ndd1Z0RkNUu6y5r6fq6q3/+x6vHHJP1mzfubTzfH3lnrZyW93NVNNBLW1HcR8XVJ/3ual7xG0sej7SZJT7D9lAzqylrWATDH6W4eH5K+ZHuv7c0Z1LRa0v0d309U23KQ6lzNJ8W5elJEPFA9/h9JT5rndUttj9u+yXYdIdHNsf/0NdUfHg9LWllDLQupSZI2VV0tn7V9bo31dCvnf3MvtL3f9hdsPyd1MZ2SLwXRo5vHvzgiDtn+BbXvMvadKplT1tRz3dTVhb6fqxROV1fnNxERtucbC/206lydJ2mP7YMR8b1e19pA/yrp2oiYtv37ardQLk1cU65uVft9NGX7ckn/IumCxDX9VPIAiB7cPD4iDlX/fdD2dWo3Yx/3h1oPajokqfOvojXVtkU5U11d/o6+nqsu9P1c2f6B7adExANVN8GD8/yO2XP1fdtfk3Sh2v3jvdLNsc++ZsL2Ekk/L2myhzUsuKaI6Nz/1WpfU0mtlvfRYkXEjzoeX2/7Q7ZHIyKLReKy7gLyiZvHb4x5bh5v+2dtnz37WO2LtKe8It+vmiTdIukC20+3/TNqX7yrZRTJQvT7XHUpxbnaLemt1eO3SnpMS8X2Ctsj1eNRSRsk3dHjOro59s5aXy9pz3x/CPWrpjl96xsl3VljPd3aLel3q9FAl0h6uKObLxnbT569ZmP7YrU/c+sM8IVJfRX6dF+S7lG7X29f9TU7GuIcSddXj89Te6TCfkm3q931kLSm6vvLJd2t9l+MtdZU7e+1avd7Tkv6gaQbMjhXZ6wp0blaKekrkr4r6cuSnlhtH5N0dfX4RZIOVufqoKR31FTLY45d0l+o/QeGJC2V9M/V++4/JZ3Xh/Nzppquqt4/+yV9VdIz+1DTtZIekDRTvafeIekKSVdUz1vS31U1H9RpRsL1ua53dZyrmyS9qB91dfvFUhAAUKisu4AAAPUhAACgUAQAABSKAACAQhEAAFAoAgAACkUAAJmpJjPxbxO1402GIlWzoj9fLdJ1m+03un2vhNHq+bFq+QfZ/jPbH7P9Ddv32X6d7e1u31fhi7aHq9fda/uqat33cdsX2b7B9vdsX9Gx7/fYvqVaTO3Pq21r3V6D/+Nqz87OYYE1DDgCAKW6TNJ/R8TzI+K5kr54htc/Q+0FzzZK+oSkr0bE8yQdlfTrHa/7r4hYL+kbkj6q9vINl0ia/aB/ldqLgV0sab2kF9h+SfWzF0j6UEQ8JyLuW/whAqdHAKBUByW90vY2278SEQ+f4fVfiIiZ6ufO0onAOChpbcfrdndsvzkiHomIw5Km3b573Kuqr2+rvVLkM3Vidcj7or2WPdAXyVcDBVKIiLvdvm3g5ZL+yvZX1F7ee/aPoqVzfmS6+rmW7Zk4sYZKSyf/O5ru2D7dsX32dZZ0VUR8uPOXu32rxx8v5piAhaIFgCLZPkfSTyLiE5I+oPZt/e6V9ILqJZtq2vUNkt5ue3lVx+rq3gxA39ECQKmeJ+kDtltqr+T4B5KWSbrG9l+qfR/jnouIL9l+lqT/qFYJnpL0FkmP1rE/4HRYDRQACkUXEAAUigAAgEIRAABQKAIAAApFAABAoQgAACgUAQAAhfp/rOG9WPsCRBEAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "tags": [], - "needs_background": "light" - } - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Y6gWdO1CINp-", - "colab_type": "text" - }, - "source": [ - "Additionally, if we look below, using only two features has hindered our predictive accuracy. This sucks! Fortunately, some very clever mathematicians came up with ways to get around this. " - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "l7re2E6KHUbs", - "colab_type": "code", - "outputId": "24c2f5c2-ff71-420e-874c-2bbdee6f9a8f", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 68 - } - }, - "source": [ - "reg = LinearRegression()\n", - "\n", - "reg.fit(X, y)\n", - "print(\"All Features:\\t\\t\", reg.score(X, y))\n", - "reg.fit(X[[\"summer\", \"sep\"]], y)\n", - "print(\"With PCA Features:\\t\", reg.score(X[[\"summer\", \"sep\"]], y), end=\"\\n\\n\")" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "All Features:\t\t 0.7526018827767169\n", - "With PCA Features:\t 0.4633153344681292\n", - "\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "tflTZWS688xW", - "colab_type": "text" - }, - "source": [ - "## Dimensionality Reduction\n", - "\n", - "With feature selection, we were only able to capture 33% of the original variance, which isn't great. To capture more variation while still remaining in two variables, have to utilize some clever math.\n", - "\n", - "These clever mathematical techniques are known as feature creation, where we try to create new variables that helps us visualize higher dimensional data. There are many different techniques for dimensionality reduction all of which attempts to accomplish different things. Let's start off with the simplest and most popular one: **Principle Component Analysis**." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "cZ52OQlGAm6Q", - "colab_type": "text" - }, - "source": [ - "### Principle Component Analysis (PCA)\n", - "\n", - "Simply put, Principle Component Analysis create new features that maximizes variation. What PCA is **not** doing is feature selection, rather it is creating an entirely new arbitrary feature that is a combination of all the features. \n", - "\n", - "PCA involves some simple linear algebra, but SciKit-Learn has a PCA implementation. Note that all dimensionality reduction algorithms in SciKit-Learn is operated very similarity to machine learning algorithms you learned in the previous chapters. Create the object and then run `fit()` or `fit_transform()`." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "FESc4NoI8-3v", - "colab_type": "code", - "outputId": "1d1b00d1-65f5-40dd-ca5f-93f9ba84ef7c", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 255 - } - }, - "source": [ - "from sklearn.decomposition import PCA\n", - "\n", - "# We want to be able to plot this on a 2-D scatter plot so we choose 2 Dimensions\n", - "dimension_we_want = 2\n", - "\n", - "pca = PCA(n_components=dimension_we_want)\n", - "X_2d = pca.fit_transform(X) \n", - "X_2d = pd.DataFrame(X_2d, columns=[\"PCA1\", \"PCA2\"])\n", - "\n", - "print(\"\\n% Variance Explained:\", pca.explained_variance_ratio_.sum(), end=\"\\n\\n\")\n", - "X_2d.head()" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "stream", - "text": [ - "\n", - "% Variance Explained: 0.6462543462926849\n", - "\n" - ], - "name": "stdout" - }, - { - "output_type": "execute_result", - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
PCA1PCA2
02.620822-1.437859
12.1664460.732196
22.359265-0.067934
31.518218-0.792417
41.5199200.451721
\n", - "
" - ], - "text/plain": [ - " PCA1 PCA2\n", - "0 2.620822 -1.437859\n", - "1 2.166446 0.732196\n", - "2 2.359265 -0.067934\n", - "3 1.518218 -0.792417\n", - "4 1.519920 0.451721" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 36 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "Gj87JXmmBdNu", - "colab_type": "code", - "outputId": "4574186e-9f94-4b91-9cf9-ba4a68b9f866", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 296 - } - }, - "source": [ - "X_2d.plot.scatter(x=\"PCA1\", y=\"PCA2\")" - ], - "execution_count": 0, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 37 - }, - { - "output_type": "display_data", - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAATe0lEQVR4nO3dcWzcZ33H8c/nEtfxcLZGjpWucUMqUtCqYIKwOjZv0lrYKIgFtVk1OsHEYIuQhkQlpoSqbGOaJtGwMaSViUWUwaSKqpvbBdFWNFVgUAQFB7mmaQrqGKyOGA1eSmOwXaf33R93Xm3X8Tm5+93zu3veL8mS73eX+30vd/597nme3/P8HBECAOSnkroAAEAaBAAAZIoAAIBMEQAAkCkCAAAytTF1ARdi69atsXPnztRlAEBHOX78+E8iYnDl9o4KgJ07d2p8fDx1GQDQUWz/cLXtdAEBQKYIAADIFAEAAJkiAAAgUwQAAGSKAADQEtMz83rs6Wc1PTOfuhSsU0edBgqgnI5MnNLBsUn1VCpaqFZ1aN+w9u7ZnrosNEALAEBTpmfmdXBsUnMLVZ2dP6e5haoOjE3SEugABACApkydmVVPZfmhpKdS0dSZ2UQVYb0IAABNGdrSp4Vqddm2hWpVQ1v6ElWE9SIAADRloL9Xh/YNa1NPRZt7N2pTT0WH9g1roL83dWlogEFgAE3bu2e7Rndt1dSZWQ1t6ePg3yEIAAAtMdDfy4G/w9AFBACZIgAAIFPJAsD2JtvftP2Y7RO2/ypVLQCQo5RjAPOSrouIGds9kh6x/WBEfCNhTQCQjWQBEBEhaaZ+s6f+E6nqAYDcJB0DsL3B9oSkZyQdjYhHV3nMftvjtsdPnz7d/iIBoEslDYCIeCEi9kgaknSN7d2rPOZwRIxExMjg4EuuaQwAuEilOAsoIp6V9CVJ16euBQBykfIsoEHbl9Z/75P025KeTFUPAOQm5VlAvyzps7Y3qBZE90TEFxLWAwBZSXkW0KSk16baPwDkrhRjAACA9iMAACBTBAAAZIoAAIBMEQAAkCkCAAAyRQAAQKYIAADIFAEAAJkiAAAgUwQAAGSKAACATBEAAJApAgAAMkUAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMhUsgCwfYXtL9l+wvYJ2+9PVQsA5Ghjwn2fk/SBiPi27c2Sjts+GhFPJKwJALKRrAUQET+KiG/Xfz8r6aSk7anqAYDclGIMwPZOSa+V9Ogq9+23PW57/PTp0+0uDQC6VvIAsN0vaUzSLRHx3Mr7I+JwRIxExMjg4GD7CwSALpU0AGz3qHbwvysi7k1ZCwDkJuVZQJZ0p6STEfGxVHUAQK5StgBGJb1T0nW2J+o/b0lYDwBkJdlpoBHxiCSn2j8A5C75IDAAIA0CAAAyRQAAQKYIAADIFAEAAJkiAIAVpmfm9djTz2p6Zj51KUChUq4GCpTOkYlTOjg2qZ5KRQvVqg7tG9bePaxRiO5ECwCom56Z18GxSc0tVHV2/pzmFqo6MDZJSwBdiwAA6qbOzKqnsvxPoqdS0dSZ2UQVAcUiAIC6oS19WqhWl21bqFY1tKUvUUVAsQgAoG6gv1eH9g1rU09Fm3s3alNPRYf2DWugvzd1aUAhGAQGlti7Z7tGd23V1JlZDW3p4+CPrkYAACsM9Pdy4EcW6AICgEwRAACQKQIAQCGYUV1+jAEAaDlmVHcGWgB1fFsBWoMZ1Z2DFoD4tgK00uKM6jm9OKlucUY1Z1eVS/YtAL6tdB9ac2kxo7pzZBUAqx0YWP+luxyZOKXR24/pHZ96VKO3H9PnJ06lLik7zKjuHNl0AZ2vm4dvK91jaWtusfvhwNikRndt5eDTZsyo7gxZtADW6ubh20r3oDVXLgP9vXrNFZfyt1RiWbQAGg1K8W2lO9CaAy5M0haA7U/bfsb240XuZz0HBr6tdD5ac8CFSd0C+IykOyT9S5E7WTwwHFgxBsCBofvQmgPWL2kARMRXbO9sx744MOSD1TyB9UndAmjI9n5J+yVpx44dTT0XBwYAeFHpzwKKiMMRMRIRI4ODg6nLAdqOiW0oSulbAEDOWKYERSp9CwDIFcuUoGipTwP9nKSvS3qV7Snb70lZD1AmTGxD0VKfBXRzyv0DZcbENhSNLiCgpJjYVl7dMjDPIDBQYsxfKZ9uGpgnAICSY/5KeXTbirN0AQHAOnXbwDwBAADr9LJLNmj+3AvLtnXywDwBsIZuGejBi3hPcbGOTJzSW+94RJWKJUm9G9zxA/OMAZxHNw30oIb3FBdrad//orB1//t+Q7u2bU5YWXPW1QKw3bPKtq2tL6cYF/qtjxmY3Yf3FM1Yre+/d0NFP3v+hfP8i86wZgDYvtb2lKQf2X5oxdLNDxVZWKtczEXCu22gB7ynaE63Tspr1AI4JOlNEbFV0mFJR22/vn6fC62sBS72W1+3vtk54z3tTGUZs+nWSXmNxgAuiYgTkhQR/2b7pKR7bR+UFIVX16RG1wI+H64g1n14TztP2cZsunFSXqMAWLB9WUT8jyRFxAnbb5D0BUmvKLy6JjXzra8b3+zc8Z52jrJOuOq2SXmNuoA+KGnb0g0RMSXptyR9pKCaWqbZZhsXiu8+vKedgTGb9lizBRARD5/nrs2Snm99Oa3Ht758TM/M8z53CcZs2mPd8wBsD0q6SdLNki6XdF9RRbVatzXb8FJl6y9GcxizaY81A8D2Zkk3SvoDSa+UdK+kKyNiqA21AetS1v5iNIfWe/EatQCekfRNSR+S9EhEhO0bii8LWL+LPdsL5UfrvViNBoFvldQr6R8l3Wq79Gf+ID/0FwMXZ80AiIiPR8TrJb2tvunfJV1u+6DtVxZeHbAO3TpJByiaIy5sPpft3aoNBP9+ROwqpKrzGBkZifHx8XbuEh2Es4CA1dk+HhEjK7c3GgTeJWlbRHxtcVtEPG77QUn/3Poy0clSH4DpLwYuTKNB4I+rNg6w0k8l/b2k3215RehInIYJdJ5Gg8DbIuI7KzfWt+0spCJ0HJZaBjpTowC4dI37OMUCkso5bb8sq0gCZdYoAMZt/8nKjbb/WNLxZndu+3rb37X9lO0PNvt8SKNsp2FezDUggBw1CoBbJP2R7S/b/rv6z39Ieo+k9zezY9sbJH1C0pslXS3pZttXN/OcSKNMp2HSHQWsX6PF4H4s6ddtXytpd33z/RFxrAX7vkbSUxHxfUmyfbdq8w2eaMFzo83KMm2fWcEoo9RnyJ1Po9NAN0l6r6Rdkr4j6c6IONeifW+X9PSS21OSfnWVGvZL2i9JO3bsaNGuUYQynIZZtu4ooMxnyDXqAvqspBHVDv5vlvS3hVe0QkQcjoiRiBgZHBxs9+7RYcrUHQWUvUuy0TyAqyPi1ZJk+07VFoZrlVOSrlhye6i+DWhKWbqjgLJ3STa8JOTiLxFxzm7pdeC/Jekq21eqduB/u2rLTgNNK0N3FFD2LslGXUCvsf1c/eespOHF320/18yO62MJ75P0RUknJd2zeAF6AOgGZe+SbHQW0IYidx4RD0h6oMh9AEBKZe6SXPclIQEAF6esXZKNuoAAAF2KAACATBEAAJApAgAAMkUAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMgUAQAAmSIAACBTBABWNT0zr8eeflbTM/OpS0HG+BwWiyuC4SWOTJzSwbFJ9VQqWqhWdWjfsPbu2Z66LGSGz2HxaAFgmemZeR0cm9TcQlVn589pbqGqA2OTfANDW/E5bA8CAMtMnZlVT2X5x6KnUtHUmdlEFSFHfA7bgwDAMkNb+rRQrS7btlCtamhLX6KKOgt91q3B57A9CAAsM9Dfq0P7hrWpp6LNvRu1qaeiQ/uGNdDfm7q00jsycUqjtx/TOz71qEZvP6bPT5xKXVLH4nPYHo6I9u/UvknShyX9iqRrImJ8Pf9uZGQkxsfX9VA0aXpmXlNnZjW0pY8/unWYnpnX6O3HNLfw4rfWTT0Vfe3gdfz/NYHPYWvYPh4RIyu3pzoL6HFJN0r6p0T7RwMD/b0d8QdXlgPEYp/1nF4MgMU+6074fyyrTvkcdqokARARJyXJdordo0uU6TRB+qzRiRgDQEdq1WmCrRq0pc8anaiwFoDthyVdtspdt0XEkQt4nv2S9kvSjh07WlQdOl0rulxa3YLYu2e7RndtLUWXFLAehQVARLyxRc9zWNJhqTYI3IrnROdrtstlaQtiMUQOjE1qdNfWpg7c9Fmjk9AFhI7UbJcLE42ARIPAtm+Q9A+SBiXdb3siIt6UohZ0rma6XBi0BRK1ACLivogYiojeiNjGwR8Xa6C/V6+54tIL7nZh0BZgNVBkjEFb5I4AQNYYtEXOGAQGgEwRAACQKQIAADJFAABApggAACi5oi40xFlAAFBiRa56SwsAAEqqVaveng8BAAAlVfSaVQQAAJRU0WtWEQAAUFJFr1nFIDAAlFiRa1YRAABQckWtWUUXELJU1HnVQCehBYDsFHledSebnplnaezMEADISlHXAu50hGKe6AJCVrgW8EsVPdkI5UUAICtcC/ilCMV8EQDICtcCfilCMV+MASA7XAt4ucVQPLBiDCD3/5ccEADIEtcCXo5QzBMBAEASoZgjxgAAIFNJAsD2R20/aXvS9n22L01RBwDkLFUL4Kik3RExLOl7km5NVAcAZCtJAETEQxFxrn7zG5KGUtQBADkrwxjAuyU9eL47be+3PW57/PTp020sqxxYtAxAUQo7C8j2w5IuW+Wu2yLiSP0xt0k6J+mu8z1PRByWdFiSRkZGooBSS4v1WQAUqbAAiIg3rnW/7XdJequkN0REVgf29WDRMgBFS3UW0PWSDkjaGxE/T1FD2bE+C4CipRoDuEPSZklHbU/Y/mSiOkqL9VkAFC3JTOCI2JViv52E9VkAFI2lIEqM9VkAFIkAKDnWZwFQlDLMAwAAJEAAAECmCAAAyBQBAACZIgAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMgUAQAAmSIAACBTBACyMD0zr8eeflbTM/OpSwFKg+Wg0fWOTJzSwRUX1tm7Z3vqsoDkaAGgq03PzOvg2KTmFqo6O39OcwtVHRibpCUAiABAl5s6M6ueyvKPeU+loqkzs4kqAsqDAEBXG9rSp4Vqddm2hWpVQ1v6ElUElAcBgK420N+rQ/uGtamnos29G7Wpp6JD+4a5zCYgBoGRgb17tmt011ZNnZnV0JY+Dv5AHQGALAz093LgB1agCwgAMpUkAGz/te1J2xO2H7J9eYo6ACBnqVoAH42I4YjYI+kLkv4iUR0AkK0kARARzy25+TJJkaIOAMhZskFg238j6Q8l/VTStWs8br+k/ZK0Y8eO9hQHABlwRDFfvm0/LOmyVe66LSKOLHncrZI2RcRfruM5T0v6YeuqLNRWST9JXUQb8Xq7W06vtxtf68sjYnDlxsICYL1s75D0QETsTlpIi9kej4iR1HW0C6+3u+X0enN6ranOArpqyc23SXoyRR0AkLNUYwAfsf0qSVXVunTem6gOAMhWkgCIiH0p9ttmh1MX0Ga83u6W0+vN5rUmHwMAAKTBUhAAkCkCAAAyRQAUyPZHbT9ZX/foPtuXpq6pSLZvsn3CdtV2V55GZ/t629+1/ZTtD6aup2i2P237GduPp66laLavsP0l20/UP8fvT11T0QiAYh2VtDsihiV9T9Ktiesp2uOSbpT0ldSFFMH2BkmfkPRmSVdLutn21WmrKtxnJF2fuog2OSfpAxFxtaTXS/rTbn9/CYACRcRDEXGufvMbkoZS1lO0iDgZEd9NXUeBrpH0VER8PyKel3S3avNYulZEfEXS/6auox0i4kcR8e3672clnZS0PW1VxSIA2ufdkh5MXQSasl3S00tuT6nLDxC5sr1T0mslPZq2kmJxRbAmrWfNI9u3qda8vKudtRVhvWs8AZ3Kdr+kMUm3rFi5uOsQAE2KiDeudb/td0l6q6Q3RBdMumj0ervcKUlXLLk9VN+GLmG7R7WD/10RcW/qeopGF1CBbF8v6YCkvRHx89T1oGnfknSV7SttXyLp7ZI+n7gmtIhtS7pT0smI+FjqetqBACjWHZI2Szpav/zlJ1MXVCTbN9iekvRrku63/cXUNbVSfUD/fZK+qNoA4T0RcSJtVcWy/TlJX5f0KttTtt+TuqYCjUp6p6Tr6n+vE7bfkrqoIrEUBABkihYAAGSKAACATBEAAJApAgAAMkUAAECmCABgFbZfqJ8G+Ljtf7X9C/Xtl9m+2/Z/2j5u+wHbr1zy726xPWf7l5ZsG6ivMjlj+44UrwdYDQEArG42IvZExG5Jz0t6b32i0H2SvhwRr4iI16m2wuu2Jf/uZtUmjN24ZNucpD+X9GftKR1YHwIAaOyrknZJulbSQkT8/4S+iHgsIr4qSbZfIalf0odUC4LFx/wsIh5RLQiA0iAAgDXY3qja+v/fkbRb0vE1Hv521ZaI/qpqM2e3rfFYIDkCAFhdn+0JSeOS/lu1NWIauVnS3RFRVW1BsZsKrA9oGquBAqubjYg9SzfYPiHp91Z7sO1XS7pKtXWfJOkSSf+l2npQQCnRAgDW75ikXtv7FzfYHrb9m6p9+/9wROys/1wu6XLbL09VLNAIi8EBq7A9ExH9q2y/XNLHJb1OtUHdH0i6RbUVQt8SEU8ueezHJP04Im63/QNJv6hay+BZSb8TEU8U/TqAtRAAAJApuoAAIFMEAABkigAAgEwRAACQKQIAADJFAABApggAAMjU/wFur9UWE4oSkQAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "tags": [], - "needs_background": "light" - } - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "cT7Rv9lEDSji", - "colab_type": "text" - }, - "source": [ - "As we can see from above, the two new features (known as Principle Components) are not like any of our input features. Additionally, these two new components explains 64.6% of all the original variation. Let's see how well this performs in explaining our dataset." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "eBHVGhd-CJF8", - "colab_type": "code", - "colab": {} - }, - "source": [ - "reg = LinearRegression()\n", - "\n", - "reg.fit(X, y)\n", - "print(\"All Features:\\t\\t\", reg.score(X, y))\n", - "reg.fit(X_2d, y)\n", - "print(\"With PCA Features:\\t\", reg.score(X_2d, y), end=\"\\n\\n\")" - ], - "execution_count": 0, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bsoN_a8XI4p2", - "colab_type": "text" - }, - "source": [ - "As expected, with a higher explained variance ratio, we perform better in predicting the quality of the wine. With Dimensionality Reduction, we were able to capture most of the variation in the dataset while still being able to view it in two dimensions. In the most basic of terms, PCA creates a variable projected along the axis of maximum variable.\n", - "\n", - "![](https://raw.githubusercontent.com/bfkwong/data/master/IMG_0187%202.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "vwDiZRX3JYLN", - "colab_type": "text" - }, - "source": [ - "### Optional: Linear Algebra Behind PCA\n", - "\n", - "Principle Component Analysis chooses principle components along te **axis of greatest variance**. In Linear Algebra terms, the axis of greatest variance is the **Eigenvector with the largest Eigenvalue of the covariance matrix ($\\Sigma$)**\n", - "\n", - "The following are the steps in order to do PCA manually. \n", - "\n", - "1. Given data matrix $M$, generate the covariance matrix of $M$ denoted as $\\Sigma$\n", - "2. We then compute the Eigenvector and Eigenvalue of covariance matrix $\\Sigma$\n", - "3. Project the features to the Eigenvector with the largest Eigenvalue using the dot product (cross product for more than 1 dimensions)" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "J26dEAoAD0q7", - "colab_type": "code", - "colab": {} - }, - "source": [ - "import numpy as np\n", - "\n", - "# Step 1: Calculate covariance matrix \n", - "cov_mtrx = np.cov(X.T)\n", - "\n", - "# Step 2: Calculate Eigenvector and Eigenvalue\n", - "W,v = np.linalg.eig(cov_mtrx)\n", - "\n", - "# Step 3: Find the largest Eigenvalue and project our data onto the corresponding Eigenvector\n", - "idx_largest_eigenval = np.argmax(W)\n", - "eigenvec = v[:,idx_largest_eigenval]\n", - "\n", - "total = []\n", - "for row in X.index: \n", - " total.append(np.dot(X.loc[row], eigenvec))\n", - "\n", - "pd.DataFrame(pd.Series(total), columns=[\"PCA1\"]).head()" - ], - "execution_count": 0, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_G7NWh_1nf_k", - "colab_type": "text" - }, - "source": [ - "One interesting thing you may see here is that the eigenvalue corresponds to the variance explained by its corresponding eigenvalue. The eigenvalue of PCA1 is 2.26 and the variance of PCA1 is also 2.26" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "9QCALn43iTYd", - "colab_type": "code", - "colab": {} - }, - "source": [ - "idx_largest_eigenval = np.argmax(W)\n", - "variance = pd.Series(total).var()\n", - "\n", - "print(\"Largest Eigenvalue:\\t\", W[idx_largest_eigenval])\n", - "print(\"Variance of Eigenvector:\", variance)" - ], - "execution_count": 0, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bmLOaYzvEeG7", - "colab_type": "text" - }, - "source": [ - "### Multidimensional Scaling (MDS)\n", - "\n", - "We now pay a visit to our good friend, the Euclidean distance. One incredibly useful aspect of Euclidean distance is that it works in higher dimensions. The formula $$\\sqrt{x_1^2 + x_2^2}$$ is for two dimensional Euclidean distance, but to move it to a third dimension, it is as easy as adding a $x^3$ variable. One thing to recognize is that Euclidean distance in all dimension is still a number. \n", - "\n", - "Why am I rambling on about something you learned in middle school? Well the realization that Euclidean distance is scalar in all dimensions means that we can preserve the variance of n-th dimensional data in two dimensions as long as we try to ensure that the Euclidean distances in the n-th dimensional is proportion to the Euclidean distance in 2 dimensions. That was a lot to take in, the following image explains the concept.\n", - "\n", - "![](https://raw.githubusercontent.com/bfkwong/data/master/IMG_0188.jpg)\n", - "\n", - "Notice how when we reduced our dimensions from 2 to 1, the distances between points A, B, and C remained the same. Meaning that x, y, and z remained the same between the two dimensions. While distances may not always be preserved perfectly between dimensions, MDS attempts to preserve it as well as possible. " - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "L4AjVFrwn78I", - "colab_type": "code", - "colab": {} - }, - "source": [ - "from sklearn.manifold import MDS\n", - "# We want to be able to plot this on a 2-D scatter plot so we choose 2 Dimensions\n", - "dimension_we_want = 2\n", - "\n", - "mds = MDS(n_components=dimension_we_want)\n", - "X_2d = mds.fit_transform(X) \n", - "X_2d = pd.DataFrame(X_2d, columns=[\"Dimension 1\", \"Dimension 2\"])\n", - "\n", - "display(X_2d.head())\n", - "print(\"Percentage variance explained:\", X_2d.var().sum()/X.var().sum())\n", - "X_2d.plot.scatter(x=\"Dimension 1\", y=\"Dimension 2\")" - ], - "execution_count": 0, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mlIA-o_cEnSD", - "colab_type": "text" - }, - "source": [ - "By using MDS, we were actually able to preserve over 95% of the variance from the original datasets. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "88kI9JxzFO6n", - "colab_type": "text" - }, - "source": [ - "### Linear vs Nonlinear Dimensionality Reduction \n", - "\n", - "Thus far, we have explored PCA (a linear reduction technique) and MDS (a nonlinear reduction technique). While there are a lot of differences between the two methods. The key differences could be boiled down to just the following statement: **linear dimensionality reduction technique only stretch and shift the data while nonlinear techniques make more drastic changes to the data**.\n", - "\n", - "This sometimes leads to nonlinear techniques being better at capturing variance but losing the overall shape of the data whereas linear techniques are better at keeping the general shape of the original data but loses more variance along the way. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7uprQWlUGC37", - "colab_type": "text" - }, - "source": [ - "# Exercises" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "UnMJKgsTkDoZ", - "colab_type": "text" - }, - "source": [ - "1. Consider the Iris dataset (https://raw.githubusercontent.com/dlsun/pods/master/data/iris.csv). Drop the \"SepalWidth\" and \"PedalWidth\" columns and then apply PCA on \"SepalLength\" and \"PedalLength\" with `n_components = 2`. How many percent of the variance was PCA able to capture in this case? What happens when we use PCA to compress 2D data into 2D data?" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "HCXKtA8AlSQq", - "colab_type": "code", - "colab": {} - }, - "source": [ - "" - ], - "execution_count": 0, - "outputs": [] - } - ] -} \ No newline at end of file diff --git a/10-Textual-Data/10.5 Sentiment_Analysis.ipynb b/10-Textual-Data/10.5 Sentiment_Analysis.ipynb new file mode 100644 index 0000000..a235b4f --- /dev/null +++ b/10-Textual-Data/10.5 Sentiment_Analysis.ipynb @@ -0,0 +1,1700 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "name": "Copy of 10.5 Regular Expression and Sentiment Analysis.ipynb", + "provenance": [], + "collapsed_sections": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "3UYXoMwEpG40", + "colab_type": "text" + }, + "source": [ + "# 10.5 Regular Expression and Sentiment Analysis\n", + "\n", + "Sentiment analysis is the use of natural language processing to quantify subjective information. Our goal in this section is to use machine learning to identify whether a piece of text captures positive, negative, or neutral emotions. Sentiment analysis has become more prevalent in our world through its application in algorithmic traders, recommendation systems, and market research. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2vgbSDB4v5rF", + "colab_type": "text" + }, + "source": [ + "## Sentiment Analysis\n", + "\n", + "Consider the following sentences:\n", + "\n", + "1. \"I am so happy to be here right now!\"\n", + "2. \"I'm pretty sad about this whole thing.\"\n", + "\n", + "Most people would agree that the first sentence exhibits positive emotion and the second sentence exhibits negative emotion. We perceive it to be this way because the first sentence has the word happy and the second sentence has the word sad. With this very simple idea in mind, we can build a very naïve classifier that determines if a sentence exhibits positive, negative, or neutral emotion. \n", + "\n", + "Our output being a range between -1 and 1 with sentence towards -1 as having negative sentiment and sentices towards +1 having positive sentiment.\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "5tcQToKM0eBY", + "colab_type": "code", + "outputId": "8f297650-236b-475d-d25c-992a963b8a4b", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 68 + } + }, + "source": [ + "positive_words = set([\"happy\", \"great\", \"fanstastic\", \"love\", \"appreciate\", \"grateful\"])\n", + "negative_words = set([\"sad\", \"gross\", \"disturbing\", \"bitter\", \"sorry\", \"pathetic\"])\n", + "\n", + "def sentiment_analyzer_v2(sentence): \n", + " sentence = sentence.lower().split(\" \")\n", + "\n", + " pos_word_cnt = 0\n", + " neg_word_cnt = 0\n", + "\n", + " for word in sentence: \n", + " if word in positive_words: \n", + " pos_word_cnt += 1\n", + " elif word in negative_words: \n", + " neg_word_cnt += 1\n", + " \n", + " return (pos_word_cnt - neg_word_cnt) / (pos_word_cnt + neg_word_cnt)\n", + " \n", + "print(sentiment_analyzer_v2(\"This is making me happy !\"))\n", + "print(sentiment_analyzer_v2(\"This is making me sad !\"))\n", + "print(sentiment_analyzer_v2(\"I am neither happy nor sad .\"))" + ], + "execution_count": 1, + "outputs": [ + { + "output_type": "stream", + "text": [ + "1.0\n", + "-1.0\n", + "0.0\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gk2g-nvq1-Xf", + "colab_type": "text" + }, + "source": [ + "Given a big enough dictionary of positive and negative words, this algorithm can work pretty well. But it takes a lot of work to figure out what words are happy and then type it into a list, its just not efficient. So, being the clever Data Scientists that we are, let's create a machine learning algorithm. \n", + "\n", + "Here is the schematics: let's get a list of texts that are labelled as either positive, negative, or netural. We use a count vector to see what words occured in which text and how many times that word occured. We then use a machine learning algorithm to train on this count vector along with the sentiment label. In essense, this algorithm is saying \"if these words occured $n$ number of times in a text, then it is likely for it to be a specific sentiment\"\n", + "\n", + "Download a set of tweets from (link) and you will see that there are two features that we care about: polarity and text. Polarity tells us what the sentiment is (0 for negative, 2 for neutral, 4 for positive). To keep things consistent with our naive algorithm above, let's map the polarity such that positive is +1, neutral is 0, and negative is -1. With this encoding, the closer our prediction is to 1, the more positive the sentiment is and the closer our prediction is to -1, the more negative the sentiment is. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "d1Vy9AYt4F3-", + "colab_type": "code", + "outputId": "a6c0836d-268a-4fda-9983-5dc182601e16", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 204 + } + }, + "source": [ + "import pandas as pd\n", + "\n", + "df_tweets = pd.read_csv(\"https://raw.githubusercontent.com/bfkwong/data/master/twitter_sentiment.csv\")\n", + "df_tweets[\"polarity\"] = df_tweets[\"polarity\"].map({4:1, 2:0, 0:-1})\n", + "df_tweets.head()" + ], + "execution_count": 2, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
polaritytweet_idtweet_datequeryusertext
013Mon May 11 03:17:40 UTC 2009kindle2tpryan@stellargirl I loooooooovvvvvveee my Kindle2. ...
114Mon May 11 03:18:03 UTC 2009kindle2vcu451Reading my kindle2... Love it... Lee childs i...
215Mon May 11 03:18:54 UTC 2009kindle2chadfuOk, first assesment of the #kindle2 ...it fuck...
316Mon May 11 03:19:04 UTC 2009kindle2SIX15@kenburbary You'll love your Kindle2. I've had...
417Mon May 11 03:21:41 UTC 2009kindle2yamarama@mikefish Fair enough. But i have the Kindle2...
\n", + "
" + ], + "text/plain": [ + " polarity ... text\n", + "0 1 ... @stellargirl I loooooooovvvvvveee my Kindle2. ...\n", + "1 1 ... Reading my kindle2... Love it... Lee childs i...\n", + "2 1 ... Ok, first assesment of the #kindle2 ...it fuck...\n", + "3 1 ... @kenburbary You'll love your Kindle2. I've had...\n", + "4 1 ... @mikefish Fair enough. But i have the Kindle2...\n", + "\n", + "[5 rows x 6 columns]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 2 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zIFResqfRiOt", + "colab_type": "text" + }, + "source": [ + "### Text Normalization\n", + "\n", + "The goal of normalizing is to remove excess noise so that the algorithm only has to focus on what is important. Think of this process as the text version of `StandardScaler`.\n", + "\n", + "**Lemmatization** is one popular normalization technique. During lemmatization, the words `studies` and `studying` gets lemmatized to `study`. In essense, the process of lemmatization turns different forms of the same word (i.e. studies, studying) into the same base lemma (study). This helps reduce the noise in our dataset. \n", + "\n", + "**Stop word removal** is another way to normalize our text in order to reduce noise. Stop words such as \"there\", \"how\", \"then\", \"we\" offer no additional clues for deciding what the sentiment of a sentence is. Thus, it makes sense for us to remove these words before training out algorithm \n", + "\n", + "Lemmatization and stop word removal are both tedious tasks for us to do, which is why NLTK provides us with functions to remove them. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "tloDK32sSxNt", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 306 + }, + "outputId": "bb0d1c6d-14a7-42c9-959e-2be71e324325" + }, + "source": [ + "import nltk\n", + "\n", + "nltk.download('punkt')\n", + "nltk.download('stopwords')\n", + "nltk.download('wordnet')\n", + "\n", + "from nltk.tokenize import word_tokenize\n", + "from nltk.corpus import stopwords\n", + "from nltk.stem.wordnet import WordNetLemmatizer\n", + "\n", + "stop_words=set(stopwords.words(\"english\"))\n", + "tweets = list(df_tweets[\"text\"])\n", + "for tweet in range(len(tweets)): \n", + " tweets[tweet] = [x for x in word_tokenize(tweets[tweet]) if x not in stop_words]\n", + "\n", + "lem = WordNetLemmatizer()\n", + "for tweet in range(len(tweets)): \n", + " tweets[tweet] = [lem.lemmatize(x) for x in tweets[tweet]]\n", + "\n", + "df_tweets[\"processed_text\"] = [\" \".join(x) for x in tweets]\n", + "df_tweets.head()" + ], + "execution_count": 3, + "outputs": [ + { + "output_type": "stream", + "text": [ + "[nltk_data] Downloading package punkt to /root/nltk_data...\n", + "[nltk_data] Package punkt is already up-to-date!\n", + "[nltk_data] Downloading package stopwords to /root/nltk_data...\n", + "[nltk_data] Package stopwords is already up-to-date!\n", + "[nltk_data] Downloading package wordnet to /root/nltk_data...\n", + "[nltk_data] Package wordnet is already up-to-date!\n" + ], + "name": "stdout" + }, + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
polaritytweet_idtweet_datequeryusertextprocessed_text
013Mon May 11 03:17:40 UTC 2009kindle2tpryan@stellargirl I loooooooovvvvvveee my Kindle2. ...@ stellargirl I loooooooovvvvvveee Kindle2 . N...
114Mon May 11 03:18:03 UTC 2009kindle2vcu451Reading my kindle2... Love it... Lee childs i...Reading kindle2 ... Love ... Lee child good re...
215Mon May 11 03:18:54 UTC 2009kindle2chadfuOk, first assesment of the #kindle2 ...it fuck...Ok , first assesment # kindle2 ... fucking roc...
316Mon May 11 03:19:04 UTC 2009kindle2SIX15@kenburbary You'll love your Kindle2. I've had...@ kenburbary You 'll love Kindle2 . I 've mine...
417Mon May 11 03:21:41 UTC 2009kindle2yamarama@mikefish Fair enough. But i have the Kindle2...@ mikefish Fair enough . But Kindle2 I think '...
\n", + "
" + ], + "text/plain": [ + " polarity ... processed_text\n", + "0 1 ... @ stellargirl I loooooooovvvvvveee Kindle2 . N...\n", + "1 1 ... Reading kindle2 ... Love ... Lee child good re...\n", + "2 1 ... Ok , first assesment # kindle2 ... fucking roc...\n", + "3 1 ... @ kenburbary You 'll love Kindle2 . I 've mine...\n", + "4 1 ... @ mikefish Fair enough . But Kindle2 I think '...\n", + "\n", + "[5 rows x 7 columns]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 3 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zMUQylZN4JQC", + "colab_type": "text" + }, + "source": [ + "With this data, let's use the CountVectorizer to turn the tweets into a collection of words. To make sure we exclude any hashtags and @ symbols, we will also specify a regular expression tokenizer to only include alphabet characters with the `tokenizer` parameter.\n", + "\n", + "Additionally, we want to specify `ngram_range = (1,2)`. This is to help provide context to the words. Consider the double negative string `I do not dislike` which carries positive sentiment. If we split the words into unigrams, we get words like `not` and `dislike`, which are negative words. By using a bigram, we are able to train the algorithm to realize that `not dislike` is actually a positive term. Thus, allowing the algorithm to be able to handle difficult to decipher sentiments like double negatives and sarcasm. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "25ZsIkbos6gR", + "colab_type": "code", + "colab": {} + }, + "source": [ + "from sklearn.feature_extraction.text import CountVectorizer\n", + "from nltk.tokenize import RegexpTokenizer\n", + "\n", + "token = RegexpTokenizer(r'[a-zA-Z]+')\n", + "cv = CountVectorizer(lowercase=True,stop_words='english',ngram_range = (1,2),tokenizer = token.tokenize)\n", + "tweet_text_cv = cv.fit_transform(df_tweets['processed_text'])" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mSjQ6Pms4Ang", + "colab_type": "text" + }, + "source": [ + "Why are we doing this? Given that we have a label for whether the piece of text exhibits positive, negative, or neutral emotions, we can use the count vector to see what words tend to occur in positive sentences, and etc. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "I9h-lDkt5QAb", + "colab_type": "code", + "outputId": "15b3389d-78b5-4c80-e6bf-f3eb13fc43c2", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 270 + } + }, + "source": [ + "vocab = [[x, cv.vocabulary_[x]] for x in cv.vocabulary_]\n", + "vocab.sort(key=lambda x:x[1])\n", + "vocab = [x[0] for x in vocab]\n", + "\n", + "df_twitter_cv = pd.DataFrame(tweet_text_cv.todense(), columns=vocab)\n", + "df_twitter_cv[\"polarity\"] = df_tweets[\"polarity\"]\n", + "\n", + "df_twitter_cv.head()" + ], + "execution_count": 40, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
aaplaapl esabortionabortion zealotabsolutelyabsolutely blowabsolutely hilariousaccannisaccannis edogaccessaccess damnaccess throttleaccidentaccident guessaccident locationaccordingaccording createaccostsaccosts rogeraccountaccount requestacgacg customachingaciaacia pillsactuallyactually quiteadad adobead wadamadam lambertaddadd peopleaddictionaddiction thankaddictiveadidasadidas billups...years greatyeeeeeyeezyyeezy khakiyemayesyes gmyes lolyes myes videoyesterdayyesterday cbsykyoyo teachyorkyork timesyoutubeyoutube adobeyryr oldytzyuanyuan investedyummmmmyzealotzealot nzerozero desirezetzet oziczlffzomgzomg gzoomzoom lebronzydrunaszydrunas awesomepolarity
00000000000000000000000000000000000000000...0000000000000000000000000000000000000001
10000000000000000000000000000000000000000...0000000000000000000000000000000000000001
20000000000000000000000000000000000000000...0000000000000000000000000000000000000001
30000000000000000000000000000000000000000...0000000000000000000000000000000000000001
40000000000000000000000000000000000000000...0000000000000000000000000000000000000001
\n", + "

5 rows × 5483 columns

\n", + "
" + ], + "text/plain": [ + " aapl aapl es abortion ... zydrunas zydrunas awesome polarity\n", + "0 0 0 0 ... 0 0 1\n", + "1 0 0 0 ... 0 0 1\n", + "2 0 0 0 ... 0 0 1\n", + "3 0 0 0 ... 0 0 1\n", + "4 0 0 0 ... 0 0 1\n", + "\n", + "[5 rows x 5483 columns]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 40 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qRecjyTw7iB0", + "colab_type": "text" + }, + "source": [ + "If a certain word occurs very frequently in texts that are labeled as positive texts, then we can make the assumption that the word is positive. So if in the future we encounter a sentence with this word, we should classify the sentence as positive. \n", + "\n", + "With this idea in mind, let's train a model to predict whether a sentence exhibits positive, negative, or neutral emotions. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "5MQxI6NTtkQ6", + "colab_type": "code", + "outputId": "d5cd83c2-9d91-44ac-c4ca-a2806be56931", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + } + }, + "source": [ + "from sklearn.model_selection import train_test_split\n", + "\n", + "from sklearn.linear_model import LinearRegression\n", + "from sklearn import metrics\n", + "\n", + "X_train, X_test, y_train, y_test = train_test_split(tweet_text_cv, \n", + " df_tweets['polarity'], \n", + " test_size=0.3, \n", + " random_state=1)\n", + "\n", + "sentiment_analyzer = LinearRegression().fit(X_train, y_train)\n", + "predicted = sentiment_analyzer.predict(X_test)\n", + "print(\"LinearRegression R^2:\\t\", sentiment_analyzer.score(X_test, y_test))\n", + "print(\"LinearRegression MSE:\\t\",metrics.mean_squared_error(y_test, predicted))" + ], + "execution_count": 41, + "outputs": [ + { + "output_type": "stream", + "text": [ + "LinearRegression R^2:\t 0.3904061882859056\n", + "LinearRegression MSE:\t 0.43156532563528044\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "336lT6cO9C5A", + "colab_type": "text" + }, + "source": [ + "Let's see this bad boy in action. Consider the following sentences. The model was able to correctly classify the sentence has having positive leaning polarity" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "truyzLGe8iB6", + "colab_type": "code", + "outputId": "58d6be48-f315-400b-cad3-15b1de6e870f", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + } + }, + "source": [ + "# Clearly positive sentence\n", + "test = cv.transform([\"I love being here!\"]).todense()\n", + "sentiment_analyzer.predict(test)" + ], + "execution_count": 47, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "array([0.23681886])" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 47 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "QeZ66XQUUj5A", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "759a34ab-8f7d-4606-e8d3-911d219fe465" + }, + "source": [ + "# Clearly negative sentence\n", + "test = cv.transform([\"I hate this.\"]).todense()\n", + "sentiment_analyzer.predict(test)" + ], + "execution_count": 48, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "array([-0.41152944])" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 48 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "QnY1HqkAVH09", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "49e33736-cd9a-4090-af71-fee58fb0fa5c" + }, + "source": [ + "# Ambiguous positive sentence with double negative\n", + "test = cv.transform([\"I do not dislike school.\"]).todense()\n", + "sentiment_analyzer.predict(test)" + ], + "execution_count": 9, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "array([0.05733004])" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 9 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZYD55k4NXLSp", + "colab_type": "text" + }, + "source": [ + "Since we used a linear model in this, we can analyze the coefficients of each variable to see what contributes most to positive and negative sentiment. Recall that each variable represents either an ngram, the ngram that is the biggest is the `most positive` and the ngram that is the smallest is the `most negative`. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "zXUbPfHOYb9o", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 450 + }, + "outputId": "859752b3-6459-416e-f604-499074f36d5a" + }, + "source": [ + "ngrams = [x for x in df_twitter_cv.columns if x != \"polarity\"]\n", + "\n", + "df_word_coef = pd.DataFrame([ngrams,sentiment_analyzer.coef_], index=[\"word\", \"coef\"]).T.set_index(\"word\")\n", + "df_word_coef.sort_values(\"coef\", ascending=False)" + ], + "execution_count": 10, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
coef
word
loves twitter0.441781
loves0.441781
g0.404112
cool0.387307
loved0.38585
......
gm-0.389355
comcast-0.390205
fighting-0.406231
fighting latex-0.406231
hate-0.421888
\n", + "

5482 rows × 1 columns

\n", + "
" + ], + "text/plain": [ + " coef\n", + "word \n", + "loves twitter 0.441781\n", + "loves 0.441781\n", + "g 0.404112\n", + "cool 0.387307\n", + "loved 0.38585\n", + "... ...\n", + "gm -0.389355\n", + "comcast -0.390205\n", + "fighting -0.406231\n", + "fighting latex -0.406231\n", + "hate -0.421888\n", + "\n", + "[5482 rows x 1 columns]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 10 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-ivL_T4qa16K", + "colab_type": "text" + }, + "source": [ + "As expected, words like `hate` and `fight` has very negative connotations to it while words like `loves` and `cool` has very positive connotation to it. Another thing we can look at is the intercept, which tells us the overall sentiment of the entire training corpus. As you can see below, the overall sentinment of the training corpus is rather neutral." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "rocJ7syaYd82", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "49a88a1a-c8b6-4e89-c529-049eecd3a8a6" + }, + "source": [ + "sentiment_analyzer.intercept_" + ], + "execution_count": 11, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0.01035837109566326" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 11 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4iu58XS1bjc0", + "colab_type": "text" + }, + "source": [ + "## NLTK Implementation \n", + "\n", + "This is a lot of tedious work, and whenever there is a lot of tedious work, you can bet that there's a library for that. The following is the NLTK sentiment analysis algorithm using a very similar technique as we implemented above. \n", + "\n", + "NLTK is a different library than SciKit-Learn so it will require us to do our preprocessing a bit differently. The remainder of the section will walk you through the differences." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yzphuVSGrUIr", + "colab_type": "text" + }, + "source": [ + "### Text Preprocessing\n", + "\n", + "NLTK requires that your training examples are in the form of a list of tuples with 2 elements where the first element are word tokens and the second element is the class. An example would be: \n", + "\n", + "```\n", + "[([\"I\", \"love\", \"Pepsi\"], 1), ([\"I\", \"am\", \"not\", \"a\", \"fan\", \"of\", \"Coke\"], -1), ...]\n", + "```\n", + "\n", + "The following code creates this encoding as well as creating a train test split:" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "KSz0heZnmATQ", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 425 + }, + "outputId": "1d597d9d-7b74-4b8b-a0ca-a8a924ec7c42" + }, + "source": [ + "from nltk.tokenize import TweetTokenizer\n", + "from nltk.classify import NaiveBayesClassifier\n", + "from nltk.sentiment import SentimentAnalyzer\n", + "from nltk.sentiment.util import *\n", + "import random\n", + "\n", + "tweet_tknize = TweetTokenizer()\n", + "df_tweetsnltk = df_tweets.copy()[[\"polarity\", \"processed_text\"]]\n", + "\n", + "polarity_score = df_tweetsnltk.polarity\n", + "tokenized_tweets = list(df_tweetsnltk.processed_text.apply(tweet_tknize.tokenize))\n", + "\n", + "tweets_formatted = []\n", + "for x in range(len(polarity_score)):\n", + " tweets_formatted.append((tokenized_tweets[x], polarity_score[x]))\n", + "\n", + "training_tweets = tweets_formatted[:int(0.70 * len(tweets_formatted))]\n", + "testing_tweets = tweets_formatted[int(0.70 * len(tweets_formatted)):]\n", + "\n", + "tweets[0:2]" + ], + "execution_count": 28, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[['@',\n", + " 'stellargirl',\n", + " 'I',\n", + " 'loooooooovvvvvveee',\n", + " 'Kindle2',\n", + " '.',\n", + " 'Not',\n", + " 'DX',\n", + " 'cool',\n", + " ',',\n", + " '2',\n", + " 'fantastic',\n", + " 'right',\n", + " '.'],\n", + " ['Reading',\n", + " 'kindle2',\n", + " '...',\n", + " 'Love',\n", + " '...',\n", + " 'Lee',\n", + " 'child',\n", + " 'good',\n", + " 'read',\n", + " '.']]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 28 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "znirIYg1srFx", + "colab_type": "text" + }, + "source": [ + "### Feature Extraction\n", + "\n", + "Now, that we have the tokenize string and its labels. Let's create our `SentimentAnalyzer` object and extract unigrams to prepare the tweets for training: " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "hIByTveaenDj", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Create our SentimentAnalyzer\n", + "sentim_analyzer = SentimentAnalyzer()\n", + "\n", + "# Get all words/tokens that is in our trianing set\n", + "# This formats our data in the way that the next function requires it\n", + "all_words = sentim_analyzer.all_words(training_tweets)\n", + "\n", + "# Get the formatted all_words list and create unigrams out of it\n", + "unigram_feats = sentim_analyzer.unigram_word_feats(all_words, min_freq=4)\n", + "\n", + "# We add this feature to our SentimentAnalyzer object\n", + "sentim_analyzer.add_feat_extractor(extract_unigram_feats, unigrams=unigram_feats)" + ], + "execution_count": 0, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ISXRyMlfvMji", + "colab_type": "text" + }, + "source": [ + "### Training our Sentiment Analyzer\n", + "\n", + "We will be using a NaiveBayesClassifer for this instance. NLTK only supports classifiers for sentiment analysis" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ICZ4jrtcsUZd", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "ba1e989e-2e4a-4560-8ebd-486d181619ee" + }, + "source": [ + "training_set = sentim_analyzer.apply_features(training_tweets)\n", + "trainer = NaiveBayesClassifier.train\n", + "classifier = sentim_analyzer.train(trainer, training_set)" + ], + "execution_count": 57, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Training classifier\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TZLNMDM8vaun", + "colab_type": "text" + }, + "source": [ + "### Testing our Sentiment Analyzer" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "wigp271ivLsI", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 204 + }, + "outputId": "e39420fe-1650-4f94-f9f0-0f7a858f4f9f" + }, + "source": [ + "testing_set = sentim_analyzer.apply_features(testing_tweets)\n", + "sorted(sentim_analyzer.evaluate(testing_set).items())" + ], + "execution_count": 58, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Evaluating NaiveBayesClassifier results...\n" + ], + "name": "stdout" + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[('Accuracy', 0.6333333333333333),\n", + " ('F-measure [-1]', 0.6837606837606838),\n", + " ('F-measure [0]', 0.6304347826086957),\n", + " ('F-measure [1]', 0.5714285714285714),\n", + " ('Precision [-1]', 0.625),\n", + " ('Precision [0]', 0.8055555555555556),\n", + " ('Precision [1]', 0.52),\n", + " ('Recall [-1]', 0.7547169811320755),\n", + " ('Recall [0]', 0.5178571428571429),\n", + " ('Recall [1]', 0.6341463414634146)]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 58 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rUFp7fSwvvD7", + "colab_type": "text" + }, + "source": [ + "Now that we know it works, let's test it with some random tweets we pulled from Twitter." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "AptkeGbNvcbC", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "b709bf50-b3f1-4e87-b2ea-0dd815693f98" + }, + "source": [ + "# Clearly positive sentence\n", + "sentim_analyzer.classify(\"I love being here!\".split(\" \"))" + ], + "execution_count": 63, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "1" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 63 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "j8jolufOwzVy", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "f430a625-7430-4598-8448-acc1c286a163" + }, + "source": [ + "# Clearly negative sentence\n", + "sentim_analyzer.classify(\"I hate this.\".split(\" \"))" + ], + "execution_count": 64, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "-1" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 64 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "2j7DvCPIyG3s", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "095aff26-2077-49e4-e14c-bdfb1622965d" + }, + "source": [ + "# Ambiguous positive sentence with double negative\n", + "sentim_analyzer.classify(\"I do not dislike school.\".split(\" \"))" + ], + "execution_count": 65, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "1" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 65 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZXaLsyo7y4Z3", + "colab_type": "text" + }, + "source": [ + "# Exercises" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "x5ANpEW3y5Mx", + "colab_type": "code", + "colab": {} + }, + "source": [ + "" + ], + "execution_count": 0, + "outputs": [] + } + ] +} \ No newline at end of file