diff --git a/3-NeuralNetworks/03-Perceptron/lab/README.md b/3-NeuralNetworks/03-Perceptron/lab/README.md index 4c5f8835..f7248b20 100644 --- a/3-NeuralNetworks/03-Perceptron/lab/README.md +++ b/3-NeuralNetworks/03-Perceptron/lab/README.md @@ -10,7 +10,7 @@ Using the code we have developed in this lesson for binary classification of MNI 1. For each digit, create a dataset for binary classifier of "this digit vs. all other digits" 1. Train 10 different perceptrons for binary classification (one for each digit) -1. Define a function that will classify an input digit +1. Define a function that will classify an input digit > **Hint**: If we combine weights of all 10 perceptrons into one matrix, we should be able to apply all 10 perceptrons to the input digits by one matrix multiplication. Most probable digit can then be found just by applying `argmax` operation on the output. diff --git a/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb b/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb index 6bdc2341..a0ddc2f1 100644 --- a/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb +++ b/3-NeuralNetworks/04-OwnFramework/OwnFramework.ipynb @@ -708,7 +708,7 @@ "source": [ "To compute $\\partial\\mathcal{L}/\\partial W$ we can use the **chaining rule** for computing derivatives of a composite function, as you can see in the formulae above. It corresponds to the following idea:\n", "\n", - "* Suppose under given input we have obtanes loss $\\Delta\\mathcal{L}$\n", + "* Suppose under given input we have obtained loss $\\Delta\\mathcal{L}$\n", "* To minimize it, we would have to adjust softmax output $p$ by value $\\Delta p = (\\partial\\mathcal{L}/\\partial p)\\Delta\\mathcal{L}$ \n", "* This corresponds to the changes to node $z$ by $\\Delta z = (\\partial\\mathcal{p}/\\partial z)\\Delta p$\n", "* To minimize this error, we need to adjust parameters accordingly: $\\Delta W = (\\partial\\mathcal{z}/\\partial W)\\Delta z$ (and the same for $b$)\n", @@ -821,7 +821,7 @@ "source": [ "## Training the Model\n", "\n", - "Now we are ready to write the **training loop**, which will go through our dataset, and perform the optimization minibatch by minibatch.One complete pass through the dataset is often called **an epoch**:" + "Now we are ready to write the **training loop**, which will go through our dataset, and perform the optimization minibatch by minibatch. One complete pass through the dataset is often called **an epoch**:" ] }, { @@ -1169,7 +1169,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - " Adding several layers make sense, because unlike one-layer network, multi-layered model will be able to accuratley classify sets that are not linearly separable. I.e., a model with several layers will be **reacher**.\n", + " Adding several layers make sense, because unlike one-layer network, multi-layered model will be able to accurately classify sets that are not linearly separable. I.e., a model with several layers will be **reacher**.\n", "\n", "> It can be demonstrated that with sufficient number of neurons a two-layered model is capable to classifying any convex set of data points, and three-layered network can classify virtually any set.\n", "\n", diff --git a/3-NeuralNetworks/04-OwnFramework/README.md b/3-NeuralNetworks/04-OwnFramework/README.md index d80634b5..d1adbc42 100644 --- a/3-NeuralNetworks/04-OwnFramework/README.md +++ b/3-NeuralNetworks/04-OwnFramework/README.md @@ -19,7 +19,7 @@ Let's start with formalizing the Machine Learning problem. Suppose we have a tra * For regression problem, when we need to predict a number, we can use **absolute error** ∑i|f(x(i))-y(i)|, or **squared error** ∑i(f(x(i))-y(i))2 * For classification, we use **0-1 loss** (which is essentially the same as **accuracy** of the model), or **logistic loss**. -For one-level perceptron, function *f* was defined as a linear function *f(x)=wx+b* (here *w* is the weight matrix, *x* is the vector if input features, and *b* is bias vector). For different neural network architectures, this function can take more complex form. +For one-level perceptron, function *f* was defined as a linear function *f(x)=wx+b* (here *w* is the weight matrix, *x* is the vector of input features, and *b* is bias vector). For different neural network architectures, this function can take more complex form. > In the case of classification, it is often desirable to get probabilities of corresponding classes as network output. To convert arbitrary numbers to probabilities (eg. to normalize the output), we often use **softmax** function σ, and the function *f* becomes *f(x)=σ(wx+b)* @@ -68,7 +68,9 @@ In this lesson, we have built our own neural network library, and we have used i ## 🚀 Challenge -In the accompanying notebook, you will implement your own framework for building and training multi-layered perceptrons. You will be able to see in detail how modern neural networks operate. Proceed to the [OwnFramework](OwnFramework.ipynb) notebook and work through it +In the accompanying notebook, you will implement your own framework for building and training multi-layered perceptrons. You will be able to see in detail how modern neural networks operate. + +Proceed to the [OwnFramework](OwnFramework.ipynb) notebook and work through it. ## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/8) diff --git a/3-NeuralNetworks/04-OwnFramework/lab/README.md b/3-NeuralNetworks/04-OwnFramework/lab/README.md index ddbce6bf..ebf39c17 100644 --- a/3-NeuralNetworks/04-OwnFramework/lab/README.md +++ b/3-NeuralNetworks/04-OwnFramework/lab/README.md @@ -6,7 +6,6 @@ Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/a Solve the MNIST handwritten digit classification problem using 1-, 2- and 3-layered perceptron. Use the neural network framework we have developed in the lesson. - ## Stating Notebook Start the lab by opening [MyFW_MNIST.ipynb](MyFW_MNIST.ipynb) diff --git a/3-NeuralNetworks/05-Frameworks/Overfitting.md b/3-NeuralNetworks/05-Frameworks/Overfitting.md index 515036f9..adbf14ce 100644 --- a/3-NeuralNetworks/05-Frameworks/Overfitting.md +++ b/3-NeuralNetworks/05-Frameworks/Overfitting.md @@ -1,6 +1,6 @@ # Overfitting -Overfitting is an extremely important concept in machine learning, and it is very important to get it right! +Overfitting is an extremely important concept in machine learning, and it is very important to get it right! Consider the following simple problem of approximating 5 dots (represented by `x` on the graphs below): @@ -33,12 +33,13 @@ If you can see that overfitting occurs, you can do one of the following: * Increase the amount of training data * Decrease the complexity of the model - * Use some [regularization technique](../4-ComputerVision/08-TransferLearning/TrainingTricks.md), such as [Dropout](../4-ComputerVision/08-TransferLearning/TrainingTricks.md#Dropout), which we will consider later. + * Use some [regularization technique](../../4-ComputerVision/08-TransferLearning/TrainingTricks.md), such as [Dropout](../../4-ComputerVision/08-TransferLearning/TrainingTricks.md#Dropout), which we will consider later. ## Overfitting and Bias-Variance Tradeoff Overfitting is actually a case of more generic problem in statistics, called [Bias-Variance Tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff). If we consider possible sources of error in our model, we can see two types of errors: + * **Bias errors** are caused by our algorithm not being able to capture the relationship between training data correctly. It can result from the fact that our model is not powerful enough (**underfitting**). * **Variance errors**, which are caused by the model approximating noise in the input data instead of meaningful relationship (**overfitting**). -During training, bias error decreases (as our model learns to approximate the data), and variance error increases. It is important to stop training - either manually (when we detect overfitting) or automatically (by introducing regularization) to prevent overfitting. +During training, bias error decreases (as our model learns to approximate the data), and variance error increases. It is important to stop training - either manually (when we detect overfitting) or automatically (by introducing regularization) to prevent overfitting. diff --git a/3-NeuralNetworks/05-Frameworks/README.md b/3-NeuralNetworks/05-Frameworks/README.md index 2b721e8e..3e9f7ce2 100644 --- a/3-NeuralNetworks/05-Frameworks/README.md +++ b/3-NeuralNetworks/05-Frameworks/README.md @@ -19,7 +19,7 @@ High-level API| [Keras](https://keras.io/) | [PyTorch Lightning](https://pytorch **High-level APIs** pretty much consider neural network as a **sequence of layers**, and make constructing most of the neural networks much easier. Training the model usually requires preparing the data and then calling `fit` function to do the job. -High-level API allows you to construct typical neural networks very fast, without worrying about lots of details. At the same time, low-level API offer much more control over training process, and thus they are used a lot in research, when you are dealing with new neural network architectures. +High-level API allows you to construct typical neural networks very fast, without worrying about lots of details. At the same time, low-level API offer much more control over training process, and thus they are used a lot in research, when you are dealing with new neural network architectures. It is also important to understand that you can use both APIs together, eg. you can develop your own network layer architecture using low-level API, and then use it inside the larger network constructed and trained with high-level API. Or you can define a network using high-level API as a sequence of layers, and then use your own low-level training loop to perform optimization. Both APIs use the same basic underlying concepts, and they are designed to work well together. @@ -35,4 +35,4 @@ Low-Level API | [TensorFlow+Keras Notebook](IntroKerasTF.ipynb) | [PyTorch](Intr --------------|-------------------------------------|-------------------------------- High-level API| [Keras](IntroKeras.ipynb) | *PyTorch Lightning* -After mastering the frameworks, let's recap the notion of [overfitting](Overfiting.md). \ No newline at end of file +After mastering the frameworks, let's recap the notion of [overfitting](Overfitting.md). diff --git a/3-NeuralNetworks/images/Overtitting.png b/3-NeuralNetworks/images/Overfitting.png similarity index 100% rename from 3-NeuralNetworks/images/Overtitting.png rename to 3-NeuralNetworks/images/Overfitting.png