• In this topic, we will see how to manage a Deep Learning project. We will go from the early project idea to the maintenance of the associated Deep Learning product.
  • Most of this knowledge can be acquired through years of experience working and leading Deep Learning projects in research or in the industry. You can consider this topic as a crash course based on Professor Ng’s experience.
  • During this topic, we will use as an illustration the example of: developing a door lock that unlocks itself when it identifies your face through a camera. To achieve this task, you need to store a reference picture of yourself. You will have to train a Neural Network to tell whether two input images are from the same person, ie you. This is a binary classification task. If the images are of the same person, the system should open the door.

Project Timeline

  • Projects can be carried for very different amounts of time based on the complexity and the quantity of research involved in the project. However, most of them can be divided into the following steps:

    1. Project selection

    2. Data acquisition

    3. Deep Learning model design

    4. Model training

    5. Model testing

    6. Deployment

    7. Maintenance

  • We will describe in more details each step in the following topics.

Project selection

  • Before selecting a project and investing time and money in it, there are some criteria that you should consider:

    • Interest: of course, you would like to pick a project that interests you!

    • Impact: how will your project impact people’s lives? Will your project help people?

    • Data: are there already available datasets? If not, how can you acquire data? At this stage, you need to evaluate how much data you need and how much time (and money) it would take to acquire it. This step also involves thinking about the kind of preprocessing you would need to perform on your data.

    • Domain knowledge: if your project belongs to a domain you are familiar with, you can provide a unique insight on the problem that other Deep Learning engineers do not necessarily have.

  • Before committing to a project, it is important to evaluate its feasibility. You do not want to spend months acquiring data if you realize afterwards that your project had zero chance of success.
  • The issue is that many companies’ CEOs are not aware of what Deep Learning can or cannot do. Good insights regarding the limits of Deep Learning can be obtained through years of experience. But you can get a good sense of it by staying updated about the state-of-the-art: regularly reading the most cited recent papers is a good idea.
  • Moreover, a quick way to assess the feasibility of a project is to evaluate human performances on this task. If humans can do this task well given the same inputs, then there is a good chance that you could train a Deep Learning model to accomplish this task.
  • In our example, humans are very good at face recognition. We should be able to train a Neural Network to do the same.

Acquiring Data

  • Even if having more data never hurts, big data is not compulsory for Deep Learning projects. You may have some prior knowledge having worked on this (or a similar) task that tells you whether or not having a lot of a data is a prerequisite.
  • But if you do not have prior knowledge, it is better to work at first on a smaller dataset collected in a few days. This will allow you to iterate more quickly: get a model running and identify possible problems.
  • You may become aware of inherent issues with your dataset or the acquisition process. It is preferable to know these things after a few days rather than months of useless data acquisition.
  • In our example about face recognition you need to collect a dataset of face pictures. In the end, you will probably need a dataset in the millions of samples. But if you start by spending two days scrapping internet for pictures of faces this could already help you identify early issues. For example, you may realize that because your dataset is only composed of daylight pictures, your algorithm fails at recognizing matching pairs when one picture has been taken at sunset. Then you will know that you probably need to include more pictures taken with low light.

Deep Learning model design

  • What Neural Networks architecture is the most suited for your task?
  • This step involves doing literature review. There will be a specific class dedicated to running an efficient literature review.
  • In our example, the inputs are images: a Convolutional Neural Network will be the most suited architecture. We will see in more details in class what architectures to choose for the purpose of face recognition; and how to train the model.

Train model

  • This step consists of several steps, namely, we first need to shuffle and split our dataset to train/validation/test datasets. The ratio of this split depends on the size of the dataset at hand. In traditional machine learning, a 60/20/20 split is a reasonable split. However, nowadays datasets are much larger than before. Because of this growth in the size of datasets it makes sense to assign lower portions of the dataset to the validation and test tasks, and reserve most of the dataset for training. For example if we have a dataset of size 1M then the split 90/5/5 will be a good choice.
  • The next step is initializing the weights in our model. As we will have a session dedicated to this topic we will skip this step for now.
  • Before diving into the training process, it is very important to make sure that our model makes sense and our code is bug free. One way for testing this is to first try to overfit and get perfect results on a handful number of training examples. If we fail to overfit to this small set of training examples then, we will realize that there is something wrong with our code. This method will save our time and effort.
  • Another step in training is determining the best hyperparameters for our model such as learning rate, number of hidden layers, etc. We will elaborate on this in one of the Coursera videos.

Test model

  • To be able to quantify your algorithm’s performance, you need to choose an evaluation metric to rank your models. You can additionally choose some satisfying metrics: these metrics tell you whether or not you should reject a given model.
  • For our example of face detection, you may want to rank your models based on accuracy: the percentage of times the model makes a good prediction. However, you can not allow your model to make false positive predictions: this would allow a stranger to enter your home! So you probably want the false positive rate to be less than 0.01%
  • Most of the time, a Deep Learning model will not work the 1st time. If your model does not satisfy your requirements, you need to run some error analysis to evaluate why your model is doing badly. This may involve iterating a couple of times over steps 2 and 5.
  • In our example, you may realize that in real life, your face detection algorithm is performing poorly on night pictures. You may want to go back to data acquisition and obtain more pictures taken at darker hours of the day or include a pre-processing of night pictures.

Deployment

  • Once you have a working Deep Learning model, you will want to think about how to deploy it in the real world. At this step, you will need to work with the whole system to make it fully autonomous and robust.
  • In our example, most of the time nothing is happening at the front door. It would be too power-expensive to run a face recognition Neural Network non-stop 24/7. Instead you can think about adding an activity detection subsystem. In the best case, the subsystem will only trigger your face recognition network if there is a human standing at the door. This will drastically decrease the power consumption of the whole system.
  • So the question arises: will you build a Machine Learning or non-Machine Learning activity detection system?
  • You could train a small neural network to recognize when a human is in front of the door. This neural network would be a lot smaller and consume much less energy than the face recognition network. Or you could use a non-Machine Learning approach. The activity detection can be an image processing algorithm checking changes in the frame pixels. If the frame pixels’ change is greater than a given threshold then the face recognition network is triggered.
Type non-ML ML
Pros Easy to implement More accurate
  More robust Less false positives
Cons Less accurate Longer to train and need data
  More false positives Sensible to data mismatch
  • In this case, it would probably be better to implement the “quick and dirty” non-Machine Learning approach first. it is easy to conceive, and it allows to try the whole pipeline quickly. Moreover, it can be more robust: this algorithm is independent of the training data. If your system ends up being deployed in a different country where the light angles are different, the Deep Learning approach may fail as your model has never seen pictures lit this way. We call this a data mismatch problem.
  • You can always in a second time evaluate the performances of the non-Machine Learning approach. If the number of false positives is too high, the subsystem will trigger the face recognition network too often. This could lead to a huge waste of energy, and in that case, you may want to switch to the machine learning approach instead.
  • Finally: where to run your face recognition model? Do you want to run it on a remote server through wifi: cloud detector? Or do you want to run it on an onboard chip directly on your door: edge detector?
Type Cloud detector Edge detector
Pros Larger model Faster response
  User feedback Privacy
  Easy Update No Wifi needed
Cons Sensible to networks problem Hard to update

Maintenance

  • Maintenance of the model is especially important in industry applications.
  • Imagine our model takes as input some sensor readings. Sensors might lose their accuracy over time, a phenomenon known as encrustment of sensor. As a result, a measured 100 degrees might not be 100 degrees anymore. This implies that the inputs to our model might change over time.
  • Consequently, it is important keep the model updated and try to adapt to the change of environment, which we refer to as model maintenance.