Deep Learning: Shallow and Deep Nets

Deep learning is a field that uses artificial neural networks very frequently. One common application is convolutional neural networks, which are used to classify images, video, text, or sound.

Neural networks that operate on two or three layers of connected neuron layers are known as shallow neural networks. Deep learning networks can have many layers, even hundreds. Both are machine learning techniques that learn directly from input data.

Deep learning is especially well suited to complex identification applications such as face recognition, text translation, and voice recognition. It’s also a key technology used in advanced driver assistance systems and tasks including lane classification and traffic sign recognition.

Techniques Used with Neural Networks

Common machine learning techniques for designing artificial neural network applications include supervised and unsupervised learning, classification, regression, pattern recognition, and clustering.

Supervised Learning

Supervised neural networks are trained to produce desired outputs in response to sample inputs, making them particularly well suited for modeling and controlling dynamic systems, classifying noisy data, and predicting future events. Deep Learning Toolbox™ includes four types of supervised networks: feedforward, radial basis, dynamic, and learning vector quantization.

Classification

Classification is a type of supervised machine learning in which an algorithm “learns” to classify new observations from examples of labeled data.

Regression

Regression models describe the relationship between a response (output) variable and one or more predictor (input) variables.

Pattern Recognition

Pattern recognition is an important component of artificial neural network applications in computer vision, radar processing, speech recognition, and text classification. It works by classifying input data into objects or classes based on key features, using either supervised or unsupervised classification.

For example, in computer vision, supervised pattern recognition techniques are used for optical character recognition (OCR), face detection, face recognition, object detection, and object classification. In image processing and computer vision, unsupervised pattern recognition techniques are used for object detection and image segmentation.

Unsupervised Learning

Unsupervised neural networks are trained by letting the neural network continually adjust itself to new inputs. They are used to draw inferences from data sets consisting of input data without labeled responses. You can use them to discover natural distributions, categories, and category relationships within data.

Clustering

Clustering is an unsupervised learning approach in which artificial neural networks can be used for exploratory data analysis to find hidden patterns or groupings in data. This process involves grouping data by similarity. Applications for cluster analysis include gene sequence analysis, market research, and object recognition.

With tools and functions for managing large data sets, MATLAB^® offers specialized toolboxes for working with machine learning, artificial neural networks, deep learning, computer vision, and automated driving.

With just a few lines of code, MATLAB lets you develop neural networks without being an expert. Get started quickly, create and visualize models, and deploy models to servers and embedded devices.

With MATLAB, you can integrate results into your existing applications. MATLAB automates deploying your artificial neural network models on enterprise systems, clusters, clouds, and embedded devices.

Typical Workflow for Designing Neural Networks

Each neural network application is unique, but developing the network typically follows these steps:

1. Access and prepare your data
2. Create the artificial neural network
3. Configure the network’s inputs and outputs
4. Tune the network parameters (the weights and biases) to optimize performance
5. Train the network
6. Validate the network’s results
7. Integrate the network into a production system

Classification and Clustering of Shallow Networks

MATLAB and Deep Learning Toolbox provide command-line functions and apps for creating, training, and simulating shallow neural networks. The apps make it easy to develop neural networks for tasks such as classification, regression (including time-series regression), and clustering. After creating your networks in these tools, you can automatically generate MATLAB code to capture your work and automate tasks.

Preprocessing, Postprocessing, and Improving Your Network

Preprocessing the network inputs and targets improves the efficiency of shallow neural network training. Postprocessing enables detailed analysis of network performance. MATLAB and Simulink^® provide tools to help you:

• Reduce the dimensions of input vectors using principal component analysis
• Perform regression analysis between the network response and the corresponding targets
• Scale inputs and targets so they fall in the range [-1,1]
• Normalize the mean and standard deviation of the training data set
• Use automated data preprocessing and data division when creating your networks

Improving the network’s ability to generalize helps prevent overfitting, a common problem in artificial neural network design. Overfitting occurs when a network has memorized the training set but has not learned to generalize to new inputs. Overfitting produces a relatively small error on the training set but a much larger error when new data is presented to the network. Learn more about how you can use cross-validation to avoid overfitting.

Two solutions to improve generalization include:

• Regularization modifies the network’s performance function (the measure of error that the training process minimizes). By including the sizes of the weights and biases, regularization produces a network that performs well with the training data and exhibits smoother behavior when presented with new data.
• Early stopping uses two different data sets: the training set, to update the weights and biases, and the validation set, to stop training when the network begins to overfit the data