Artificial Neural Network

6.01.1 Introduction

Artificial intelligence algorithms have long been used for modeling decision-making systems as they provide automated knowledge extraction and high inference accuracy. Artificial neural networks (ANNs) are a class of artificial intelligence algorithms that emerged in the 1980s from developments in cognitive and computer science research. Like other artificial intelligence algorithms, ANNs were motivated to address the different aspects or elements of learning, such as how to learn, how to induce, and how to deduce. For such problems, ANNs can help draw conclusions from case observations and address the issues of prediction and interpretation.

Strictly speaking, most learning algorithms used by ANNs are rooted in classical statistical pattern analysis. Most of them are based on data distribution, unlike rough set algorithms (Komorowski, Chapter 6.02). ANNs introduce a new way to handle and analyze highly complex data. Most ANN algorithms have two common features. First, its network is composed of many artificial neurons that are mutually connected. The connections are called parameters and learned knowledge from a data set is then represented by these model parameters. This feature makes an ANN model similar to a human brain. Second, an ANN model typically does not make any prior assumptions about data distribution before learning. This greatly promotes the usability of ANNs in various applications.

The study of ANNs has undergone several important stages. In the early days, ANN studies were mainly motivated by theoretical interests, that is, investigating whether a machine can replace human for decision-making and pattern recognition. The pioneering researchers (McCulloch and Pitts, 1943) showed the possibility of constructing a net of neurons that can interact to each other. The net was based on symbolic logic relations. This earlier idea of McCulloch and Pitts was not theoretically rigorous as indicated by Fitch (1944). Later in 1949, Hebb gave more concrete and rigorous evidence of how and why the McCulloch–Pitts model works (Hebb, 1949). He showed how neural pathways are strengthened once activated. In 1954, Marvin Minsky completed his doctorial study on neural networks and his discussion on ANNs later appeared in his seminal book (Minsky, 1954). This was instrumental in bringing about a wide-scale interest in ANN research. In 1958, Frank Rosenblatt built a computer at Cornell University called the perceptron (later called single-layer perceptron (SLP)) capable of learning new skills by trial and error through mimicking the human thought process. However, Minksy (1969) demonstrated its inability to deal with complex data; this somewhat dampened ANN research activity for many subsequent years.

In the period of 1970s and 1980s, ANN research was in fact not completely ceased. For instance, the self-organizing map (SOM) (Kohonen, 2001) and the Hopfield net were widely studied (Hopfield, 1982). In 1974, Paul Werbos conducted his doctorial study at Harvard University on a training process called backpropagation of errors; this was later published in his book (Werbos, 1994). This important contribution led to the work of David Rumelhart and his colleagues in the 1980s on the backpropagation algorithm, implemented for supervised learning problems (Rumelhart and McClelland, 1987). Since then, ANNs have become very popular for both theoretical studies and practical exercises.

In this chapter, we focus on two particular ANN models – Rumelhart's multilayer perceptron (MLP) and Kohonen's SOM. The former is a standard ANN for supervised learning while the latter for unsupervised learning. Both adopt a trial-and-error learning process. MLP aims to build a function to map one type of observation to another type (e.g., from genotypes to phenotypes) and SOM explores internal structure within one data set (genotypic data only).

In contrast to Rosenblatt's SLP, Rumelhart's MLP introduces hidden neurons corresponding to hidden variables. An MLP model is in fact a hierarchical composition of several SLPs. For instance, let us consider a three-layer MLP for mapping genotypes to phenotypes. If we have two variables x₁ and x₂ describing genotypic status, we can build up two SLPs, z₁ = f(x₁,x₂) and z₂ = f(x₁,x₂), for some specified function f(∘). Based on z₁ and z₂, a higher level SLP is built, y = f(z₁,z₂), where y is called model output corresponding to collected phenotypic data denoted by t. x₁, x₂, and t are observed data (collected through an experiment) while z₁ and z₂ are unobserved – z₁ and z₂ are hidden variables. For this example, MLP models the nonlinear relationship between genotypic and phenotypic data without knowing what the true function between them is. Both SLP and MLP are supervised learning models so that during learning, observations of phenotypes act as a teacher to supervise parameter estimation.

Kohonen's net, on the other hand, is an unsupervised learning algorithm. The objective of SOM is to reveal how observations (instances or samples) are partitioned. This is similar to cluster analysis, which however does not infer how clusters correlate. SOM on the other hand can provide information on how clusters correlate. SOM is an unsupervised learning algorithm because it does not use phenotypic data for model parameter estimation.

We will discuss parameter estimation, learning rule, and learning algorithms for both MLP and SOM. The parameter optimization process is commonly based on minimizing an error function, chosen for a specific problem. We will show how the learning rules are derived for MLP and SOM based on their error functions and then discuss some biological applications of these two ANN algorithms.

5 Application of ANNs in Pharmaceutical Formulation Development

Pharmaceutical formulation development requires the optimization of formulation and process variables. These relationships are difficult to model using classical methods. One of the difficulties in the quantitative approach to formulation design is the understanding of relationships between causal factors and individual pharmaceutical responses. Furthermore, a desirable formulation for one property is not always desirable for the other characteristics. The use of ANNs seems to be most suitable for dealing with complex multivariate nonlinear relationships. ANNs can identify and learn correlative patterns between input and output data pairs. Once trained, they may be used to predict outputs from new sets of data. One of the most useful properties of ANNs is their ability to generalize. These features make them suitable for solving problems in the area of optimization of formulations in pharmaceutical product development. The usefulness of neural networks for formulation optimization has been reported by various researchers, and different strategies are being used continuously for formulation optimization. Several authors have reported on the use of formulation variables, such as the level of excipients used in a formulation as input or causal factors and the percentage of drug released at different stages of a dissolution test as response factors for ANNs. However, Takayama et al. and Ibric et al. used response factors from dissolution models to train networks and to optimize pharmaceutical formulations, and it was reported that the predicted values from an ANN model were found to be in close agreement with those of experimentally generated data. Ibric et al. used a generalized regression neural network to optimize sustained release formulation compositions for aspirin. The amount of polymer and the compression pressure were used as causal factors and the in vitro dissolution test sampling time points and Korsmeyer–Peppas model parameters were used as response factors. Formulation optimization using ANN models has also been performed using a generalized distance function with optimal drug release parameters being used as response factors. The f1 (or difference) and f2 (or similarity) fit factors are typically used for the comparison of experimentally generated and predicted in vitro dissolution profiles when conducting optimizations with ANN models.

Abstract

Artificial neural networks (ANNs) as artificial intelligence have unprecedented utility in medicine. The capacity of ANNs to analyze large amounts of data and detect patterns warrants application in analysis of medical images, classification of tumors, and prediction of survival. Some of the systems already have been in clinical use, such as C.Net and BioSleep, among others. However, evidence for ANNs use in medicine is not very solid; thus, it is imperative for clinicians to have a complete understanding of ANNs' use and limitations for their effective applications. This chapter discusses the unique aspects of ANNs applications for most common types of cancers.

4.2.5 Artificial neural networks

The ANN model used in H2O AutoML is based on a multilayer feedforward ANN trained with stochastic gradient descent (Candel, Parmar, LeDell, & Arora, 2016). ANN is the first and simplest type of ANN and consists of input, hidden, and output layers. The hidden layer includes multiple layers, where each neuron in one layer has a directed connection to the neurons of the subsequent layer. ANN is capable of learning weights that map any input to the output in the connection using gradient descent. ANN works well with tabular data as input, while not as well with sequential data (Fan, Qian, Xie, & Soong, 2014).

Abstract

Artificial neural networks (ANNs) and other machine learning algorithms are extensively used in many aspects of drug design and delivery. The concepts of target validation in drug design and the concepts of drug delivery are discussed. The role of ANNs in target validation is discussed, including their capabilities and limitations. Examples of the uses of ANNs in target discovery, target validation, and hit evaluation are given. The uses of ANNs in pharmacokinetics and pharmacodynamics are discussed and examples are given for multiple aspects of absorption, distribution, metabolism, and elimination as well as toxicity. Applications of ANNs' drug delivery are discussed, including in vivo–in vitro correlation, quantitative structure–property relationship modeling, preformulation, and formulation, with the latter two topics examined in depth. It is concluded that ANNs can be of use whenever data are incomplete and/or generalization from a set of examples is required.

Abstract

The usefulness of artificial neural networks (ANNs) in controlled drug delivery systems is expressed and explained in this chapter. The chapter includes the definition of ANN, an explanation of different models of ANN, basic backpropagation ANN model architecture, learning processes of ANN models, and prediction or optimization functions of the ANN model. Because of their capacity for making predictions as well as pattern recognition, moreover modeling, ANNs have been useful in various aspects of pharmaceutical research. In this chapter, the explanation of how to use ANN to design and develop controlled release drug delivery systems is discussed. Possible applications of ANN in the design and development of controlled release dosage forms are also summarized to make the users cognizant of using this tool to solve pharmaceutical problems.

Abstract

Artificial neural networks (ANNs) were designed to simulate the biological nervous system, where information is sent via input signals to a processor, resulting in output signals. ANNs are composed of multiple processing units that work together to learn, recognize patterns, and predict data. ANNs do not require regimented experimental design and have the ability to function even with incomplete data. They can be used in multifaceted, nonlinear systems with applications in the field of pharmacokinetic modeling. Pharmacokinetic/pharmacodynamic studies are used to predict meaningful correlations among doses administered, drug concentration levels, and pharmacological response. ANNs can be useful tools in analyzing the data involved in physiological processes, which can have vast amounts of complex variables with nonlinear relationships. ANNs are also convenient for handling data involving dosage form technology because they can simultaneously handle multiple independent and dependent variables without initial definition of causal relationships between the variables and response. The clinical applications of using ANNs in pharmacokinetic modeling are further discussed in this chapter, with specific examples from clinical studies being conducted.

3.2.1 Neural networks

ANNs are an approach to machine learning and classification, based loosely upon the structure of biological brains (Haykin, 1994). ANNs are, indeed, very powerful classifiers; it has been mathematically proven that an ANN can learn any mathematical function to arbitrary precision, given enough training time and sufficient data (Hecht-Nielsen, 1989).¹⁵

ANNs are built up of nodes, representing neurons, and connections between nodes, representing axons and dendrites carrying information. Connections in an ANN are weighted. The neurons are organised in layers, with an input layer representing one particular input data vector, an output layer providing a result of the classification and one or more hidden layers, so called because they are not connected to the outside world. The simplest and by far the most widely used form of ANN is the perceptron, a fully connected feed-forward network (Figure 1.9). Different numbers of nodes, and different patterns of connectivity can be used, but the vast majority of ANNs used for practical purposes are perceptrons, fully connected feed-forward networks (Figure 1.9; Sarle, 1994).

Figure 1.9. A simple fully connected feed-forward neural network with an input layer consisting of five nodes, one hidden layer of three nodes and an output layer of one node.

This value is then modified using a nonlinear transfer, or squashing, function to produce the node’s output value (Figure 1.10).

Figure 1.10. The sigmoid function. No matter how high or low the input value, the output is between 0.0 and 1.0.

In the case of hidden layer nodes, this output becomes an input into the next layer of nodes; in the case of output nodes, this value is the final output (Haykin, 2008). The transfer function is often a sigmoid, but any squashing-type nonlinear equation can be used. The transfer function ensures that the numbers being dealt with by the ANN remain within defined bounds, becoming neither too large nor too small to be manipulated.

ANNs must be trained using known, labelled data. At the beginning of the training process, the weights on the network are initialised to small, random values. The objective of the training process is to modify the weights in such a way that they produce a consistent output for different classes of data: for example, if the aim is to classify proteins as ‘secreted’ or ‘nonsecreted’, the ANN could read in a vector of numbers representing characteristics of a protein—GC content, hydrophobicity, etc.—and output 0.0 for a secreted protein or 1.0 for a nonsecreted protein. This is the approach taken by tools such as SignalP, which uses a combination of ANNs and hidden Markov models to predict secreted proteins (Dyrløv Bendtsen, Nielsen, von Heijne, & Brunak, 2004).

The strength of ANNs lies in learning a mapping between inputs and outputs and being able to generalise this relationship to unseen data. The potential value of this power for learning the characteristics of existing, biological modules and extending the resulting knowledge to the design of new, modular synthetic systems has long been recognised (Hartwell et al., 1999; Purnick & Weiss, 2009).

ANNs have been applied to learning the mapping between the characteristics and the composition of bacterial community assemblages, both in terms of species abundance and of their distribution (Larsen, Field, & Gilbert, 2012). Using data on taxon abundances derived from measurements of 16S rRNA over 6 years, the researchers derived an environmental interaction network (EIN) in which nodes are microbial taxa and edges between them represent causal relationships. The EIN, in turn, was used to derive an ANN representation in the form of mathematical equations that best explain the data. This ANN was used to predict existing microbial taxa distributions, but these authors point out that this approach could also be used to design new ecosystems, by identifying the contribution of different factors to the make-up of the community. Synthetic ecosystems could be used, for example, to clean up contaminated sites or even to compensate for the effects of climate change.

Neural networks are powerful general-purpose learners of input/output mappings. They are widely used in a range of fields, including computational biology, but have not to date been widely used in the design of individual synthetic genetic circuits. When the synthetic biologist’s focus moves to engineering populations, and even entire systems, these algorithms have been shown to be valuable. The design of ANNs is something of a black art, and there are no rules as to the number and arrangement of neurons, connections, transfer functions and squashing functions needed to solve any given problem. In addition, ANNs generally require large amounts of labelled training data; the practitioners’ rule of thumb is that the training dataset should contain at least five cases for each connection in the network. Despite these drawbacks, ANNs are powerful and flexible algorithms for classification and input/output mapping and deserve a place in the arsenal of any practitioner of CI.

Introduction

Artificial Neural Networks are computational techniques that belong to the field of Machine Learning (Mitchell, 1997; Kelleher et al., 2015; Gabriel, 2016). The aim of Artificial Neural Networks is to realize a very simplified model of the human brain. In this way, Artificial Neural Networks try to learn tasks (to solve problems) mimicking the behavior of brain. The brain is composed by a large set of elements, specialized cells called neurons. Each single neuron is a very simple entity, but the power of the brain is given by the fact that neurons are numerous and strongly interconnected between them. The brain learns because neurons are able to communicate between each other. A picture of a biological neuron is shown in Fig. 1.

Fig. 1. Illustration of a biological neuron and its synapsis.

In analogy with the human brain, Artificial Neural Networks are computational methods that use a large set of elementary computational units, called themselves (artificial) neurons. And Artificial Neural Networks due their power to the numerous interconnections between neurons. Each neuron is able to only perform very simple tasks, and Artificial Neural Networks are able to perform complex calculations because they are typically composed by many artificial neurons, strongly interconnected between each other and communicating with each other. Before studying complex Artificial Neural Networks, that are able to solve large scale real-life problems, we must understand how the single neurons and simple networks work. For this reason, we begin the study of Artificial Neural Networks (from now on simply Neural Networks) with a very simple kind of network called Perceptron.