INFORMATIVE BOUNDS OF NEURAL NETWORKS PREDICTION FOR COMPOSITE FATIGUE LIFE UNDER VARIABLE AMPLITUDE LOADING

– In this study, the informative bounds of neural networks (NN) prediction with respect to the utilization of less fatigue data for fatigue life assessment of composite material covering a wide range of stress ratios R was examined and investigated. Fiberglass reinforced polyester of [90/0/±45/0] S lay-up with fatigue data of various stress ratios ( R = 0.1, 0.5, 0.7, 0.8, 0.9, -0.5, -1, -2 and 10) was examined in the present paper. Multi-layer Perceptrons (MLP) trained with Levenberg-Marquardt algorithm was utilized to result in fast and efficient NN model and Bayesian regularization technique was incorporated to deal with limited training data chosen for the model. The developed NN model was trained with fatigue data from only two stress ratios, where three sets of two stress ratio values were formed and used as the training sets, namely R = 0.1 and 0.5, R = 0.1 and -1, and R = 0.1 and 10, respectively. It was obtained that fatigue data from R = 10 produced the widest bounds of prediction, namely having the highest estimated standard deviation value from the fatigue lives predicted. Furthermore, it is revealed in the current study knowing the fact that fatigue data from R = 10 have the highest estimated standard deviation and subsequently including the fatigue data as one of the training data set, the NN model trained could produce the lowest mean squared error (MSE) value for the results of fatigue life prediction. This is justifying also the selection of training set of R = 0.1 and 10 as best training set in the previous study, which is based on the stress ratios’ better relative positions in the corresponding constant life diagram (CLD). Finally, taking the highest estimated standard deviation value from fatigue data of R = 10 as the conservative estimated bounds of NN prediction, it was shown that for the NN prediction of fatigue life whose noticeable discrepancies with the experimental data, the discrepancies were well confined within the conservative bounds of prediction.


Introduction
Modeling of composite fatigue life under complex and spectrum loading conditions remains a challenge to researchers in this field and hence it is still so attractive (Harris, 2003). Recently, new approaches have been pursued by the researchers giving the attention to the soft-computing frameworks. Among the frameworks, neural network (NN) is one of soft-computing tools that has been showing its potential and convenience for the application of composite fatigue life prediction, in particular under variable amplitude loading. A new route of fatigue life prediction has been formed in the field of application consisting of very active research during the recent years.
Considering further the inherent advantages of using NN relating with learning from examples, nonlinearity, feature extraction and generalization with respect to new loading or environment (Haykin, 2009), the researchers exploited NN to produce the non-parametric models capable of performing composite fatigue life prediction under variable states of stress (different stress ratio R values) with ease and efficient implementation. Following Lee and Almond (2003), Al-Assaf and El-Kadi (2001), and El-Kadi and Al-Assaf (2002) performed fatigue life assessment of unidirectional glass fiber/epoxy laminate using several NN paradigms, namely feed-forward (FF), modular (MN), radial basis function (RBF) and principal component analysis (PCA) networks. The works have presented comprehensive analysis and discussion about the utilization of different NN models in predicting fatigue life of unidirectional composite laminate. Freire Junior et al. (2007) also noticed the potential of NN models of feed-forward and modular on building Constant Life Diagrams (CLD) using only three S-N curves for predicting fatigue lives of multidirectional composite. It is also interesting to note that the comparative study between ANN models and conventional equations in the analysis of fatigue failure of GFRP has been also presented by the authors (2009), which showed that the NN outperformed conventional equations in the analysis of fatigue failure of GFRP. Furthermore, Vassilopoulos et al. (2007) also showed that using only randomly selected 40 -50 % of the experimental data were sufficient to produce reliable CLD using NN, emphasizing the NN convenience in performing fatigue life prediction of composite materials based on limited amounts of experimental data (Hidayat and Yusoff, 2009;Hidayat, 2015).
In the recent works, NN models have represented the predictive models for composite fatigue life assessment in which their feasibility and applicability have been also justified for a wider range of fatigue states. It can be also noted that there is interest and requirement for optimizing the NN prediction with the utilization of limited fatigue data. With this respect, it also turns out that there would be high concern and interest with the informative bounds of the NN fatigue life prediction. While producing the informative bounds of prediction is another potential advantage of using NN, this is, however, still less explored and not further examined in the previous investigations. In addition, with respect to the utilization of the less fatigue data, the bounds of NN prediction will provide additional important information and useful description in relation with any discrepancies of interest between fatigue lives predicted by NN and those obtained by experiments. The present authors notice that there appeared such noticeable discrepancies in the previous studies. Thus, it is necessary to provide a fatigue analysis with the information of prediction bounds.
In this study, the informative bounds of NN prediction with respect to the utilization of less fatigue data for fatigue life assessment of composite material covering a wide range of stress ratios R was examined and investigated. Fiberglass reinforced polyester of [90/0/±45/0]S lay-up with fatigue data of various stress ratios (R = 0.1, 0.5, 0.7, 0.8, 0.9, -0.5, -1, -2 and 10) was examined in the present paper. In Hidayat and Yusoff (2009), predicting the composite fatigue lives has been previously carried out with Multi-layer Perceptrons (MLP) trained with the Levenberg-Marquardt algorithm incorporating Bayesian regularization technique. The MLP trained with the training algorithm of Levenberg-Marquardt was utilized to result in fast and efficient NN model, while the Bayesian regularization technique was incorporated to deal with limited training data chosen for the model. The developed NN model was trained with fatigue data from only two stress ratios.
The fatigue life analysis of the composite is now extended, in which the corresponding bounds of NN prediction are further investigated and discussed. Note that the composite materials have been examined intensively in the previous research works concerning with composite fatigue life modeling and prediction.
The value of the present study is that supporting the obtained NN fatigue life prediction with the corresponding informative bounds of prediction, which will further support any subsequent product design decisions with fast and comprehensive information of fatigue lives. Another contribution of this study is presenting and describing how the bounds of the NN prediction in relation with the value of mean squared error (MSE) obtained during the testing phase of NN, which is also still not revealed in the previous study.
This paper is organized as follows: section II presents a brief explanation of NN theory and the NN architecture employed in the present paper. Material and Methods were presented in Section III. Furthermore, simulation results obtained were presented and discussed in Section IV. Conclusions drawn from the present study and direction for further study were and highlighted in section V.

Neural Networks Learning
NN is inspired by the biological network of neurons in the human brain that learns from external experience, handles imprecise information, stores the essential characteristics of the external input and generalizes previous experience. It consists of many interconnected simple processing units, called neurons or nodes, which are organized into layers. The nodes receive input data which are converted, by means of a suitable activation function, into a single output or a set of outputs. Depending on how information from the external inputs passed and stored, the architecture of NN can belong to feed-forward and feed-back networks (Bishop, 1995;Gupta et al., 2003;Nocedal and Wright, 2006).
In the present investigation, the feed-forward NN architecture of Multi-layer Perceptrons (MLP) with one hidden layer was employed. Figure 1 shows an MLP with one hidden layer and single output. In Fig 1, the notations given are: p input sets, L number of elements in input vector, s number of hidden nodes, n the summed up of weighted inputs, a the output of activation function in the corresponding layer, w 1 j,i and b 1 j input weight and bias (i = 1 to L, j = 1 to s), w 2 1, j and b o layer weight and output bias, and y the MLP output. Superscripts 1 and 2 represent the first layer of hidden and the second layer of output, respectively.
where: τ(.) is a sigmoid function used in the nodes of hidden layer, given by: Given the observation set O consisting of Q training examples, learning in NN for realization of the estimate fˆ means adjusting to vector of parameters weight w and biases b using a set of learning rule or learning algorithm in such a way that fˆ minimizes the objective function or empirical error defined as: 2 and generalizes well when a novel input vector p test never seen before is fed into the NN. It should be also mentioned that the availability of the training examples is not always unlimited, while at the same time reliable NN prediction is needed. This is frequently the case for fatigue analysis.

Material
The investigated material was E-glass/polyester, typical materials used in wind turbine blade applications. Named DD16 with the corresponding lay-up of [90/0/±45/0] S , representing the earlier material used in the applications. Moreover, the material has been studied extensively in recent years (Mandell and Samborsky, 2010). The corresponding database containing fatigue data of various R values (R = 0.1, 0.5, 0.7, 0.8, 0.9, -0.5, -1, -2 and 10) makes it suitable for the study purpose.
From the fatigue data, stress ratio R, maximum stress (S max ) and minimum stress (S min ) values were used as input set and the output was the corresponding fatigue cycles (log N) for the input set. For each particular R value, mean fatigue life values were used. All the data were normalized in range of -1 to 1.
Furthermore, several sets of the stress ratios were examined and evaluated, through which very useful understanding in how NN responds to this fatigue life prediction problem can be assessed and also which stress ratios might be appropriate to be selected and collected first. Furthermore, it is also possible through this study to describe how the bounds of the NN prediction in relation with the value of mean squared error (MSE) obtained during the testing phase of NN and describing also the relation with the stress ratios used as training sets, as highlighted in the previous section.
The rational of using the combinations of stress ratios are then as three-fold: First, the selected combinations of stress ratios represented two halves sector of CLD. Thus, the selections represent different representative conditions of fatigue state and capture the behavior of fatigue data throughout the CLD regions to give better prediction results for the NN modeling. Second, failure mechanisms of fatigue under T-T condition are different from the ones under C-C condition. Failures of fiber breaking, interfacial de-bonding and fiber pull-out play important role in fatigue under T-T, meanwhile fiber kinking, buckling and fiber-matrix shear-out dominate in fatigue under C-C. Therefore, the selections were chosen to examine the NN modeling effectiveness in constructing the CLD regions. Third, the error bounds of fatigue life predicted by the NN which are correlated with the training data from the set of stress ratios used. Thus, the estimated standard deviation values will be further known with respect to the variations of training sets chosen. The later will be more highlighted in the next section of simulation results and discussion.
In the current study, three combinations of stress ratios were determined and selected for the composite. The training sets were R = 0.1 and 0.5 (T-T and T-T), R = 0.1 and -1 (T-T and T-C), and R = 0.1 and 10 (T-T and C-C). The combinations of R values are summarized in Table 1.

Levenberg-Marquardt with Bayesian Regularization
Regularization is a way to prevent over fitting, a condition where good generalization is hard to occur as previously mentioned. Bayesian regularization was utilized in this study to improve the generalization capability, thus the quality of NN prediction. It minimizes a linear combination of squared errors and weights. The idea is to find a balance between the number of parameters and goodness of fit by penalizing large models. Incorporating the regularization, the objective function of Eq.
(2) is subsequently modified as follows: where: α is a weight decay parameter, β is an inverse noise variance parameter and W is the total number of weights.
The noise is assumed to apply to the target data t q and its distribution is assumed to be a zero-mean Gaussian distribution. The new objective function now consists of the sum of squared errors (E D ) and the sum of squared weights (E w ) terms. In addition, parameters α and β are introduced for penalizing large models. Using the modified cost function, it is also clear that there is a need to re-estimate the parameters accordingly. One estimation method is the Gauss-Newton approximation implemented to the Bayesian learning within the framework of the Levenberg-Marquardt algorithm (Foresee and Hagan, 1997;MacKay, 2004).

Bounds of NN Prediction
It is worth to note that the Bayesian scheme that incorporated in the NN modeling also allows the estimation of the corresponding bounds of NN prediction to be made. The bounds of NN prediction represent the predictive distribution, where the distribution has the realization of the NN model fˆ(p; wMP) as the mean and the corresponding variance given by: where: σ t is the standard deviation of the predictive distribution, G is the gradient of the NN model output with respect to the weights (and biases), H is the Hessian matrix. Here, the most probable value for the weight vector, denoted by wMP, corresponds to the minimum of the right-hand side in Eq. (5).
Further, the standard deviation of the predictive distribution can be interpreted as an error bar on the mean value fˆ(p; wMP). It can be seen that the error bar has two contributions, one arising from the intrinsic noise on the target data and one arising from the width of the posterior distribution of the network weights, corresponding to the first term and the second term in Eq. (5), respectively. Because the realization of the NN model corresponds to the training examples inputted, it can be easily understood that the standard deviation or the corresponding bounds of NN prediction produced will correlate with the training data sets. For complete description, see (MacKay, 2004).

Results and Discussion
Figs 2-4 show the fatigue life prediction results of NN when using training sets of R = 0.1 and 0.5, R = 0.1 and -1, and R = 0.1 and 10, respectively. It can be seen that the NN prediction results were consistent with the experimental data, in which the NN model shows its ability to generalize well in a wide range of R values.   The NN prediction was measured in mean squared error (MSE). The MSE values of prediction for the three training sets were 0.25, 0.23 and 0.19, respectively. The values show that best fatigue life prediction results were achieved when using fatigue data from stress ratios of R = 0.1 and 10 as training data set.
As previously mentioned, the results depicted in Figs 2-4 are the realization of the NN model for the fatigue life predictions based on the training data of the stress ratio sets chosen. With respect to the utilization of the different training sets, the error bounds of fatigue life predicted by the NN can be further described, which are represented by the estimated standard deviation values. The values are further investigated and described in the present section. It is also interesting to note that, besides correlated with the training set used, the standard deviations are also examined and stated in relation with the value of mean squared error (MSE) obtained during the testing phase of NN and also with respect to the number of hidden nodes used in the NN modeling. This is, to the authors' best knowledge, never revealed in the previous study.
Looking at further Figs 2-4, it can be seen that there exist rather large or significant discrepancies between the NN prediction and the experimental data for stress ratio R = -0.5. The discrepancies are observed for all the training sets and are shown more clearly in  For all the training sets chosen, the informative bounds of prediction for the stress ratio and for the other stress ratios are listed in Table 2 in term of the corresponding standard deviation values. Note that here only some stress ratios, namely, R = 0.5, -0.5, -1, -2 and 10, are selected as the representatives. Note also that the results were taken from many simulation runs, in which the number of hidden nodes employed was 10.
From Table 2, it can be seen clearly that for the different training sets used, the estimated standard deviation value produced different results for fatigue lives of the testing sets. In addition, for each the predicted fatigue life, which corresponds to a certain stress level, the estimated standard deviation resulted in also different value. This means that the widths of the informative bounds of NN prediction produced, hence, will also be different.
Hence, the corresponding bounds of NN prediction are correlated with the training data sets used or the examples inputted. Furthermore, it is also interesting to note that the lower or higher MSE value of the NN fatigue life prediction with respect to the training sets used did not necessarily produce the lower or higher estimated standard deviation value for each stress level of the same stress ratio R. For example, the estimated standard deviation values for the predicted fatigue lives of R = -0.5 with respect to the training set I compared to those of training set II and of training set III, and so forth. As previously mentioned, the training sets gave the MSE values of 0.25, 0.23 and 0.19, respectively. Here, it must be pointed out again that the results read for the number of hidden nodes of 10. Later, a sensitivity analysis of the estimated standard deviation value with respect to the variation of hidden nodes number will be also examined and discussed.
Nevertheless, it can be observed clearly that the fatigue data from R = 10 produced the highest estimated standard deviation values, both resulted from the training sets I and II. It can then be said that the stress ratio produced the widest bounds of prediction. In general, the corresponding estimated standard deviation values for the stress ratio were also high.
It is interesting to also note that including or involving the fatigue data of R = 10 with the highest estimated standard deviation as one of the training data set, the NN model trained could produce the lowest MSE value for the results of fatigue life prediction, namely MSE = 0.19. Knowing this fact, it could be said that not only with respect to the MSE value of the NN fatigue life prediction obtained will have certain stress ratio(s) a strategic value, for example, as the one(s) to be collected first, but also with respect to the estimated standard deviation value produced in relation with the selection of the training set. Moreover, this justifies also the selection of training set of R = 0.1 and 10 as best training set in the previous study, which is based on the stress ratios' better relative positions in the corresponding constant life diagram (CLD).
Further, one subsequently could take the widest bounds of prediction as the conservative estimated bounds of NN prediction and benefited from the values as the informative bounds of the predicted fatigue lives, in particular to describe any noticeable discrepancies with those of experimental data. Note that the results obtained were corresponding with the utilization of the hidden nodes number of 10. Further sensitivity analysis was examined to investigate the estimated standard deviation values in relation with the variation of the hidden nodes number. In addition, from the information obtained through the sensitivity analysis, the estimated standard deviation values corresponding to the number of hidden nodes that resulted in the best MSE value of the NN fatigue life prediction could also be obtained. Thus, all the result consistencies are highlighted and ensured through the analysis. It is important to point out that for each variation of the number of hidden nodes, the estimated standard deviation for the predicted fatigue lives of R = 10 always produced the highest value. The estimated standard deviation values for the predicted fatigue lives of R = 10 are selected and presented in Table 3.
It can be seen clearly from Table 3 that the estimated standard deviation values were consistently decreasing with respect to the increase of the hidden nodes number in the range of the hidden nodes number variation considered in the present study, for both the training sets examined.
It was also obtained from the previous study (Hidayat and Yusoff, 2009) that the number of hidden nodes of 15 was the optimum number of hidden nodes for the material examined in the present paper. Thus, the highest estimated standard deviation value as the conservative bounds of NN prediction was consistently taken from the optimum number of hidden nodes, which is 1.406 and of the training set II, as also shown in Table 3. Figs 6 and 7 show the bounds of NN prediction, lower and upper, by considering the highest estimated standard deviation value from the stress ratio R = 10, which is resulted in from the training set II, as shown previously in Table 3. The noticeable discrepancies with the experimental data in Figs 6 and 7 are the ones of the fatigue lives of R = -0.5 predicted using the training sets I and II, respectively. As can be seen from Figs 6 and 7, the informative bounds of NN prediction represented as the lower and upper bounds of prediction were provided and the noticeable discrepancies were well confined within the conservative bounds of prediction from the highest estimated standard deviation value. Combined forward (Hidayat et al., 2016;Hidayat et al., 2020) and inverse techniques in fatigue life assessment of materials may be further explored to obtain deeper analysis on the aspect of prediction bounds.

Conclusions
The informative bounds of NN prediction for the composite fatigue life under variable amplitude loading were examined and investigated in the paper. The developed NN model has been extended to account for the estimated standard deviations of the fatigue lives predicted by the NN model trained using the Levenberg-Marquardt algorithm with Bayesian regularization. Examining fiberglass reinforced polyester of [90/0/±45/0], it was obtained that fatigue data from R = 10 produced the widest bounds of prediction, namely having the highest estimated standard deviation value from the fatigue lives predicted. It is also further revealed in the current study knowing the fact that fatigue data from R = 10 have the highest estimated standard deviation and subsequently including the fatigue data as one of the training data set, the NN model trained could produce the lowest MSE value. The MSE values of prediction results were 0.25, 0.23 and 0.19 for the training sets of R = 0.1 and 0.5, of R = 0.1 and -1, and of R = 0.1 and 10, respectively. The result obtained is justifying also the selection of training set of R = 0.1 and 10 as best training set in the previous study, which is based on the stress ratios' better relative positions in the corresponding constant life diagram (CLD). Finally, taking the highest estimated standard deviation value from fatigue data of R = 10 as the conservative estimated bounds of NN prediction, it was shown that the present noticeable discrepancies with the experimental data were well confined within the conservative bounds of prediction. Further investigations on the estimated standard deviation value produced with respect to a wider range of the variation of the hidden nodes number and/or the other material composite applications are interesting for further research and study.