Deep Learning CNN for the Prediction of Grain Orientations on EBSD Patterns of AA5083 Alloy

Indexing of Electron Backscatter Diffraction (EBSD) is a well-established method of crystalline material characterization that provides phase and orientation information about the crystals on the material surface. A deep learning Convolutional Neural Network was trained to predict crystal orientation from the EBSD patterns based on the mean disorientation error between the predicted crystal orientation and the ground truth. The CNN is trained using EBSD images for different deformation conditions of AA5083. Keywords-AA5083; microstructure; EBSD; machine learning; deep learning

INTRODUCTION Material science has resulted from the establishment of the new field of material informatics. Image data is one of the most prevalent forms of data in material research, given to advancements in numerous material imaging techniques. Deep Learning (DL) has lately resulted in ground-breaking improvements in a variety of domains, including the prediction of material characteristics. Experiments and simulations are used by materials science and engineering researchers to try to comprehend the Processing-Structure-Property-Performance (PSPP) interactions, which are far from completely understood. In fact, practically everything in materials science is dependent on these PSPP interactions. There is a need to better comprehend this complicated system of PSPP connections to identify and build new improved materials with desired qualities [1,2]. DL has emerged as a useful method for big data analysis. It employs complicated algorithms and Artificial Neural Networks (ANNs) to teach computers to learn, categorize, and identify data/images in the same way with the human brain [3]. The Convolutional Neural Network (CNN) is a type of ANN frequently used in DL for image/object detection and categorization [4]. CNNs are important in a variety of activities/functions such as image processing, computer vision tasks such as localization and segmentation, and predicting cracks on materials using microscope techniques by identifying grains and boundaries. CNNs are particularly popular in DL because they play a vital role in these rapidly increasing and new domains [5]. In addition, one of the most advanced techniques in material creation and investigation is Electron Backscatter Diffraction (EBSD). The most prevalent method for EBSD indexing is the Hough transform-based method, which is based on computing angles between linear features derived from the direction pattern. The fundamental issue with this approach is that its performance degrades fast in the presence of noise [1].
In this paper, a DL ANN is utilized to train a model that can predict the grain orientation due to different deformation conditions, i.e. temperature and train rate. The novelty of this work lies within the developed model which can predict the material crystallography without the need to do the timeconsuming EBSD characterization. The work shows the pattern changes due to the deformation which can be used to tailor the process in order to achieve the desired grain orientations.

II. ELECTRON BACKSCATTER DIFFRACTION
There are many examples of CNN applications in DL regarding material images. The most familiar CNN training technique is the EBSD which is a technique that uses a Scanning Electron Microscope (SEM) to provide crystallographic information on a sample's microstructure. A stationary electron beam interacts with a tilted crystalline sample in EBSD, and the diffracted electrons generate a pattern that a fluorescent screen can detect. The diffraction pattern reflects the crystal structure and orientation in the sample region where it was created. As a result, the diffraction pattern may be utilized to detect crystal orientation, distinguish crystallographically distinct phases, characterize grain boundaries, and offer information regarding local crystalline perfection [6][7][8]. EBSD has become a well-known accessory for SEM, which is commonly employed to give crystallographic information. As a result, EBSD is currently being used in a variety of diverse applications on material characterization [9][10][11].
Authors in [12] provided the first DL solution for the use of EBSD indexing. The aim was to provide an end-to-end solution that does not require specific expertise or image processing computing. The system is based on entering the raw EBSD patterns to predict the 3 numerical ones using CNNs designed to learn the spatial dependencies among pixels and extract relevant features. The network consists of many layers, including pooling, and a fully connected layer for the output regression. Historically, the focus of material research was to optimize the composition and process in order to provide target materials with optimum microstructure and performance matching. However, this strategy is inefficient since it relies on a lot of experimentation and trial-and-error experience. As a result, the Material Genome Project has been suggested to accelerate material research and development and aimed to develop internal links between processes, composition, microstructures, and properties to design microstructures with the required performance. According to this connection, the material's composition and procedure are developed and optimized. As a result, the key challenge of developing and optimizing materials is defining the quantitative relation between material composition/process, organizational structure, and performance [13][14][15].
Authors in [16] studied the material characteristics prediction using EBSD, which not only contains structural information, but it is also easier for computers to grasp. They built an EBSD-based digital knowledge graph representation and subsequently designed a representation learning network to integrate graph features. Finally, using graph embedding, they employed an ANN to predict material performance. They tested their approach on magnesium and compared it to existing machine learning and computer vision methods. The findings demonstrated the scientific validity of the suggested approach, as well as the practicality of property calculation. They made a representation method of the EBSD grain knowledge graph and used it for the prediction of organizational performance. Resulting from the application of the node and edge representations, they built a graph feature convolution network to incorporate the grain knowledge graph and achieve graph feature extraction. Then, using the graph feature, a feature mapping network based on an ANN was developed to predict material properties. They compared the proposed approach with two other classic statistics-based machine learning approaches and visual feature extraction methods. To get the material properties, they used traditional machine learning methods to directly calculate the attribute features of all grains. Furthermore, they utilized the CNN model to predict visual features from the microstructure map and predict performance. They demonstrated that the CNN technique is better than the other methods [16].

III. EXPERIMENTAL WORK
The chemical composition of the received materials is analyzed below. The specimens were machined in a rod shape with a diameter of 5mm and a length of 10mm. The rods were heat treated for 1h at 500°C for homogenization. The rod shape is shown in Figure 1(a). In order to guarantee consistency, different locations were tested for the microstructure of the material before and after heat treatment as shown in Figure  1(b). Table I    The experimental results show the effect of the stain rate and temperature of the stress which in turn have an effect on the kinetics of static recrystallisation and the stored energy distribution in the microstructure. After deformation, the specimens were water-quenched and cut into 8 pieces in order to construct a full recrystallisation curve. The specimens were then mechanically polished and electro-etched in 10% oxalic acid. The recrystallized grains were easily recognized by their smaller size and strain-free microstructure. The results of deforming AA5083 at different strain rates and temperatures (Table II)  According to the calculations, a final length of 9.8mm is expected for every sample (with the initial length of 10mm). All dilatometry work to date has been done on AA5083 alloy samples. The initial and the post-deformation size of the tested specimens were measured and recorded. All Al samples were prepared for the EBSD map. The deformed materials, after being recrystallized, are shown in Figures 6-14 for different polishing and EBSD deformation conditions.        IV. MACHINE LEARNING DL is a division of machine learning, which is effectively a three or more-layered neural network. These neural networks imitate the activity of the human brain, although with limited success, having the capacity to "learn" from vast volumes of data. While a single-layer neural network may still produce approximate predictions, more hidden layers can optimize and tune for accuracy [17]. Many Artificial Intelligence (AI) applications and services are relying on DL to boost automation by executing analytical and physical activities without human interaction [18]. Authors in [1], said that DL is the rediscovery of neural networks, which were algorithmically conceived in the '80s. During the recent years, the availability of big data and more capable computers has allowed these networks to become deeper. DL has enabled revolutionary advancements in a variety of domains, including computer vision and speech recognition [19]. Authors in [20] also said that the feedforward deep network, also known as the Multilayer Perceptron (MLP), is the archetypal DL model. An MLP is just a mathematical function that converts a set of input values to output values. Many smaller functions are combined to make the function. Each application of a distinct mathematical function may be thought of as presenting a new representation of the input. One viewpoint on DL is provided by the concept of finding the best representation for the data. Another viewpoint on DL is that it enables a machine to learn a multi-step computer program. Each layer of the image represents the state of the computer's memory following the execution of another set of instructions in parallel. Deeper networks can execute more instructions simultaneously [21]. To summarize, DL is an approach related to the AI technology. It is a form of machine learning technology, which allows computer systems to develop with experience and data. There are various types of DL models that are both accurate and successful at dealing with issues that are too intricate for the human brain, but the CNN technique has been applied in this research.

A. Convolutional Neural Networks
CNNs are a type of neural networks that is used to process input using a predefined, grid-like topology. Time-series data, which can be thought of as a 1D grid with samples at regular time intervals, and picture data, which can be thought of as a 2D grid of pixels, are two examples. CNNs have had a remarkable amount of success in practical applications. The term "convolutional neural network" refers to the network's use of a mathematical procedure known as convolution. Convolution is a subset of linear operations. Convolutional networks are basically neural networks with at least one layer that uses convolution instead of basic matrix multiplication, [20,21].

B. Structure of CNN
There are three main layers in a CNN structure, which are the convolutional layer, the pooling layer, and the archetypal fully connected layer.

1) The Convolutional Layer
This is the CNNs core layer. Its parameters are made up of a series of filters. Although these filters are small, they cover the entire depth of the input volume. The fundamental job of the convolutional layer is to extract high-level information. The first one is in charge of extracting low-level characteristics such as color, edges, and so on. Following convolutional layers remove the high-level characteristics, resulting in a comprehensive knowledge of the image.

2) The Pooling Layer
The function of this layer is to minimize the spatial size of the picture representation. As a result, it also helps to minimize the amount of computing and processing in the neural network. Furthermore, it extracts positionally and rotationally invariant dominant features. The Max operator is used to perform one type of pooling. This procedure selects the highest value from each neuron cluster in the previous layer. The other type of pooling is Average pooling, which yields the cluster's average value. Max pooling exceeds average pooling since it also works as a noise suppressor. In addition to convolutional layers, there are several pooling layers. The more of these layers there are, the more low-level features will be retrieved. However, the processing power required will increase as well. After the picture has been processed through all the present convolutional and pooling layers, feature extraction is complete.

3) The Fully Connected Layers
The fully connected layer, as the last layer, is a feedforward neural network. The flattened output of the prior pooling/convolutional layer is used as the input to the fully connected layer. To flatten is to unroll a three-dimensional matrix or array into a vector. A unique mathematical computation is performed for each FC layer. The SoftMax activation function is used over the last layer after the vector has passed through all of the fully linked layers. This is used to compute the probability that the input belongs to a specific job. Consequently, the final output is the various probabilities of the input picture belonging to distinct classes. The procedure is repeated for several image types and individual images within those types. This trains the network to distinguish between different images [23]. DL techniques are one of the superior future tools to materials developments and analysis, data-driven approaches in computer science are increasingly being used with great success on a wide range of material data. More data repositories are providing images and raw data to analyze. In addition, the advances in data science provide new algorithms and tools to analyze the data, holding much promise for successfully realizing the materials prediction goals and assisting in the discovery, design, and deployment of next generation materials [24,25].

A. CNN Training
The collected EBSD data are used for training and testing of the machine learning networks, EBSD pictures of the microstructure along the rolling direction of the thickness of the alumina sheet are obtained using electron microscopy. Machine learning requires a large amount of input data, the scanned EBSD images are digitized using 1332×417 image size, and the length is in the same direction of rolling ( Figure  3). There are slightly more than 2000 grains captured form each EBSD obtained from the different deformation conditions. Roughly, a total of 20000 grins are used in the training and testing procedure. The gain orientations are used as an input to train the machine learning algorithm. Due to size limitations, each image was subdivided into sub images in order to minimize the size of the input rather than descaling the image which would reduce resolution. Each EBSD image was divided to 8 images of 100×100 size. Some parts of the images were eliminated, such as low brightness edges and drawing scale. For the implementation of the DL algorithm, two schemes were used, the first network is a classifier of the deformation conditions, while the second is a predictor of the grain orientations after deformation. MATLAB DL algorithm was used to train the first network based on CNN, while the second network was based on the LSTM network for predicting the grain orientation. ReLU activation function was used for the input side (unsupervised learning) and Sigmoid function was used for the fully connected supervised learning output side. The algorithms were implemented in MATLAB and executed on an i5 computer. In order to obtain the best model, multiple runs were executed such that the best run was used to obtain the best network. Figure 15 shows the flowchart of the training and testing procedure as well as the selection of the best trained network.  Figure 16 shows the CNN network topology. This system has a single-colored image input, which is indicated as three colors (RGB), and an output channel which is divided into 10 bins to indicate the orientation of the grains. Initially, the input is processed by the multi-color convolution system, followed by the merging layer, and a down pooling layer. There are 3 stages of CNN, the 1 st consists of 8 filters, the 2 nd consists of 16 www.etasr.com Suker: Deep Learning CNN for the Prediction of Grain Orientations on EBSD Patterns of AA5083 Alloy filters, and the last of 32 filters. Each filter has 3×3 size. Finally, the last layer is the fully connected layer which contains 10 classes, presenting each deformation condition. The training algorithm is set to train the network using the nondeformed image only. Then transfer learning is used (purple box in Figure 15) with the initial system trained with the nondeformed EBSD on how the grins are misoriented due to the deformation conditions. The learning is done in batches using the contrastive divergence search algorithm. In this procedure, the training data set is randomized and then split into subsets. The algorithm processes the data in mini batches and evaluates the cost function. Subsequently, the network weights are updated accordingly. This is done consecutively over all the training batches. The complete process is known as an epoch.
For the training process a batch size of 10 is used [20]. The data are randomized and divided into 80% training and 20 testing sets. The training was validated every 10 epochs in order to avoid over fitting. A total of 100 epochs were used for the training process. After the training of all the different CNNs for the multiple parameter sets is complete, the best fit is selected after being tested. Figure 17 shows the training and validation process with the error and the loss function for training and validation data. The training accuracy reported in this process is 88.33%, while the testing is 81.2%.

B. Grain Orientation Modeling
The grain orientation prediction is based on the grain distribution and the number of grains at each direction. Figure  18 shows the grain orientation map as a function of color. Red grains are oriented at the 001 direction, green grains are oriented at the 111 direction, and blue grains are oriented at the 101 direction [26]. The orientation can be correlated to the color in the RGB color map. The color map ranges from 0 for black to 256 for white. Red color is 76, blue is 30 and green is 150. Hence, 001 grain will be at the 76 scale, 111 grain will be at the 150 scale, and 101 at the 30 scale. Each EBSD map is converted into a histogram of the grain colors as shown in Figure 19. Each Figure shows the as received orientations, and the three strain rates deformation as specific temperature, i.e. 350 o C, 375 o C and 400 o C. The orientation data obtained from the EBSD figures are collated together and randomized then divided into 70% training, 15% validation, and 15% testing subsets. The network used to train the data is an LSTM with 3 layers of 100, 50, and 20 nodes, followed by a fully connected layer and a regression layer. The network parameters set are: • MaxEpochs 500  After the NN training is completed, the best network with the least error is selected with the comparison of the best fit at epoch 100. The fitting and testing results are shown at Figure  20. The reported results are: Training performance = 0.7808, Validation performance = 0.8446, and Testing performance = 0.6843. VI. DISCUSSION In this study, AA5058 is the investigated material with regard to its microstructure. Machine learning networks were developed to predict the output based on all the interdependencies. The results proved that machine learning has learned the distribution functions of the singular parameters accurately. The generated output resembled the grain distribution accurately. The machine learning algorithm was trained on a relatively small sample size, due to the limited number of experimental tests that were conducted. Everyday research has limitations on the amount of deformation and the creation of the of EBSD pictures. The results from this study proved the concept on small sample sizes. Bigger sample size can improve the accuracy and generalize the system.
It is monitored that the microstructure changes due to deformation do impose changes in the grain orientation but in different magnitudes. In general, most of the grains have a shift in the orientation form 111 toward 101 and 001 orientation, and this is explainable due to the rolling. However, at low temperatures (350 o C), the effect of the strain rate seems to be minimal, but at higher temperatures, the grains seem to be retaining their original orientation at high strain rate, while at lower strain rate, the changes are maintained, due to the lower working rate. This could be explained as no change at high strain rate, or the material has memory such that the grains will move back into their initial conditions after deformation. This is a phenomenon that needs to be investigated experimentally by freezing the deformation at different stages.

VII. CONCLUSION
This study presented a methodology for predicting the grain orientation for AA5058 after hot deformation at different conditions. Most studies use singular distribution functions to describe the input for the microstructure model. However, this study showed that machine learning approach can be used to describe any given microstructure changes due to deformation. The manufacturing process incorporates multiple steps, like homogenization, hot rolling, cold rolling, and tempering. They often show a complex microstructure where grains need to be considered accurately as this effects the materials properties and manufacturing. The aim of this study was to investigate the effect of deformation on the grains' formation and orientation due to the deformation conditions and predict the orientation of the grains after deformation. For future work, other materials such as steel and copper alloys will be investigated. Steel alloys in particular, have many stages (martensite and austinite), having an influence on the grain orientation and material characteristics. The model needs to be extended to accommodate the different behaviors by including the phase information.