In a cooperative project, the Fraunhofer IWM has now developed such a method based on neural networks. It delivers reliable and reproducible results and is a fundamental building block for the end-to-end digitalization of steel processing within Industry 4.0. The results have now been published in the scientific journals “Nature Communications” and “npj – computational materials”.

Evaluating the microstructure of high-performance steels using neural networks

Steel for cars, trains or vans must be able to withstand extreme loads, such as vibrations and impacts caused by bumps and potholes while driving. In the event of an accident, it has to absorb the force of the impact and deform in a specific manner. For this reason, random samples of steel are tested during production and before processing. Metal specialists examine steel samples under a microscope to detect any weak points where micro-cracks could later develop and become major damage. These examinations are time consuming and their quality depends on the experience and attention of the metallographers.

For this reason, the materials processing industry has long been calling for automatic processes in which computers would quickly inspect the steel with an “expert's eye” and always with the same reliability. So far, however, automatic image analysis has reached its limits because the microstructure of steel is extremely complex: under the microscope, the surface of a steel sample appears as a wild mosaic of different structures. Identifying the critical weak points here is an artistic endeavor.

Efficiently trained neural networks recognize defect structures even in light microscopic images

In a joint project, a team from the Fraunhofer Institute for Mechanics of Materials IWM, Saarland University and Carnegie Mellon University in Pittsburgh has now succeeded in training artificial neural networks to analyze steel surfaces. The results have been published in the scientific journal “Nature Communications”. In doing so, the researchers overcame two previous challenges in image processing: the insufficient efficiency and interpretability of neural networks.

“Humans are incredibly good at efficiently learning structures and recognizing them in very different environments and contexts,” says Ali Riza Durmaz, lead author of the paper and a materials expert at the Fraunhofer IWM. “Artificial neural networks, on the other hand, must first be trained with a variety of image data.” This is time consuming because the images with which a neural network is trained must first be annotated by hand: the image defines exactly where the target objects are located. With chaotic patterns such as one finds on steel surfaces, this is a Sisyphean task.

Usually, metallographers use light microscopy or electron microscopy images to examine a steel sample. Much less commonly, the more complex electron backscatter diffraction (EBSD) is used, which describes the surface of the steel more comprehensively. The team has succeeded in linking the optical microscopy images with EBSD images, using the EBSD image data as a kind of automatic annotation.

“This automation allowed us to create a significantly larger, more informed and consistent data foundation to efficiently train neural networks,” Durmaz says. Now, he says, the software is able to detect defects even in light microscopy images, which can be produced quickly and easily. The focus was on high-quality complex-phase steels, which are for instance used automotive construction. These exhibit a special type of microstructure - the so-called bainite phase: Under the microscope, parallel structures can be seen that resemble wooden laths lying next to each other. These structures are not always clearly visible. Therefore, it can be difficult to distinguish the microstructures of the bainite phase from undesirable defects. The neural networks trained by the research team can now do this.

Safely detect microstructure deviations in different material surfaces

Almost simultaneously with the publication in Nature Communications, Ali Riza Durmaz has now published a complementary research paper in the journal “npj – computational materials”. In it, together with experts from the University of Saarland and the engineering school Mines ParisTech, he has solved another challenge of automatic image analysis.

Until now, the problem has been that neural networks “trained” on one type of steel microstructure can rarely be used for different steels or other materials. “The problem starts at the very beginning,” says Ali Riza Durmaz. “Depending on how you process a steel sample, how you grind it, etch it or expose it under a microscope, the microstructure appears different.” A neuronal network is then often off the mark when evaluating the image, he says.

Using certain learning methods, known as transfer learning, Ali Riza Durmaz and his team have now succeeded in making neural networks more flexible. “They are able to generalize like a human and recognize structures in different environments – i.e. in different steel samples or other materials,” Durmaz reports.

Industrial companies have long wanted such an automated image analysis method. At conferences, inquiries are constantly made asking for clear instructions on how to prepare samples in order to evaluate them automatically – how to treat the image data or adjust the pixel density, for example. “With the results from our two studies, we are now providing exactly that: a flexible analysis method that reliably and reproducibly detects deviations in the microstructure of different metal surfaces,” explains Ali Riza Durmaz. “This is also important especially for the end-to-end digitalization of metal production within Industry 4.0. Automated image recognition is a fundamental building block for the fully digitalized process chain.”

Publications

Durmaz, A.R.; Müller, M.; Lei, B.; Thomas, A.; Britz, D.; Holm, E.A.; Eberl, C.; Mücklich, F.; Gumbsch, P., A deep learning approach for complex microstructure inference, Nature Communications 12/1 (2021) Art. 6272, 15 pp.

Goetz, A.; Durmaz, A.R.; Müller, M.; Thomas, A.; Britz, D.; Kerfriden, P.; Eberl, C., Addressing materials' microstructure diversity using transfer learning, npj Computational Materials 8 (2022) Art. 27; 13 pp.

Fraunhofer-Institut für Werkstoffmechanik IWM

Wöhlerstrasse 11, 79108 Freiburg, Germany

Contact person is Ali Riza Durmaz

Tel.: +49 761 5142- 0

info@iwm.fraunhofer.de

www.iwm.fraunhofer.de