--- title: Training Artificial Neural Networks description: In this blog we will look at ai-driven image analysis for microscopy and how it contributes to advanced and accelerated scientific biology research. image: https://www.arivis.com/hubfs/5e7ca6971a6f9520424f00c8_fading_nuclei_segmentation.png --- arivis Scientific Image Analysis - [Contact](https://www.arivis.com/contact) Close - Products - ZEISS arivis Pro - [ZEISS arivis Pro](https://www.arivis.com/products/pro) - [ZEISS arivis Pro VR](https://www.arivis.com/products/pro/vr) - [ZEISS arivis Hub](https://www.arivis.com/products/hub) - [ZEISS arivis Cloud](https://www.arivis.com/products/cloud-ai) - Resources - [Publications](https://www.arivis.com/resources/publications) - [Case Studies](https://www.arivis.com/resources) - [Mitochondria Analysis with Deep Learning](https://www.arivis.com/case-study/deep-learning-segmentation-mitochondria) - [Compartmentalization of Chromatin](https://www.arivis.com/case-study/distribution-and-compartmentalization-chromatin-nucleus) - [Gastrulation & Migration Phenotypes](https://www.arivis.com/case-stydy/gastrulation-and-migration-phenotypes) - [Advanced 4D Tracking of Meiotic Spindle Dynamic](https://www.arivis.com/case-study/advanced-4d-tracking-of-meiotic-spindle-dynamic) - [Migration of Interneurons in Developing Mouse Brain](https://www.arivis.com/case-study/migration-of-interneurons-in-the-developing-mouse-brain) - [Exploring Whole Organisms at Single-cell Resolution](https://www.arivis.com/case-study/exploring-whole-organisms-at-single-cell-resolution) - [Image Analysis Workflow for EM images](https://www.arivis.com/use-case/image-analysis-workflow-for-mining-em-images) - [Webinars and Workshops](https://www.arivis.com/resources/webinars-workshops-trainings) - [Tutorials](https://www.arivis.com/tutorials-series) - [AI eBook](https://www.arivis.com/ai-ebook) - [AI Blog](https://www.arivis.com/blog) - How to's - [AI, ML and DL](https://www.arivis.com/applications/ai-machine-learning-deep-learning) - [Importing from Open Source](https://www.arivis.com/applications/exchange-objects-open-source-software) - [Region Growing](https://www.arivis.com/applications/region-growing) - [Membrane-based Segmentation](https://www.arivis.com/use-case/membrane-based-segmentation) - [Colocalization](https://www.arivis.com/use-case/colocalization) - [Customizable Magic Wand](https://www.arivis.com/use-case/customizable-magic-wand) - [Color Settings](https://www.arivis.com/use-case/use-case-color-settings) - [Tracking](https://www.arivis.com/applications/dynamics-and-tracking) - [Blob Finder](https://www.arivis.com/use-case/blob-finder) - [Visualization in Charts and 4D viewer](https://www.arivis.com/use-case/use-case-visualization-in-charts-and-4d-viewer) - Service & Support - [Knowledge Base](https://kb.arivis.com/) - [Software Newsletter](https://www.arivis.com/contact/newsletter) - [About](https://www.arivis.com/about-us) - [News](https://www.arivis.com/arivis-news) - [Career](https://www.arivis.com/career) - [Partners](https://www.arivis.com/technology-partner) - [Awards](https://www.arivis.com/awards) SEARCH [![Contact](https://hubspot-no-cache-eu1-prod.s3.amazonaws.com/cta/default/9058989/f6b72482-6e31-424f-9f83-11d8125b109d.png)](https://hubspot-cta-redirect-eu1-prod.s3.amazonaws.com/cta/redirect/9058989/f6b72482-6e31-424f-9f83-11d8125b109d) [![partner-logo-zeiss-1](https://www.arivis.com/hs-fs/hubfs/arivis-website-assets/images/partner/partner-logo-zeiss-1.png?width=48&height=48&name=partner-logo-zeiss-1.png)](https://www.arivis.com) [![partner-logo-zeiss-1](https://www.arivis.com/hs-fs/hubfs/arivis-website-assets/images/partner/partner-logo-zeiss-1.png?width=40&height=40&name=partner-logo-zeiss-1.png)](https://www.arivis.com) - Products - ZEISS arivis Pro - [ZEISS arivis Pro](https://www.arivis.com/products/pro) - [ZEISS arivis Pro VR](https://www.arivis.com/products/pro/vr) - [ZEISS arivis Hub](https://www.arivis.com/products/hub) - [ZEISS arivis Cloud](https://www.arivis.com/products/cloud-ai) - Resources - [Publications](https://www.arivis.com/resources/publications) - [Case Studies](https://www.arivis.com/resources) - [Mitochondria Analysis with Deep Learning](https://www.arivis.com/case-study/deep-learning-segmentation-mitochondria) - [Compartmentalization of Chromatin](https://www.arivis.com/case-study/distribution-and-compartmentalization-chromatin-nucleus) - [Gastrulation & Migration Phenotypes](https://www.arivis.com/case-stydy/gastrulation-and-migration-phenotypes) - [Advanced 4D Tracking of Meiotic Spindle Dynamic](https://www.arivis.com/case-study/advanced-4d-tracking-of-meiotic-spindle-dynamic) - [Migration of Interneurons in Developing Mouse Brain](https://www.arivis.com/case-study/migration-of-interneurons-in-the-developing-mouse-brain) - [Exploring Whole Organisms at Single-cell Resolution](https://www.arivis.com/case-study/exploring-whole-organisms-at-single-cell-resolution) - [Image Analysis Workflow for EM images](https://www.arivis.com/use-case/image-analysis-workflow-for-mining-em-images) - [Webinars and Workshops](https://www.arivis.com/resources/webinars-workshops-trainings) - [Tutorials](https://www.arivis.com/tutorials-series) - [AI eBook](https://www.arivis.com/ai-ebook) - [AI Blog](https://www.arivis.com/blog) - How to's - [AI, ML and DL](https://www.arivis.com/applications/ai-machine-learning-deep-learning) - [Importing from Open Source](https://www.arivis.com/applications/exchange-objects-open-source-software) - [Region Growing](https://www.arivis.com/applications/region-growing) - [Membrane-based Segmentation](https://www.arivis.com/use-case/membrane-based-segmentation) - [Colocalization](https://www.arivis.com/use-case/colocalization) - [Customizable Magic Wand](https://www.arivis.com/use-case/customizable-magic-wand) - [Color Settings](https://www.arivis.com/use-case/use-case-color-settings) - [Tracking](https://www.arivis.com/applications/dynamics-and-tracking) - [Blob Finder](https://www.arivis.com/use-case/blob-finder) - [Visualization in Charts and 4D viewer](https://www.arivis.com/use-case/use-case-visualization-in-charts-and-4d-viewer) - Service & Support - [Knowledge Base](https://kb.arivis.com/) - [Software Newsletter](https://www.arivis.com/contact/newsletter) - [About](https://www.arivis.com/about-us) - [News](https://www.arivis.com/arivis-news) - [Career](https://www.arivis.com/career) - [Partners](https://www.arivis.com/technology-partner) - [Awards](https://www.arivis.com/awards) [![Contact](https://hubspot-no-cache-eu1-prod.s3.amazonaws.com/cta/default/9058989/f6b72482-6e31-424f-9f83-11d8125b109d.png)](https://hubspot-cta-redirect-eu1-prod.s3.amazonaws.com/cta/redirect/9058989/f6b72482-6e31-424f-9f83-11d8125b109d) [Sebastian Soyer](https://www.arivis.com/blog/author/sebastian-soyer) Oct 13, 2023, 4:54:21 PM 15 min read # Training Artificial Neural Networks [Previous Post](https://www.arivis.com/blog/why-german-research-institute-fraunhofer-trusts-arivis-cloud) [Next Post](https://www.arivis.com/blog/scalable-image-analysis-for-biotech-and-pharma) ### **Bringing ANNs and Intellesis together** Training artificial neural networks (ANNs) to tackle semantic segmentation problems has become a very popular task and an increasing number of solutions have become available that require little technical understanding to train ANNs. Of course, this statement does not hold for models highly optimized for specific use cases where solving the problem at hand requires a high amount of experience, creativity and some “magic”. However, in many cases it is possible to achieve decent results with widely used ANNs model architectures, a sufficient amount of data and enough compute power. While about a decade ago solving computer vision tasks required an expert to design algorithms and understand the math behind them, ANNs and simple-to-use frameworks like [TensorFlow](https://www.tensorflow.org/) and [PyTorch](https://pytorch.org/) allow learning those algorithms from data and therefore can be treated as a black-box even achieving super-human performance in some cases. This simplicity and the high expressiveness of neural network models are presumably the reason that ANNs have become extremely popular and there is no way around this technology in research as well as industry. It should, however, be mentioned here that it is still extremely helpful to understand what happens behind the scenes. ### **The challenges of deploying a trained model to large images** While research mostly focuses on training and evaluating models, deployment is a different issue that poses different challenges. Especially in microscopy there usually are not only captured small images but stacks of images, time series of images or tiled scenes that are stitched to one large image of multiple or even hundreds of gigabytes. The training process, in contrast, is typically performed on small image crops (e.g. 256 x 256) such that the data for the calculations fit into the GPU or main memory of the employed machine. Applying a trained model on considerably larger images as mentioned above brings about several pitfalls that need to be handled. ![5e818feaeb01c52f634765a6\_nuclei\_input](https://www.arivis.com/hs-fs/hubfs/5e818feaeb01c52f634765a6_nuclei_input.jpg?width=450&height=325&name=5e818feaeb01c52f634765a6_nuclei_input.jpg)Input image with nuclei ![5e818f44349ea25259bf2588\_nuclei\_segmentation](https://www.arivis.com/hs-fs/hubfs/5e818f44349ea25259bf2588_nuclei_segmentation.jpg?width=450&height=325&name=5e818f44349ea25259bf2588_nuclei_segmentation.jpg)After Segmentation The [ZEN Intellesis](https://www.zeiss.com/microscopy/int/products/microscope-software/zen-intellesis-image-segmentation-by-deep-learning.html) infrastructure provides a solution for that problem. It applies a sophisticated tiling strategy that transforms multi-dimensional image captures – as they are generated by, for example, ZEISS microscopes – to image tiles of shape (height x width x color channels) where height and width are defined by the applied ANNs model. Finally, it performs inference with the model and reassembles the resulting segmentation masks to fit the initially captured image with intact metadata. It, therefore, offers a solution to apply a trained ANN to large images without worrying too much about the required memory and the size of the image. ### **czmodel – From TensorFlow to Intellesis** Starting with ZEN Blue3.2 it will be possible to run prediction with custom segmentation models implemented in TensorFlow 2. To be compatible with the Intellesis infrastructure, a TensorFlow/Keras model has to comply with our specification and some metadata must be provided that is described in detail in the official ANN model specification in the project description of our [czmodel package](https://pypi.org/project/czmodel/). This blogpost introduces a python library called ***czmodel*** that offers a set of tools to easily convert a trained TensorFlow/Keras model to a CZMODEL file that can be directly loaded into ZEN Intellesis. Generating a CZMODEL file this way is just a matter of a few lines of code and can therefore easily be integrated into any training pipeline as well as custom Apeer modules. In this blog post, we assume that there is some kind of training pipeline in place that generates trained TensorFlow/Keras models as instances of ***tensorflow.keras.Model***. For a simple example of such a pipeline see the attached Jupyter notebook in our [demo project](https://notebooks.azure.com/sebastian-soyer/projects/czmodel). Note that the notebook is not to be understood as a best practice guide for training ANN models but rather illustrates the use of the [czmodel](https://pypi.org/project/czmodel/) library with a very simple model generation process that will usually be more sophisticated. The entire ***czmodel*** package operates on so-called ***ModelSpec*** objects that contain all information needed to convert a TensorFlow/Keras model to CZMODEL. Specifically, it contains the trained model itself and a metadata object called ***ModelMetadata*** that reflects the JSON metadata described in the [ANN model specification as a python class](https://pypi.org/project/czmodel/). Thus, the typical workflow of converting a TensorFlow/Keras model to CZMODEL is to create a ***ModelMetadata*** object, wrap it into a ***ModelSpec*** object together with the corresponding TensorFlow/Keras model and, finally, provide the ModelSpec object to the conversion function that generates the CZMODEL file. The ***czmodel*** package supports three different ways to generate a CZMODEL. ### **Convert a load TensorFlow/Keras model to CZMODEL using a ModelSpec** This scenario reflects exactly the workflow described above. The *ModelMetadata* and *ModelSpec* objects for a loaded TensorFlow/Keras model named “model” are created as follows: ![5e8abada0a8ac27aff647e37\_code\_01](https://www.arivis.com/hs-fs/hubfs/5e8abada0a8ac27aff647e37_code_01.png?width=450&height=295&name=5e8abada0a8ac27aff647e37_code_01.png) ### **Convert a Saved Model on disk to CZMODEL using a ModelSpec** In this case, we assume the TensorFlow/Keras model not to reside in the main memory but on disk in the saved model format. Let “path” be the path to the ***SavedModel*** folder. We can then apply the same procedure as shown above with loaded models but instantiate the model spec object with the path of the model instead of the loaded model: **![5e8abb0d454b8a75607c62ef\_code\_02](https://www.arivis.com/hs-fs/hubfs/5e8abb0d454b8a75607c62ef_code_02.png?width=450&height=72&name=5e8abb0d454b8a75607c62ef_code_02.png)** The *czmodel* library will automatically load the model from the given path before wrapping it into a CZMODEL. ### **Convert a Saved Model on disk to CZMODEL by providing a JSON meta data file** This functionality replicates the import functionality of ZEN Intellesis to load models via a [JSON specification](https://pypi.org/project/czmodel/). If models should e.g. be exchanged with third parties, it is usually easier to deliver CZMODEL files that contain everything needed to run the model within Intellesis than exchanging JSON files with the corresponding SavedModel folders that would, nonetheless, need to be bundled another way. The ***czmodel*** library provides a simple conversion routine that does all this in one line:  ![5e8abb2c229137ac89058de9\_code\_03](https://www.arivis.com/hs-fs/hubfs/5e8abb2c229137ac89058de9_code_03.png?width=450&height=91&name=5e8abb2c229137ac89058de9_code_03.png) Note that the path to the model is implicitly defined by the provided JSON file. ### **Adding-processing and setting the tile size** Most trained models require some kind of pre-processing (e.g. normalization/standardization) to transform the input data to the distribution the model was trained on. The Intellesis inference engine requires this to be part of the model while – depending on the training infrastructure - this is not necessarily the case during training. All conversion functions in the ***model*** library wrap the provided model into a deployment model if pre-processing is applied or the spatial dimensions should be reset. They additionally accept a list of TensorFlow/Keras layers as a ***preprocessing*** argument that will be prepended to the provided model. Additionally, setting the parameter ***spatial\_dims*** allows redefining the spatial dimensions of the expected model input that implicitly defines the size of the tiles provided to the inference engine. Note that this is only possible for models that are invariant w.r.t the spatial dimensions. While the ***czmodel.util.preprocessing*** module provides a set of useful pre-processinglayers, any TensorFlow/Keras layer can in theory be used for pre-processing. Thefollowing call generates a CZMODEL from the above defined ***ModelSpec*** object, appends an ***RgbToBgr*** pre-processing layer to transform the input images to a different color space and sets the expected size of the input images to 512 x512: *![5e8abb3dbe13d4ea797588c5\_code\_04](https://www.arivis.com/hs-fs/hubfs/5e8abb3dbe13d4ea797588c5_code_04.png?width=450&height=250&name=5e8abb3dbe13d4ea797588c5_code_04.png)* ### **Remarks** The current implementation of the ***czmodel*** heavily relies on the SavedModel import/export functionality in TensorFlow 2. We, therefore, recommend to always use the latest version of TensorFlow 2 compatible with ***czmodel*** as there are continuously released improvements and bug fixes in the TensorFlow code base.   Sebastian Soyer ![5e74b2dc9c26203844c42031\_Sebastian\_Soyer-p-500-removebg-preview](https://www.arivis.com/hs-fs/hubfs/5e74b2dc9c26203844c42031_Sebastian_Soyer-p-500-removebg-preview.png?width=250&height=248&name=5e74b2dc9c26203844c42031_Sebastian_Soyer-p-500-removebg-preview.png)   ## RELATED ARTICLES **Related ZEISS Websites** [Research Microscopy](https://www.zeiss.com/microscopy/en/home.html) [ZEISS Group](https://www.zeiss.com/int) - [Privacy](https://www.zeiss.com/data-protection) - [Imprint](https://www.arivis.com/imprint) - [Contact](https://www.arivis.com/contact) Subscribe to our newsletter © Carl Zeiss Microscopy GmbH - All rights reserved - [Privacy Notice](https://zeiss.com/data-protection) - [Legal Notice](https://www.zeiss.com/microscopy/en/legal-information/legal-notice.html) - [Imprint](https://www.arivis.com/imprint) - [Contact](https://www.arivis.com/contact) - [Cookie Policy](https://www.zeiss.com/data-protection/int/cookie-policy.html)