Focused ion beam scanning electron microscopy (FIB-SEM) is a powerful imaging tool that achieves resolution under 10 nm. Though it produces highly detailed 3D image volumes, one drawback is that it is difficult to use standard image processing segmentation algorithms to detect many cellular structures of interest. This is largely because FIB-SEM highlights the entirety of the cell, generating images dense with cellular features, structural edges, and varying pixel combinations. Due to this difficulty, quantitative analysis of FIB-SEM data often relies on manual drawing of features of interest on 2D slices of a 3D image volume. Though this manual approach can be used to identify and reconstruct 3D objects from the image volume, it is tedious and time-consuming.
Authors: Andrew Bergen, Mariia Burdyniuk, and Chris Zugates, arivis
Dataset: Anna Steyer and Yannick Schwab, EMBL
Workflow chart of ZEISS arivis PRO (formerly Vision4D) pipelines used for segmentation of cellular structures.: Each PART labeled under the steps refers to a specific pipeline (see below for availability of detailed workflow and example dataset).
Having defined mitochondria and nucleus, we used the measurement and visualization tools in ZEISS arivis Pro (formerly Vision4D) to examine the cytoplasmic organization of the HeLa cell. We noticed that, even though our images are of low quality compared to current state-of-the-art FIB-SEM, we could visualize the nuclear membrane and the nuclear pores and sought to develop a method to assess their distribution.
Our method uses 3D operations to enhance and segment 3D spatially resolved nuclear pore complex (NPC)-associated objects in a way that is not possible by segmenting each 2D plane separately within the image stack. A series of pipeline workflows in ZEISS arivis PRo (formerly Vision4D) provided the NPC regions of the nuclear membrane.
To ensure accurate segmentation, we performed NPC segmentation on the subset of the stack and used these objects for neural network training that respects the 3D nature of the data. In the final part of the workflow, we used this deep learning model to reconstruct the probability map of the NPC positions and segment the nuclear pores for the entire nucleus. Overall, this work highlights how the deep learning model approach, when combined with the powerful 3D tools of ZEISS arivis Pro (formerly Vision4D), enables 3D segmentation and measurements within FIB-SEM image sets.
The image set was collected using a ZEISS Auriga Crossbeam FIB-SEM, resulting in a nm-resolution image volume of the HeLa cell (Figure 1A). To initially examine the data and begin 3D analysis, we inverted the pixel intensities (Figure 1B) to achieve positive signals in a dark background. Subsequent analysis operation functions were then, for the most part, working on positives, which makes the whole of analysis-building easier to conceptualize. Of course, since much of the image is background (non-stained cellular structures and resin), this also makes it easier for humans to focus on structures of interest (think traffic lights, dark mode, etc.). Importantly, 3D volumetric renderings of the image volume do not make sense without a positive signal in black background (as seen in Figure 1C-D).
Figure 1. Overview of HeLa cell image set
Figure 2. Generation of deep-learning models for organelles using the arivis AI toolkit on ZEISS arivis Cloud (formerly APEER)
Figure 3. Segmentation results from a deep learning trained model can predict the percent of cell volume for organelles
Figure 4. Mitochondrial surface area-to-volume ratios are negatively correlated with the distance to membranes.
The accuracy of our whole nucleus and nuclear membrane masks enabled us to use the volumetric rendering and clipping tools in ZEISS arivis Pro (formerly Vision4D) to explore the nucleus and nucleus-associated structures in 3D. Struck with how well we could see the nuclear pores in this volume, we wondered whether we could use ZEISS arivis Cloud and ZEISS arivid Pro (formerly APEER and Vision4D) to segment and measure them. However, within individual 2D planes the nuclear pores are difficult to recognize. The resolution of the image provides only a 100-150 voxels per pore (for reference, the total number of pixels within the image is greater than 1 billion) and the 3D structure of each pore is uniquely oriented to the curvature of the nuclear membrane.
Figure 5. Identification of pocket objects under nuclear pores.
Figure 6. Nuclear pore complexes (NPCs) have variable density distribution across areas of the nucleus. Several processing steps were done to create masks of nuclear pores complexes from the pocket objects. Taking the pocket objects (A), a binary masked image was generated (B) followed by a closing operation of the pockets to nuclear membrane (C). Next, the nuclear membrane and pockets were used to mask the white space in shown in panel C (D). These objects were then dilated (E). Masking using these objects enhances the visualization of nuclear pore complexes (F).
Next, we set out to mask NPCs to create ground truths for a new 3D-aware deep learning neural network that will segment the NPCs directly. We were able to use the under-NPC- objects to derive objects representing the actual pores. Our initial strategy to segment the NPC particles required computation of a directional regional growth vector from the 3D geometric centroid of each under-NPC object towards the nuclear membrane, which would in high probability put us within the associated NPC and allow us to create an accurate mask. Instead, we discovered a way to achieve a roughly similar result without the need to leverage coding. Several masking and morphology operations were utilized to segment the volume between each pocket and the outer part of the nuclear membrane (Figure 6A-D). This volume was then dilated to cover the entire NPC (Figure 6E). Making a new image mask from these objects highlights the nuclear pore complexes visualized on Figure 6F. While not as precise a result as we would get from our original concept, this worked remarkably well and will allow for a non-manual creation of ground truth annotations for many thousands of pores for the subsequent deep learning training.
Once the segmentation of the NPCs was complete, the image stack and the corresponding NPC mask were rotated 30°, 60° and 90° on the X and Y axes and the resulting stacks were resampled to provide the 3D-aware augmented images of the 2D Deep learning algorithm on ZEISS arivis Cloud (formerly APEER) (Figure 7).
Figure 7. Preparation of the ground-truth dataset for the 3D-aware nuclear pore training on ZEISS arivis Cloud (formerly APEER). Nuclear pores were segmented using 3D morphological operators in ZEISS arivis Pro (formerly Vision4D) on a small representative image subset. Using a python script, the entire 3D stack was rotated on the two axes with linear interpolation. Each of the resulting stacks was resampled and every 5th plane was used for the 2D training of the deep learning model in the cloud.
Figure 8. Training a 3D-aware neural network for nuclear pores segmentation. Several processing steps were done to create masks of nuclear pores complexes from the pocket objects. Taking the pocket objects (Figure 6), a binary masked image was generated followed by the 3D-aware resampling (Figure 7) in preparation for the deep learning model training. The resulting CZANN model was used to create the probability map in ZEISS arivis Pro (formerly Vision4D) with the Deep Learning Reconstruction operator (B). This 3D stack was filtered using the ‘Preserve bright particles’ operator and the objects were segmented using the watershed algorithm with a strict threshold (C). In the following step, the smaller subset of the particles was expanded by region-growing, while the largest particles were split and filtered with the segment feature filter (D)
We used these objects to view and quantify the 3D distribution of NPCs throughout the nuclear membrane. To accomplish this, we took two separate approaches. In the first approach, we used the ZEISS arivis Pro (formerly Vision4D) Distances operator. For each nuclear pore object, the average distance to the nearest 8 other nuclear pore objects was measured. Color-coding these objects according to this average distance to the nearest 8 objects was used to represent the density of pores across the nuclear membrane (Figure 9A). Since this measurement based on average distances gives a representation of nuclear pore densities, we wanted to compare this measure with other density measures. Therefore, as a second approach, we used the ZEISS arivis PRo (formerly Vision4D) python application program interface (API) to make a custom python operator integrated with the software. Specifically, this python operator takes the 3D centroids of all the objects and uses the kernel density function within the scikit-learn python library to calculate their 3D densities. This kernel density function calculates a score based on the number of other objects in proximity, with a Gaussian smoothing of scores over a given radius. This custom python operator then exported the resulting density scores as a numeric feature of these objects within the software. As above, we color-coded these objects according to their density score to permit visual assessment of the distribution of these scores (Figure 9B). The similarity between these two methods demonstrated that both the Distance operator and the kernel density python script are capable of consistently identifying clusters of pores.
To further characterize NPC distribution across the nuclear membrane, the nucleus was divided into two sections based on the nuclear invagination (Figure 9C). Taking the density scores of these two sections highlights that NPC density is higher within the smaller section of the nucleus with higher curvature (Figure 9D). In contrast, the larger section with a lower curvature degree has more low-density regions for nuclear pores. Overall, these distributions indicate that there is variation in our measured nuclear pore density over different portions of the nucleus.
Figure 9. Nuclear pore complexes (NPCs) have variable density distribution across areas of the nucleus. A. The average distance of each nuclear pore object to the nearest eight nuclear pore objects was measured using the Distance operator in ZEISS arivis Pro (formerly Vision4D). The nuclear pore objects were then color-coded according to these distance measurements to give a representation of the density of nuclear pores across the nuclear membrane. B. As an alternative method of analyzing the distribution of the pore objects, the densities of nuclear pore complexes were determined by taking the 3D centroid of each NPC object and calculating a Gaussian kernel density, with a kernel radius of 0.1 µm, using a custom python script. C. The density distribution of NPCs is significantly different across separate areas of the nucleus. Sectioning the nucleus into two sections, a larger and a smaller section, based upon the nuclear cleavage furrow reveals significant differences in kernel density scores. D. A two-tailed t-test was performed to calculate the significance of differences between the kernel density scores in these two sections of the nucleus.
