Is This Preprocessing Pipeline Suitable for H&E-Stained RGB Input?

[vadori]

Hi,

I’m working with RGB histology images, specifically H&E-stained images with size 256x256 from the Conic and Pannuke datasets, and would like to verify if the preprocessing function below is appropriate for using the model with the function segment_cellular_image:

def process_image_array(image_array):
    # Ensure input is float32 for calculations
    img = image_array.astype(np.float32)

    # Get original size
    h, w, _ = img.shape
    desired_size = 512

    # Compute average across channels before normalization
    avg = image_array.mean(axis=2)

    # Normalize average to [0, 1] using min-max scaling
    avg_normalized = (avg - avg.min()) / (avg.max() - avg.min() + 1e-8)

    # Create output RGB image with only blue channel set
    output = np.zeros((*avg_normalized.shape, 3), dtype=np.float32)
    output[..., 2] = avg_normalized

    # Pad to 512x512 (centered padding)
    h_new, w_new = output.shape[:2]
    pad_h = max(0, desired_size - h_new)
    pad_w = max(0, desired_size - w_new)
    pad_top = pad_h // 2
    pad_bottom = pad_h - pad_top
    pad_left = pad_w // 2
    pad_right = pad_w - pad_left

    output_padded = np.pad(
        output,
        ((pad_top, pad_bottom), (pad_left, pad_right), (0, 0)),
        mode='constant',
        constant_values=0
    )

    # Convert to (3, 512, 512)
    output_ch_first = np.transpose(output_padded, (2, 0, 1))

    return output_ch_first

In summary, I am:

• Converting the input RGB image to float32.
• Computing the average intensity across RGB channels.
• Min-max normalizing the average to the [0, 1] range.
• Creating an output RGB image where only the blue channel is set to the normalized average.
• Padding the image (centered) to 512×512 if necessary.
• Returning the final image in (3, 512, 512) channel-first format for model input.

I’d appreciate any feedback on whether this pipeline is suitable for RGB, particularly H&E-stained, histology images since I am not getting good segmentation results. The pipeline is modeled upon the instructions given in the paper.

[rossbar]

I'd recommend to try cellsam_pipeline first with the raw image without any custom preprocessing. Any necessary image processing (which essentially amounts to casting to float and normalizing to the range [0, 1]) already happens internally within cellsam_pipeline, so there's no need to do any additional preprocessing beforehand.

As to whether this posted preprocessing function will work... I expect so, but perhaps not as well as the raw image as some of the operations (e.g. the averaging) likely move the input data further away from the training corpus. No real idea though, just my opinion!

[vadori]

Hi @rossbar,

Thank you for your response. I’ll go ahead and test the cellsam_pipeline. I based the steps above on a passage from the Supplementary Materials of the paper, aiming to minimize the discrepancy between the RGB images used during training and those I’m using for inference. As for the averaging step I should probably remove it. It was the CellPose-SAM paper that led to some confusion in that regard.

From https://arxiv.org/pdf/2311.11004:

From https://www.biorxiv.org/content/10.1101/2025.04.28.651001v1.full.pdf:

[vadori]

Here are the results on 5 images, the first 2 from the CoNIC dataset and the last 3 from PanNuke, comparing outputs from the cellsam_pipeline and my own pipeline, tested with both full RGB input and blue-channel-only input (with averaging). Overall, the results aren’t very good. The cellsam_pipeline seems to perform best on some, but it still misses many cells and includes clear false positives. For CoNIC, I also upsampled the images since the model seems to perform best at 40× magnification (CoNIC is 20×, PanNuke is 40×). I know that handling varying magnifications is a non-trivial challenge, but any general suggestions for improving CellSAM’s outputs would be greatly appreciated. I need this for benchmarking.

import numpy as np
import cv2
import matplotlib.pyplot as plt
from cellSAM import segment_cellular_image, cellsam_pipeline

def center_crop_mask(mask, crop_size=256):
    h, w = mask.shape
    top = (h - crop_size) // 2
    left = (w - crop_size) // 2
    return mask[top:top + crop_size, left:left + crop_size]

def process_image_array(image_array, bluechannel=False):
    # Ensure input is float32 for calculations
    img = image_array.astype(np.float32)

    # Get original size
    h, w, _ = img.shape
    desired_size = 512

    if not bluechannel:
        output = (img - img.min(axis=(0, 1))) / (img.max(axis=(0, 1)) - img.min(axis=(0, 1)) + 1e-8)
    
    else:

        # Compute average across channels before normalization
        avg = img.mean(axis=2)

        # Normalize average to [0, 1] using min-max scaling
        avg_normalized = (avg - avg.min()) / (avg.max() - avg.min() + 1e-8)  # small epsilon to avoid divide by zero

        # Create output RGB image with only blue channel set
        output = np.zeros((*avg_normalized.shape, 3), dtype=np.float32)
        output[..., 2] = avg_normalized

    # Pad to 512x512 (centered padding)
    h_new, w_new = output.shape[:2]
    pad_h = max(0, desired_size - h_new)
    pad_w = max(0, desired_size - w_new)
    pad_top = pad_h // 2
    pad_bottom = pad_h - pad_top
    pad_left = pad_w // 2
    pad_right = pad_w - pad_left

    output_padded = np.pad(
        output,
        ((pad_top, pad_bottom), (pad_left, pad_right), (0, 0)),
        mode='constant',
        constant_values=0
    )

    # # Convert to (3, 512, 512)
    # output_ch_first = np.transpose(output_padded, (2, 0, 1))

    # return output_ch_first
    
    return output_padded

def upsample_rgb_image(image):
    return cv2.resize(image, (512, 512), interpolation=cv2.INTER_LINEAR)

def downsample_label_mask(mask, target_size):
    return cv2.resize(mask, target_size, interpolation=cv2.INTER_NEAREST)

bgr_paths = ['....datasets/conic/20x/256x256/image/crag_60-0021.png',
'....datasets/conic/20x/256x256/image/crag_9-0032.png',
'....datasets/pannuke/40x/256x256/image/pannuke_bladder_fold3_1583.png',
....datasets/pannuke/40x/256x256/image/pannuke_colon_fold1_2619.png',
'....datasets/pannuke/40x/256x256/image/pannuke_colon_fold1_2609.png'
]

for i in range(len(bgr_paths)):
    bgr_image = cv2.imread(bgr_paths[i])
    rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)
    
    if i < 2:
        rgb_image = upsample_rgb_image(rgb_image)

    #segment
    mask1 = cellsam_pipeline(rgb_image)
    mask2 = segment_cellular_image(process_image_array(rgb_image), device='cuda')[0]
    mask3 = segment_cellular_image(process_image_array(rgb_image, bluechannel=True), device='cuda')[0]

    if i < 2:
        mask2 = downsample_label_mask(mask2, (256, 256))  # from (512, 512) to (256, 256)
        mask3 = downsample_label_mask(mask3, (256, 256))  # from (512, 512) to (256, 256)

    mask2 = center_crop_mask(mask2)
    mask3 = center_crop_mask(mask3)

    plt.figure(dpi=300)
    plt.subplot(1,4,1)
    plt.imshow(rgb_image)
    plt.axis('off')
    plt.title('original image', fontsize=5)
    plt.subplot(1,4,2)
    plt.imshow(mask1)
    plt.axis('off')
    plt.title('cellsam_pipeline output', fontsize=5)
    plt.subplot(1,4,3)
    plt.imshow(mask2)
    plt.axis('off')
    plt.title('segment_cellular_image *\n process_image_array output', fontsize=5)
    plt.subplot(1,4,4)
    plt.imshow(mask3)
    plt.axis('off')
    plt.title('segment_cellular_image *\n process_image_array output\n - only blue channel', fontsize=5)

[vadori]

Another attempt. Everything is the same as before except that padding has been removed. I'd say the last option performs best. As far as I understand, the model hasn’t been exposed to colon tissue during training, so it’s not surprising that it struggles with crypt structures. To be noted that cellsam_pipeline has use_wsi = True by default. If use_wsi = False, option 1 and 2 are equivalent.

import numpy as np
import cv2
import matplotlib.pyplot as plt
from cellSAM import segment_cellular_image, cellsam_pipeline

def center_crop_mask(mask, crop_size=256):
    h, w = mask.shape
    top = (h - crop_size) // 2
    left = (w - crop_size) // 2
    return mask[top:top + crop_size, left:left + crop_size]

def process_image_array(image_array, bluechannel=False):
    # Ensure input is float32 for calculations
    img = image_array.astype(np.float32)

    # Get original size
    h, w, _ = img.shape
    desired_size = 512

    if not bluechannel:
        output = (img - img.min(axis=(0, 1))) / (img.max(axis=(0, 1)) - img.min(axis=(0, 1)) + 1e-8)
    
    else:

        # Compute average across channels before normalization
        avg = img.mean(axis=2)

        # Normalize average to [0, 1] using min-max scaling
        avg_normalized = (avg - avg.min()) / (avg.max() - avg.min() + 1e-8)  # small epsilon to avoid divide by zero

        # Create output RGB image with only blue channel set
        output = np.zeros((*avg_normalized.shape, 3), dtype=np.float32)
        output[..., 2] = avg_normalized

    # # Pad to 512x512 (centered padding)
    # h_new, w_new = output.shape[:2]
    # pad_h = max(0, desired_size - h_new)
    # pad_w = max(0, desired_size - w_new)
    # pad_top = pad_h // 2
    # pad_bottom = pad_h - pad_top
    # pad_left = pad_w // 2
    # pad_right = pad_w - pad_left

    # output_padded = np.pad(
    #     output,
    #     ((pad_top, pad_bottom), (pad_left, pad_right), (0, 0)),
    #     mode='constant',
    #     constant_values=0
    # )

    output_padded = output

    # # Convert to (3, 512, 512)
    # output_ch_first = np.transpose(output_padded, (2, 0, 1))

    # return output_ch_first
    
    return output_padded

def upsample_rgb_image(image):
    return cv2.resize(image, (512, 512), interpolation=cv2.INTER_LINEAR)

def downsample_label_mask(mask, target_size):
    return cv2.resize(mask, target_size, interpolation=cv2.INTER_NEAREST)

bgr_paths = ['D:/NucleiSegmentation/Projects/cytoark/datasets/conic/20x/256x256/image/crag_60-0021.png',
'D:/NucleiSegmentation/Projects/cytoark/datasets/conic/20x/256x256/image/crag_9-0032.png',
'D:/NucleiSegmentation/Projects/cytoark/datasets/pannuke/40x/256x256/image/pannuke_bladder_fold3_1583.png',
'D:/NucleiSegmentation/Projects/cytoark/datasets/pannuke/40x/256x256/image/pannuke_colon_fold1_2619.png',
'D:/NucleiSegmentation/Projects/cytoark/datasets/pannuke/40x/256x256/image/pannuke_colon_fold1_2609.png'
]

for i in range(len(bgr_paths)):
    bgr_image = cv2.imread(bgr_paths[i])
    rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)
    
    if i < 2:
        rgb_image = upsample_rgb_image(rgb_image)

    #segment
    mask1 = cellsam_pipeline(rgb_image)
    mask2 = segment_cellular_image(process_image_array(rgb_image), device='cuda')[0]
    mask3 = segment_cellular_image(process_image_array(rgb_image, bluechannel=True), device='cuda')[0]

    if i < 2:
        mask2 = downsample_label_mask(mask2, (256, 256))  # from (512, 512) to (256, 256)
        mask3 = downsample_label_mask(mask3, (256, 256))  # from (512, 512) to (256, 256)

    mask2 = center_crop_mask(mask2)
    mask3 = center_crop_mask(mask3)

    plt.figure(dpi=300)
    plt.subplot(1,4,1)
    plt.imshow(rgb_image)
    plt.axis('off')
    plt.title('original image', fontsize=5)
    plt.subplot(1,4,2)
    plt.imshow(mask1)
    plt.axis('off')
    plt.title('cellsam_pipeline output', fontsize=5)
    plt.subplot(1,4,3)
    plt.imshow(mask2)
    plt.axis('off')
    plt.title('segment_cellular_image *\n process_image_array output', fontsize=5)
    plt.subplot(1,4,4)
    plt.imshow(mask3)
    plt.axis('off')
    plt.title('segment_cellular_image *\n process_image_array output\n - only blue channel', fontsize=5)

[vadori]

For those interested, we recently published a paper where we used CellSAM zero-shot performance as a baseline for cell instance segmentation of digital pathology and brain cytoarchitecture data. The following is an extract from the Supplementary Materials. The full paper can be accessed here https://www.sciencedirect.com/science/article/pii/S0010482525013708.

‘We tested the cellSAM foundation model for cell segmentation using the segment_cellular_image function provided by the cellSAM library across the four datasets considered in our study. Differently from the other models tested in our work, cellSAM was applied to determine its zero-shot performance as a baseline. Since we noticed that performance can vary depending on image magnification and staining, and to ensure we were not underestimating the segmentation performance of cellSAM, we tested different pre-processing configurations. We selected six configurations that appeared to return the best results when observing the segmentation on a sample set for each dataset.
Each configuration differed in three factors. First, the patch padding size was set to either 256 × 256 pixels or 512 × 512 pixels prior to inference. Second, for the 512 × 512 setting, we optionally performed an enlargement step in which the original image was upsampled by a factor of 2, thereby eliminating the need for padding. Third, we considered two blue-channel synthesis options: either the original RGB image was used (only_blue = 0) or the blue channel was replaced with the mean of all channels while the red and green channels were set to zero (only_blue = 1), following the pre-processing approach described in the CellPose-SAM paper, where cellSAM was recently tested. In all cases, test images were normalized such that the minimum intensity was set to 0 and the maximum to 1.
The six tested configurations were labelled as follows:

1. pad256_blue1 – 256-pixel padding, no enlargement, only_blue = 1
2. pad512_blue1 – 512-pixel padding, no enlargement, only_blue = 1
3. pad512_enl_blue1 – 512-pixel padding, with enlargement, only_blue = 1
4. pad256_blue0 – 256-pixel padding, no enlargement, only_blue = 0 (full RGB).
5. pad512_blue0 – 512-pixel padding, no enlargement, only_blue = 0 (full RGB).
6. pad512_enl_blue0 – 512-pixel padding, with enlargement, only_blue = 0 (full RGB).
   The results for each configuration are reported in the table below, and the best-performing configuration (pad256_blue1) for each dataset is highlighted in bold and reported in Table 5 in the main manuscript. Qualitative results for randomly selected cases are shown in the figures below.’

[rossbar]

Thanks for sharing @vadori ! In my non-expert opinion, your experiment setup and tested parameters seem reasonable. I leave it to one of the experts whether there are any finer details that warrant comment - LUK @ulisrael @damaggu @rdilip @m274d .

The only other comment I have is related to the versioning of the pre-trained model weights --- the pre-trained model weights available to this point are from the pre-release on bioarxiv. There are now a set of "official" pre-trained model weights that are properly versioned and slated for release with the publication of the paper. These model weights are currently available, though not yet merged into the development branch of the software - see #88 for details.

[dsatyam09]

hey just to confirm is the input size which the cell sam model takes is of 512*512. bze I have an H&E stained image of (9510, 9993, 3) so I was a little confused if reducing the dimension so much would lead to a really poor segmentation or should I rather implement to tile larger images (e.g. 9510×9993) into blocks (512, 1024 or some power-of-two) and run inference patch-by-patch, then stitch results. I would really appreciate your advice on this @rossbar.

[rossbar]

Thanks for the follow-up @vadori and the ping @dsatyam09 . There are two separate issues here:

The first is @vadori 's original question: what format does cellsam expect for H&E (or similar stained) images. The answer to this question is: gray scale - in other words, multi-channel stained images should be collapsed to grayscale for best results. There is a gallery example which demonstrates the expected format for histology images. Note that the documentation recommends otherwise, so there is a definite issue with the documentation that needs to be fixed!

The second question regards image scale - @dsatyam09 your intuition is correct, you should not be rescaling images to fit a 512x512 array, but rather use use_wsi=True to activate under-the-hood tiling for large images.

[DarioS]

multi-channel stained images should be collapsed to grayscale for best results

Neither the journal article nor the software user guide specify how this was done for model training. There may be multiple marker choices for a cell compartment and knowing exactly which one training used would probably help segmentation.