Apex
Digital Labs
Surgical AI Labelling Framework
A comprehensive framework for data annotation in surgical robotics, designed to optimize the development of ML models for surgical robots.
Endeavor Roadmap
Phase 1: Epistemic Scoring (2 years)
Recommender System for Medical Image Annotation
A recommender system that assesses the difficulty of medical images by assigning them an epistemic score, helping data annotators prioritize which images require more attention.
Phase 2: Spatial Prompts (3 years)
Auto-Segmentation with User Prompts
Developing auto-segmentation algorithms that utilize various types of user prompts to guide the model in accurately segmenting images by highlighting key areas of interest.
Phase 3: Multi-Modal Prompts (5 years)
Advanced Multi-Modal Segmentation System
Developing an advanced system that integrates Large Language Models (LLMs) with visual data to enhance medical image segmentation. This multi-modal approach combines textual descriptions with visual cues, using natural language processing and pre-trained models like BERT or GPT to create a more comprehensive and accurate segmentation system.
Interactive Segmentation Demo
This demo simulates how our AI framework can assist in medical image annotation.
1. Common Approach
The traditional approach to medical image segmentation requires manual annotation by experts, which is time-consuming and prone to human error.
Instructions
- Your objective is to manually segment the heart in the image
- Use the brush tool to carefully draw the outline of the heart
- Adjust the brush size to match the precision needed
- Use the eraser to correct any mistakes in your annotation
- Use the clear button to start over if needed
Tools
Brush Size
Draw on the image to segment the selected body part
About This Demo
As you just saw, manual annotation of medical images is a time-consuming and error-prone process. Each image requires careful attention to detail, and even experienced medical professionals can make mistakes or be inconsistent in their annotations.
Training effective AI models for medical image analysis typically requires thousands of accurately labeled images. Each labeling job is expensive, with costs often reaching hundreds of dollars per hour for specialized medical expertise. The time and money spent on annotation is frequently the bottleneck in developing new medical AI applications.
For example, a single complex medical image can take over a minute for a trained expert to fully segment. With a dataset of 50,000 images, this would require hundreds of hours of expert time. At high per hour rates for specialized medical expertise, the annotation cost alone would be substantial.
2. Epistemic Score
Our epistemic uncertainty scoring system helps identify regions where the model is less confident, guiding annotation efforts to areas that need human expertise.
Instructions
- Browse through the images using the navigation buttons
- Notice how predictions are noisy accross diffrent slices and features
- Try to identify the slices and features that are problematic for the model
Can you identify which slices are problematic for the model for each feature?
Move the slider to see across the MRI slices
Loading images...
No images available for comparison.
Drag the slider to compare the original CT slice with its segmentation result.
Review where model is the least confident
Select one organ and the heat map below the canvas will show where there are more issues across the scan. Darker colors indicate more problems in that region.
About This Demo
After the initial training of a medical image segmentation model, researchers must identify which parts of the data distribution the model performs worst on to collect new data for improvement.
This process is challenging because data collection often happens in parallel often without direct supervision from the researcher, generating huge volumes of data where it's not always clear which specific images or regions are problematic for the model. Since models as the one shown can have more than 100 classes, without a systematic approach, researchers might waste resources annotating images that don't significantly improve model performance.
Our epistemic uncertainty scoring system provides a straightforward and systematic approach to identify only the images that matter most for future annotation. By quantifying the model's uncertainty in its predictions, we can:
- Prioritize images where the model is least confident.
- Focus annotation efforts on the most challenging cases.
- Reduce the number of images needed for model improvement.
- Create a more efficient annotation pipeline.
This demo simulates how our epistemic scoring system would help researchers identify the most challenging images from a dataset, allowing them to focus their limited annotation resources on the images that will have the greatest impact on model performance.
3. Assisted Segmentation
Our assisted segmentation system combines human expertise with AI to create a collaborative annotation experience.
Instructions
- Run the automatic segmentation
- Fix the brain region using the brush tool
- Adjust brush size as needed
- Use the eraser to correct mistakes
Tools
Brush Size
Draw on the image to segment the selected body part
About This Demo
This interactive demo simulates how our Surgical AI Labelling Framework can assist in medical image annotation. In a real implementation, our framework would use advanced machine learning models to:
- Automatically detect anatomical structures to avoid manual annotation on features with low epistemic uncertainty.
- Suggest optimal segmentation boundaries.
- Learn from user corrections to improve over time.
This demo is a simplified version to showcase the concept. Our actual framework will incorporate more sophisticated algorithms and models as outlined in our project roadmap.
4. Advanced Assisted Segmentation
Our advanced assisted segmentation system uses cutting-edge AI to provide intelligent suggestions and real-time feedback during the annotation process.
Instructions
- Run the automatic segmentation
- Fix the brain region using the brush tool
- Adjust brush size as needed
- Use the eraser to correct mistakes
- Ask the AI assistant questions about the image
AI Assistant
About This Demo
This interactive demo simulates how advanced assisted labelling can expand the capabilities of the tool:
- Iteracting with the segmentation model with natural language
- The user can provide descriptions of the anatomical structures they want to label
This demo is a simplified version to showcase the concept. Our actual framework will also leverage information from medical literature for better segmentation performance.