News
- Tuesday, April 18, 2017:
- We release the trained network, affinities, and re-aligned training and testing volumes of submission MALA v2. The archive is available here (33GB).
- Monday, October 17, 2016:
- #CREMIChallenge satellite event of MICCAI 2016 starts at 2pm in Omikron II.
- Monday, September 12, 2016:
- The schedule for our on-site event is online.
- Friday, September 8, 2016:
- The leaderboard is online.
- Wednesday, September 7, 2016:
- We are currently evaluating the submissions. The results will be released here soon, and all participants will be notified by mail.
- Thursday, June 16, 2016:
- #CREMIChallenge evaluation metrics description and code released.
- Wednesday, June 1, 2016:
- The test data is released and the submission system is online.
- Thursday, May 26, 2016:
- Janelia is doing infrastructure maintenance during the Memorial day weekend. We expect cremi.org to go up and down from Friday, May 27, 7PM through Tuesday, May 31, 6AM.
- Wednesday, May 4, 2016:
- #CREMIChallenge python scripts to read/write CREMI datasets are on GitHub.
- Sunday, May 1, 2016:
- First version of training data released.
Schedule
The on-site event at MICCAI 2016 will take place Monday, October 17th, 14:00-18:00, in Omikron II, with the following program:
- 14:00-14:30
- Introduction, Jan Funke
- 14:30-15:00
- Keynote I, Juan Nunez-Iglesias (VLSCI, University of Melbourne, Australia)
- 15:00-15:30
- Presentation by group IAL, category neuron segmentation
- 15:30-16:00
- Keynote II, Anna Kreshuk (HCI Heidelberg)
- 16:00-16:30
- Coffee break
- 16:30-17:00
- Presentation by group DIVE, category neuron segmentation and synaptic cleft detection
- 17:00-17:30
- Presentation by group SDG, category synaptic cleft detection
- 17:30-18:00
- Keynote III, Srinivas Turaga (Janelia Research Campus)
About
The goal of this challenge is to evaluate algorithms for automatic reconstruction of neurons and neuronal connectivity from serial section electron microscopy data. The comparison is performed not only by evaluating the quality of neuron segmentations, but also by assessing the accuracy of detecting synapses and identifying synaptic partners. The challenge is carried out on three large and diverse datasets from adult Drosophila melanogaster brain tissue, comprising neuron segmentation ground truth and annotations for synaptic connections. A successful solution would demonstrate its efficiency and generalizability, and carry great potential to reduce the time spent on manual reconstruction of neural circuits in electron microscopy volumes.
Datasets and Challenge Categories
We provide three datasets, each consisting of two (5 μm)3 volumes (training and testing, each 1250 px × 1250 px × 125 px) of serial section EM of the adult fly brain. Each volume has neuron and synapse labelings and annotations for pre- and post-synaptic partners.
Submissions can be made in the following categories:
- Segmentation
- Assessment of neuron labelings in terms of pixel-based measures (Rand index and variation of information) and topological correctness (Tolerant Edit Distance). Winners will be determined using a weighted average of the scores.
- Synapse Detection
- Detection of synapses. Winners are determined using the F-measure of false positive and false negative detections.
- Connectivity
- Identification of synaptic partners. Winners are determined using the F-measure of false positive and false negative detections of pre- and post-synaptic pairs.
Details about the evaluation procedures will be released soon.
Important Dates
- May 1
- Opening of training phase ✔
- June 1
- Opening of testing phase ✔
- August 31
- Submission deadline for test results (11:59 PM PDT) ✔
- September 7
- Announcement of results and submission deadline for extended abstracts ✔
- October 17
- Oral presentations of participants at MICCAI 2016 in Athens
After the on-site event at MICCAI 2016, the results of the challenge will be summarized in a review paper. The challenge will stay open for submission.
Organizers
- Jan Funke (Institut de Robòtica i Informàtica Industrial, UPC/CSIC, Barcelona, Spain; HHMI Janelia, Ashburn, VA, USA),
- Stephan Saalfeld (HHMI Janelia, Ashburn, VA, USA),
- Davi Bock (HHMI Janelia, Ashburn, VA, USA),
- Srini Turaga (HHMI Janelia, Ashburn, VA, USA), and
- Eric Perlman (HHMI Janelia, Ashburn, VA, USA)
Please contact the organizers at challenge@cremi.org.