News

Oct 31st, 2017
The results are now available.

Oct 30th, 2017
The solutions are now available.

Sept 8th, 2017
Update for Challenge 15 available, but will not count in evaluation.

Sept 4th, 2017
Updated mailling list and submission information.

Aug 23rd, 2017
The preliminary results have been sent out to participants, and are now available.

July 09th, 2017
We fixed the intensities in the TSV archive for challenges 046-243.

June 22nd, 2017
We added the Category 4 on a subset of the data files.

May 22nd, 2017
We have improved challenges 29, 42, 71, 89, 105, 106 and 144.

April 26th, 2017
The rules and challenges of CASMI 2017 are public now !

Jan 20th, 2017
Organisation of CASMI 2017 is underway, stay tuned!


Results in Category 3

Summary of participant performance

F1 score Mean rank Median rank Top Top3 Top10 Misses TopPos TopNeg Mean RRP Median RRP N
tkind1 3707 4.33 2.0 91 148 193 34 52 39 0.644 0.750 238
tkind2 1704 15.20 6.0 29 63 107 72 15 14 0.755 0.846 240
TsugawaYamamoto 1559 19.99 7.0 28 56 104 64 16 12 0.841 0.900 243
This summary is also available as CSV download.

Table legend:

F1 score
The Formula 1 score awards points similar to the scheme in F1 racing for each challenge based on the rank of the correct solution. In the participant table, these are summed over all challenges. Please note that the F1 score is thus not neccessarily comparable across categories.
Mean/Median rank
Mean and median rank of the correct solution. For tied ranks with other candidates, the average rank of the ties is used.
Top, Top3, Top10
Number of challenges where the correct solution is ranked first, among the Top 3 and Top 10
Misses
Number of challenges where the correct solution is missing.
TopPos, TopNeg
Top1 ranked solutions in positive or negative ionization mode.
Mean/Median RRP
The relative ranking position, which is also incorporating the length of candidate list.
N
Number of submissions that have passed the evaluation scripts.

Summary of Rank by Challenge

For each challenge, the lowest rank among participants is highlighted in bold. If the submission did not contain the correct candidate this is denoted as "-". If someone did not participate in a challenge, the table cell is empty. The tables are sortable if you click into the column header.

Category3:

tkind1 tkind2 TsugawaYamamoto
challenge-001 11.0 11.0 28.0
challenge-002 6.0 6.0 5.0
challenge-003 1.0 1.0 1.0
challenge-004 60.5 60.5 14.0
challenge-005 20.0 18.0 32.0
challenge-006 5.0 23.0 3.5
challenge-007 1.0 1.0 2.0
challenge-008 10.0 10.0 2.0
challenge-009 72.5 78.5 4.5
challenge-010 - - -
challenge-011 - - -
challenge-012 15.0 16.0 80.0
challenge-013 - 94.5 11.5
challenge-014 22.0 20.0 24.0
challenge-015 43.5 42.5 31.0
challenge-016 4.5 3.5 30.0
challenge-017 2.0 2.0 5.0
challenge-018 3.5 3.5 3.0
challenge-019 - - 2.0
challenge-020 6.0 6.0 9.0
challenge-021 - - -
challenge-022 3.0
challenge-023 - - -
challenge-024 3.0 4.0 15.5
challenge-025 - - -
challenge-026 81.0 83.0 8.0
challenge-027 - - -
challenge-028 - - -
challenge-029 5.0 5.0 2.0
challenge-030 - - 21.0
challenge-031 2.0 2.0 1.5
challenge-032 10.0 7.0 6.0
challenge-033 - - -
challenge-034 68.0 73.0 67.0
challenge-035 8.5 7.5 101.0
challenge-036 4.0 4.0 11.0
challenge-037 13.0 13.0 2.0
challenge-038 23.0 24.0 6.0
challenge-039 3.0 2.0 16.0
challenge-040 2.5 2.5 1.5
challenge-041 - - -
challenge-042 -
challenge-043 - - -
challenge-044 - - -
challenge-045 - - -
challenge-046 1.0 3.5 6.0
challenge-047 - 5.5 7.0
challenge-048 3.0 16.0 48.0
challenge-049 5.0 - 88.0
challenge-050 - 10.5 3.5
challenge-051 1.0 - -
challenge-052 1.0 - -
challenge-053 1.0 1.0 1.0
challenge-054 2.0 2.0 25.0
challenge-055 1.0 11.0
challenge-056 7.0 - -
challenge-057 1.0 1.0 1.0
challenge-058 5.0 - 11.0
challenge-059 1.0 1.5 5.0
challenge-060 4.0 55.0 61.5
challenge-061 4.0 16.5 19.0
challenge-062 2.0 9.0 9.0
challenge-063 2.0 - -
challenge-064 1.0 2.0 3.0
challenge-065 1.0 1.0 5.0
challenge-066 4.0 - -
challenge-067 2.0 40.0 44.5
challenge-068 2.0 2.0 2.0
challenge-069 1.0 1.5 1.0
challenge-070 3.0 - -
challenge-071 1.0 2.0 25.0
challenge-072 3.0 3.0 12.0
challenge-073 1.0 1.0 1.0
challenge-074 - 1.0 1.0
challenge-075 3.5 - -
challenge-076 4.0 10.0 37.5
challenge-077 7.0 - 88.5
challenge-078 4.0 33.5 54.0
challenge-079 2.0 32.5 23.0
challenge-080 1.0 15.5 5.0
challenge-081 1.0 - -
challenge-082 2.5 30.0 59.0
challenge-083 3.5 6.5 4.0
challenge-084 1.0 20.0 48.0
challenge-085 1.0 - -
challenge-086 1.0 - -
challenge-087 3.0 32.5 22.0
challenge-088 3.5 7.0 5.0
challenge-089 1.0 1.0 11.0
challenge-090 1.0 1.5 1.0
challenge-091 - - -
challenge-092 1.5 - -
challenge-093 1.0 2.0 13.0
challenge-094 4.5 - -
challenge-095 3.0 - -
challenge-096 - - -
challenge-097 1.0 - -
challenge-098 1.0 1.0 1.0
challenge-099 1.0 2.5 9.0
challenge-100 3.0 - -
challenge-101 1.0 17.5 10.0
challenge-102 2.5 13.0 16.5
challenge-103 1.5 5.0 1.5
challenge-104 5.0 - -
challenge-105 2.0 28.5 47.5
challenge-106 1.0 3.5 6.5
challenge-107 1.5 - -
challenge-108 1.0 7.0 33.0
challenge-109 1.0 20.5 15.5
challenge-110 1.0 1.0 1.0
challenge-111 - - -
challenge-112 1.0 11.0 9.0
challenge-113 3.0 5.0 2.0
challenge-114 2.0 9.0 4.0
challenge-115 1.0 1.0 -
challenge-116 1.0 1.0 1.0
challenge-117 1.0 1.0 7.0
challenge-118 - - -
challenge-119 1.0 1.0 10.0
challenge-120 1.5 - -
challenge-121 1.0 1.5 3.0
challenge-122 1.0 2.0 16.0
challenge-123 5.0 - 96.0
challenge-124 1.0 8.5 2.0
challenge-125 2.0 23.0 23.0
challenge-126 1.0 1.5 1.0
challenge-127 1.0 1.0 1.0
challenge-128 1.0 - -
challenge-129 1.0 1.0
challenge-130 - -
challenge-131 - 9.0 8.0
challenge-132 1.0 2.5 2.5
challenge-133 - 6.0 4.0
challenge-134 3.0 78.0 56.0
challenge-135 6.0 - 221.0
challenge-136 1.0 4.0 2.5
challenge-137 1.0 4.0 4.0
challenge-138 3.0 42.0 147.0
challenge-139 1.0 1.5 3.0
challenge-140 1.0 3.0 48.5
challenge-141 1.0 - -
challenge-142 2.0 9.0 9.0
challenge-143 5.0 89.0 75.5
challenge-144 - - -
challenge-145 2.0 10.0 29.0
challenge-146 - 12.5 4.0
challenge-147 2.0 23.0 17.5
challenge-148 1.0 1.0 7.5
challenge-149 1.0 - -
challenge-150 1.0 2.0 2.0
challenge-151 1.0 2.0 12.0
challenge-152 1.0 3.0 3.0
challenge-153 - - -
challenge-154 1.0 1.0 1.0
challenge-155 1.0 5.0 3.0
challenge-156 6.5 56.5 77.0
challenge-157 6.0 84.5 156.0
challenge-158 3.0 14.0 16.0
challenge-159 7.0 41.0 27.0
challenge-160 1.0 2.0 2.0
challenge-161 5.0 36.0 71.0
challenge-162 2.0 7.0 66.0
challenge-163 1.0 1.0 3.0
challenge-164 4.0 47.5 36.0
challenge-165 1.0 1.0 2.0
challenge-166 2.0 - -
challenge-167 1.0 3.5 13.0
challenge-168 4.0 - 96.0
challenge-169 1.0 1.5 1.0
challenge-170 1.0 54.5 2.5
challenge-171 3.0 - -
challenge-172 - - -
challenge-173 1.0 - -
challenge-174 1.0 7.0 9.0
challenge-175 4.0 7.0 1.0
challenge-176 4.0 - 114.0
challenge-177 1.0 2.0 1.0
challenge-178 2.0 41.0 58.0
challenge-179 - - 2.5
challenge-180 - 2.0 1.0
challenge-181 1.0 4.0 33.0
challenge-182 6.5 65.0 52.0
challenge-183 1.0 1.0 2.0
challenge-184 2.0 28.0 19.0
challenge-185 2.0 9.0 5.0
challenge-186 4.0 52.0 42.0
challenge-187 1.0 - -
challenge-188 4.0 43.5 17.5
challenge-189 3.0 - -
challenge-190 9.0 - 35.0
challenge-191 2.0 12.0 7.0
challenge-192 2.0 - -
challenge-193 3.0 30.5 25.0
challenge-194 1.0 - -
challenge-195 3.5 8.0 4.0
challenge-196 1.0 1.0 15.5
challenge-197 - - -
challenge-198 2.0 19.0 13.0
challenge-199 2.0 6.0 1.0
challenge-200 2.0 53.0 6.0
challenge-201 5.0 - -
challenge-202 1.0 - -
challenge-203 - - -
challenge-204 2.0 - -
challenge-205 1.0 23.0 6.5
challenge-206 1.0 1.5 1.0
challenge-207 1.0 2.0 7.0
challenge-208 3.0 - -
challenge-209 1.0 23.0 6.0
challenge-210 1.5 2.0 9.0
challenge-211 4.0 56.5 15.0
challenge-212 1.5 5.0 14.0
challenge-213 1.0 - -
challenge-214 2.0 10.5 10.0
challenge-215 3.0 6.5 1.0
challenge-216 1.0 22.5 25.5
challenge-217 2.5 3.5 4.0
challenge-218 1.0 2.5 3.5
challenge-219 1.0 - -
challenge-220 4.5 4.5 3.5
challenge-221 1.0 6.5 5.0
challenge-222 1.0 5.5 6.0
challenge-223 1.0 1.0 1.0
challenge-224 2.0 - -
challenge-225 2.0 12.0 5.0
challenge-226 1.0 1.0 -
challenge-227 1.0 1.0 20.0
challenge-228 - - -
challenge-229 3.0 33.0 11.0
challenge-230 1.0 - -
challenge-231 1.0 1.0 1.0
challenge-232 1.0 1.0 2.0
challenge-233 1.0 3.0 1.0
challenge-234 4.0 65.0 70.0
challenge-235 4.0 30.5 14.5
challenge-236 1.0 2.0 5.5
challenge-237 1.0 1.5 1.0
challenge-238 1.0 1.0 1.0
challenge-239 2.0 6.0 1.0
challenge-240 1.0 - -
challenge-241 - -
challenge-242 1.0 1.0 1.0
challenge-243 - 11.5 5.5
This summary is also available as CSV download.


Participant information and abstracts

ParticipantID:        TsugawaYamamoto
Category:             category3
Authors:              Tsugawa, Hiroshi(1,2) and Yamamoto, Hiroyuki (3)
Affiliations:         (1) RIKEN Center for Sustainable Resource Science,
		      	  1-7-22 Suehiro-cho, Tsurumi-ku, Yokohama,
			  Kanagawa 230-0045, Japan
                      (2) RIKEN Center for Integrative Medical Sciences,
		      	  1-7-22 Suehiro-cho, Tsurumi-ku, Yokohama,
			  Kanagawa 230-0045, Japan
                      (3) Human Metabolome Technologies, Inc., 246-2 Mizukami,
		      	  Kakuganji, Tsuruoka, Yamagata 997-0052, Japan.

Automatic methods:    yes

Abstract

Our method is an integrated approach using MS-FINDER (internal
version, i.e. under development) and Sirius 3.5 (build 3)/CSI:FingerID
GUI.  Hiroyuki Yamamoto manually predicted the structure candidates
for positive ion mode data by Sirius 3.5(build 3) with the following
conditions.  Sirius/CSI:FingerID parameters:

ion/adduct type: Unknown positive
collision energy: 0 (no set)
Parent mass: manually typed by Pepmass value recorded in mgf
Search in: PubChem, search all

Then, Hiroshi Tsugawa used MS-FINDER internal version to rank
structure candidates for positive/negative ion data with the following
settings.

Seach option: Formula prediction and structure elucidation by in silico fragmenter
#Mass spectrum tab
Mass tolerance type: ppm
Mass tolerance (MS1): 10 ppm (for challenge 1-45) and 6 ppm (for challenge 46-243)
Mass tolerance (MS2): 20 ppm
Relative abundance cut off: 1%
Mass range max: 1000
Mass range min: 50
#Formula finder tab
LEWIS and SENIOR check: checked
Isotopic ratio tolerance: 20%
Elemental ratio check: 99.7% range
Element probability check: unchecked
Element selection: O, N, P, S
Maximum report number: 5
#Structure finder
Tree depth: 2
Maximum report number: 1000
#Data source
Local databases: ChEBI, PlantCyc, KNApSAcK, FooDB, PubChem (biomolecules), UNPD, NANPDB
MINEs: unchecked
PubChem: unchecked

For negative ion mode data, the results of MS-FINDER are submitted.
(Because CSI:FingerID does not provide the function for negative ion
mode yet as long as Hiroyuki Yamamoto examined.)

For positive ion mode data, the integration of Hiroyuki Yamamoto
(i.e. CSI:FingerID) and Hiroshi Tsugawa (i.e. MS-FINDER) was performed
by a 'voting' method.  The structure candidates were ranked by both
programs, and then, the ranking orders from both programs were summed.
Then, the structure candidates were ordered by the summed value
(i.e. smaller value will be assigned as a better candidate).

ParticipantID:        tkind1
Category:	      category3
Authors:              Tobias Kind
Affiliations:         UC Davis Genome Center, USA
Automatic pipeline:   yes
Spectral libraries:   no

Abstract
We imported all 243 MS/MS challenge files into MS-Finder v2.14 (June 6,2017).
Other data import/export was handled in EXCEL VBA. Minor prior adjustments to adduct
types and number of peaks were handled in Microsoft EXCEL.

We applied the following settings in MS-Finder:

1) Methods: Formula prediction and structure elucidation with
   in-silico fragmenter: yes; no spectral database search; Precursor
   option on
2) Mass Spectrum: Mass tolerance (MS1) 6 ppm and Mass tolerance (MS2)
   20 ppm relative abundance cutoff 1%; Mass range 50-1000 Da
3) Formula Finder: Lewis and Senior check on; isotope ratio tolerance
   20%; element ration 99.7%; element probability off; elements
   CHNSOPCl; minimum reports 100
4) Structure Finder: Tree depth 2; use the fragment library for EI
   (on); use the fragmentation library for low energy CID (off);
   maximum report number 100
5) Data source: for cases 0-45 UNPD (plant metabolite library) and
   plant metabolite target library and for cases 46 to 243 only the
   plant metabolite target library. All other libraries were off.
   Compound annotation was performed in batch mode. Processing time
   was 5:17 minutes (Dual Xeon 3.1 Ghz)

The top 100 best ranked compounds were exported as SMILES with their
related scores.  No additional MS/MS library search was performed. No
additional post-curation was performed.  MS-Finder can be obtained
from
http://prime.psc.riken.jp/Metabolomics_Software/MS-FINDER/index.html
ParticipantID:        tkind2
Category:	      category3
Authors:              Tobias Kind
Affiliations:         UC Davis Genome Center, USA
Automatic pipeline:   yes
Spectral libraries:   no

Abstract:
Minor prior adjustments of precursor types, number of peaks and other 
data were performed in Microsoft Excel. Data import/export for
MS-Finder was handled Excel VBA. All 243 MS/MS spectral 
files were imported into MS-Finder v2.14 (June 6,2017).
(http://prime.psc.riken.jp/Metabolomics_Software/MS-FINDER/index.html)

We applied the following settings in MS-Finder:

1) Methods: Formula prediction and structure elucidation with
   in-silico fragmenter: yes; no spectral database search; Precursor
   option on
2) Mass Spectrum: Mass tolerance (MS1) 6 ppm and Mass tolerance (MS2)
   20 ppm relative abundance cutoff 1%; Mass range 50-1000 Da
3) Formula Finder: Lewis and Senior check on; isotope ratio tolerance
   20%; element ration 99.7%; element probability off; elements
   CHNSOPCl; minimum reports 10
4) Structure Finder: Tree depth 2; use the fragment library for EI
   (on); use the fragmentation library for low energy CID (off);
   maximum report number 100
5) Data source: As target compound database we searched the PlantCyc
   database, the UNPD plant metabolite library and the KNApSAcK plant
   DB.  Compound annotation was performed in batch mode.  Processing
   time was two days on a 16-core Dual-Xeon (3.1 Ghz).

The top 100 best ranked compounds were exported as SMILES with their
related scores.  No additional MS/MS library search or manual curation
was performed.

Details per Challenge and Participant. See legend at bottom for more details

The details table is also available as HTML and as CSV download.