News

Sept 8th, 2017
Update for Challenge 15 available, but will not count in evaluation.

Sept 4th, 2017
Updated mailling list and submission information.

Aug 23rd, 2017
The preliminary results have been sent out to participants, and are now available.

July 09th, 2017
We fixed the intensities in the TSV archive for challenges 046-243.

June 22nd, 2017
We added the Category 4 on a subset of the data files.

May 22nd, 2017
We have improved challenges 29, 42, 71, 89, 105, 106 and 144.

April 26th, 2017
The rules and challenges of CASMI 2017 are public now !

Jan 20th, 2017
Organisation of CASMI 2017 is underway, stay tuned!


Submission format

Submission to CASMI requires an abstract file plus results files for each challenge.

The abstract should be provided as a plain text file containing details on the method and, where appropriate, metadata used. The file should be named in the form: <participant>-<category>.txt. A template is available here and examples from 2014 are available here. Participant names must contain only letters and numbers, no dashes, spaces nor special characters.

For each challenge, the results should be submitted as a plain text, tab separated file with at least two columns. The first column should contain the representation of the structure as the (standard) InChI or the SMILES code. The second column should contain the score. The score should be non-negative with a higher score representing a better candidate for the evaluation script to work properly. Any further information can be contained in additional columns (but will be ignored for the automatic evaluation). Do not include headers or row names. Report the small molecules themselves, not the adduct species like [M+H]+. Molecules that you consider to be charged in the natural state should be reported as such. The file name must be in the format <participant>-<category>-<challenge>.txt, see here for an example. Only the two dashes shown are allowed in the file names. If you enter a category with different submissions, use different participant names, e.g. MetFragWithCitations and MetFragWithRT. Each submission requires its own abstract.

Updated instructions 20170904:

Please provide your submissions in a ZIP archive using one of the file hosting services like http://dropexpire.com/ , or sharing a link via Dropbox or your institutions Owncloud or similar infrastructure, and sent the link to the ZIP to casmi-team2017 (see email link at the bottom) before the deadline (see deadlines). We do confirm each submission with a quick eMail reply, so if you do not get a confirmation within 24h, please don't hesitate to ask back.

Evaluation and declaration of winners

To be considered, your submissions will have to pass the automatic evaluation without manual intervention. The evaluation pipeline filters and remove all stereo-isomers, i.e. we will calculate the InChIkey for each submission, and for all compounds with identical InChIkey first block, only retain the one with the highest score for the following ranking.

For each unknown, the absolute rank of the correct solution (ordered by score) will be determined and, if the score is equal among several candidates, the average rank over all these candidates will be used. This year, we will use the "Formula 1 Score", based on the method used in Formula 1 racing, which is the sum of the Top 1 to 10 ranks of the correct candidates weighted by the points 25 (Top1), 18, 15, 12, 10, 8, 6, 4, 2 and 1 for correct solution ranked 10th. See also the CASMI 2016 paper.

The role of the organisers will be to resolve any issues, such as (but not limited to): 1) Re-categorising an entry (submitter will be notified as soon as possible after submission) 2) Including a valid tautomer of the correct solution, if this was not considered initially in the automatic evaluation. The tautomer InChIKey will be added and all entries re-evaluated. 3) Fixing any bugs in the evaluation script - once fixed, all entries will be re-evaluated. Please report any issues to the CASMI team, see email link below. 4) Reporting format issues to participants for them to fix and re-submit immediately.