4 x 4 digit dataset | Australian credit dataset | Chromosome | Credit management | German credit | Heart | Image segmentation | Karhunen-Loeve digits | Letters (16 moments) | Satellite image | Vehicle dataset |

Machine Learning, Neural and Statistical Classification, D. Michie, D.J. Spiegelhalter, C.C. Taylor (eds), Stalog project - whole book!
More results for medical and other data.

A note of caution: comparison of different classifiers is not an easy task. Before you get into ranking of methods using the numbers presented in tables below please note the following facts.

Many results we have collected give only a single number (even results from the StatLog project!), without standard deviation. Since most classifiers may give results that differ by several percent on slightly different data partitions single numbers do not mean much.

Leave-one-out tests have been criticized as a basis for accuracy evaluation, the conclusion is that crossvalidation is safer, cf:
Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proc. of the 14th Int. Joint Conference on Artificial Intelligence, Morgan Kaufmann, pp. 1137-1143.

Crossvalidation tests (CV) are also not ideal. Theoretically about 2/3 of results should be within a single standard deviation from the average, and 95% of results should be within two standard deviations, so in a 10-fold crossvalidation you should see very rarely reuslts that are beter or worse than 2xSTDs. Running CV several times may also give you different answers. Search for the best estimator continues. Cf:
Dietterich, T. (1998) Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10 (7), 1895-1924;
Nadeau C, Bengio Y. (1999) Inference for the Generalization Error. Tech. rep. 99s-25, CIRANO, J. Machine Learning (Kluver, in print).

Even the best accuracy and variance estimation is not sufficient, since performance cannot be characterized by a single number. It should be much better to provide full Receiver Operator Curves (ROC). Combining ROC with variance estimation would be ideal. Unfortunately this still remains to be done. All we can do now is to collect some numbers in tables.

Credit management

Statlog version, 2 classes, 7 attributes, no. of training cases=15000, no. of test cases=5000;
Unfortunately this data is not public; anyone knows where to find it?

Australian credit dataset

Statlog dataset, 2 classes, 14 attributes, 690 observations, class distribution 55.5%, 44.5%.
37 missing values, A1: 12, A2: 12, A4: 6, A5: 6, A6: 9, A7: 9, A14: 13
10-fold cross-validation.

4 x 4 digit dataset

Statlog dataset, 10 classes, 16 attributes (train,test)=(9000,9000) observations)

Karhunen-Loeve digits

Statlog dataset, 10 classes, 40 attributes, (train,test) = (9000,9000) observations
Unfortunately this data is not public; anyone knows where to find it?

Vehicle dataset

Statlog dataset,  4 classes, 18 attributes, 846 observations, 9-fold cross-validation
 
 

Letters

Statlog dataset, 26 classes, 16 attributes, (train,test) = (15000,5000) observations
 
 

Chromosome dataset

Statlog dataset, 24 classes, 16 attributes, (train,test) = (20000,20000) observations; unfortunately the dataset is not public! Anyone knows where to find it?

Satellite image (SatImage)

Statlog dataset, 6 classes, 36 attributes, (train,test)=(4435,2000) observations
 
 

Image segmentation

Statlog dataset, 7 classes, 11 attributes, 2310 observations, 10-fold cross-validation
 
 

Datasets with costs

Heart disease

Statlog dataset, 2 classes, 13 attributes, 270 observations, 9-fold cross-validation.
Algorithms in italics have not incorporated costs.

The below table illustrates misclassification costs for the heart disease dataset.
The columns represent the predicted class and the rows the true class.

German credit

Statlog dataset, 2 classes, 24 attributes, 1000 observations, 10-fold cross-validation
Algorithms in italics have not incorporated costs.

The table below illustrates the cost matrix for the German credit dataset. The columns are the predicted class and the rows the true class.

Links to other Duch-Lab projects, many talks and to other papers on various subjects.

Working log (local accsess only), maintained by Wlodzislaw Duch.