14.1.5.1 Error Estimation, Classification Accuracy

Pearl, J.[Judea],
Capacity and Error Estimates for Boolean Classifiers with Limited Complexity,
PAMI(1), No. 4, October 1979, 350-356. BibRef 7910

Pearl, J.[Judea],
An application of rate-distortion theory to pattern recognition and classification,
PR(8), No. 1, January 1976, pp. 11-22.
WWW Version. 0309 BibRef

McLachlan, G.J.,
A note on the choice of a weighting function to give an efficient method for estimating the probability of misclassification,
PR(9), No. 3, October 1977, pp. 147-149.
WWW Version. 0309 BibRef

Bock, H.H.,
On some significance tests in cluster analysis,
Classification(2), 1985, pp. 77-108.
WWW Version. BibRef 8500

Lawoko, C.R.O., McLachlan, G.J.,
Asymptotic error rates of the W and Z statistics when the training observations are dependent,
PR(19), No. 6, 1986, pp. 467-471.
WWW Version. 0309 BibRef

Ganesalingam, S., McLachlan, G.J.,
Error rate estimation on the basis of posterior probabilities,
PR(12), No. 6, 1980, pp. 405-413.
WWW Version. 0309 BibRef

Chittineni, C.B.,
On the estimation of probability of error,
PR(9), No. 4, 1977, pp. 191-196.
WWW Version. 0309 BibRef

Chittineni, C.B.,
Estimation of probabilities of label imperfections and correction of mislabels,
PR(13), No. 3, 1981, pp. 257-268.
WWW Version. 0309 BibRef

van Otterloo, P.J., Young, I.T.,
A distribution-free geometric upper bound for the probability of error of a minimum distance classifier,
PR(10), No. 4, 1978, pp. 281-286.
WWW Version. 0309 BibRef

Glick, N.[Ned],
Additive estimators for probabilities of correct classification,
PR(10), No. 3, 1978, pp. 211-222.
WWW Version. 0309 BibRef

Engvall, J.L.[John L.],
A least upper bound for the average classification accuracy of multiple observers,
PR(12), No. 6, 1980, pp. 415-419.
WWW Version. 0309 BibRef

Kittler, J.V., Devijver, P.A.,
An efficient estimator of pattern recognition system error probability,
PR(13), No. 3, 1981, pp. 245-249.
WWW Version. 0309 BibRef

Lahart, M.J.,
Estimation of Error Rates in Classification of Distorted Imagery,
PAMI(6), No. 4, July 1984, pp. 535-542. BibRef 8407

Fukunaga, K., and Flick, T.E.,
Classification Error for a Very Large Number of Classes,
PAMI(6), No. 6, November 1984, pp. 779-788. See also Optimal Global Nearest Neighbor Metric, An. BibRef 8411

Fukunaga, K., and Hayes, R.R.,
Estimation of Classifier Performance,
PAMI(11), No. 10, October 1989, pp. 1087-1101.
IEEE Abstract. IEEE Top Reference.
WWW Version. BibRef 8910

Pawlak, M.[Miroslaw],
On the asymptotic properties of smoothed estimators of the classification error rate,
PR(21), No. 5, 1988, pp. 515-524.
WWW Version. 0309 BibRef

Pawlak, M., Liao, X.,
Estimation of error rates using smoothed estimators,
ICPR88(II: 954-956).
IEEE DOI may work or IEEE-CS DOI may work. 8811 BibRef

Devroye, L.,
Automatic Pattern Recognition: A Study of the Probability of Error,
PAMI(10), No. 4, July 1988, pp. 530-543.
IEEE Abstract. IEEE Top Reference.
WWW Version. BibRef 8807

Devroye, L., Gyorfi, L., Lugosi, G.,
Probabilistic Theory of Pattern Recognition,
Springer-Verlag1996. BibRef 9600

Zhu, Q.M.[Qiu-Ming],
On the minimum probability of error of classification with incomplete patterns,
PR(23), No. 11, 1990, pp. 1281-1290.
WWW Version. 0401 BibRef

Kalkanis, G., Conroy, G.V.,
Interval Error Estimators in Class Probability Trees,
PRL(17), No. 7, June 10 1996, pp. 705-712. 9607 BibRef

Durso, G., Menenti, M.,
Performance Indicators for the Statistical Evaluation of Digital Image Classifications,
PandRS(51), No. 2, April 1996, pp. 78-90. 9605 BibRef

Pal, N.R., Biswas, J.,
Cluster Validation Using Graph-Theoretic Concepts,
PR(30), No. 6, June 1997, pp. 847-857.
WWW Version. 9706 BibRef

Kloditz, C., Vanboxtel, A., Carfagna, E., Vandeursen, W.,
Estimating the Accuracy of Coarse Scale Classification Using High Scale Information,
PhEngRS(64), No. 2, February 1998, pp. 127-133. 9803 BibRef

Bax, E.,
Validation of Average Error Rate over Classifiers,
PRL(19), No. 2, February 1998, pp. 127-132. 9808 BibRef

Bax, E.[Eric],
Improved Hoeffding-style performance guarantees for accurate classifiers,
PRL(20), No. 4, April 1999, pp. 445-449. BibRef 9904

Bouchaffra, D.[Djamel], Govindaraju, V.[Venu], Srihari, S.[Sargur],
A Methodology for Mapping Scores to Probabilities,
PAMI(21), No. 9, September 1999, pp. 923-927.
IEEE Abstract. IEEE Top Reference.
WWW Version. BibRef 9909
Earlier:
A Methodology for Deriving Probabilistic Correctness Measures from Recognizers,
CVPR98(930-935).
IEEE Abstract. IEEE Top Reference. Derive a probability of correctness that can be compared across all classifiers. BibRef

Tulyakov, S., Govindaraju, V.,
Combining matching scores in identification model,
ICDAR05(II: 1151-1155).
IEEE DOI may work or IEEE-CS DOI may work. 0508Combining scores. Best score not always best, depending on number of options. BibRef

Ho, T.K., Basu, M.,
Complexity Measures of Supervised Classification Problems,
PAMI(24), No. 3, March 2002, pp. 289-300.
IEEE Abstract. IEEE Top Reference.
WWW Version. 0202 BibRef
Earlier:
Measuring the Complexity of Classification Problems,
ICPR00(Vol II: 43-47).
IEEE DOI may work or IEEE-CS DOI may work.
HTML Version. 0009 BibRef

Clarkson, E.[Eric],
Bounds on the area under the receiver operating characteristic curve for the ideal observer,
JOSA-A(19), No. 10, October 2002, pp. 1963-1968.
WWW Version. 0210 BibRef

Clarkson, E.[Eric],
Estimation receiver operating characteristic curve and ideal observers for combined detection/estimation tasks,
JOSA-A(24), No. 12, December 2007, pp. B91-B98.
WWW Version. 0801 BibRef

Berikov, V.B.[Vladimir B.], Litvinenko, A.[Alexander],
The influence of prior knowledge on the expected performance of a classifier,
PRL(24), No. 15, November 2003, pp. 2537-2548.
WWW Version. 0308 See also approach to the evaluation of the performance of a discrete classifier, An. BibRef

Dougherty, E.R.[Edward R.], Brun, M.[Marcel],
A probabilistic theory of clustering,
PR(37), No. 5, May 2004, pp. 917-925.
WWW Version. 0405 BibRef

Braga-Neto, U.[Ulisses], Dougherty, E.R.[Edward R.],
Bolstered error estimation,
PR(37), No. 6, June 2004, pp. 1267-1281.
WWW Version. 0405For further info:
WWW Version. BibRef

Braga-Neto, U.[Ulisses], Dougherty, E.R.[Edward R.],
Exact performance of error estimators for discrete classifiers,
PR(38), No. 11, November 2005, pp. 1799-1814.
WWW Version. 0509 BibRef

Brun, M.[Marcel], Sima, C.[Chao], Hua, J.P.[Jian-Ping], Lowey, J.[James], Carroll, B.[Brent], Suh, E.[Edward], Dougherty, E.R.[Edward R.],
Model-based evaluation of clustering validation measures,
PR(40), No. 3, March 2007, pp. 807-824.
WWW Version. 0611Clustering algorithms; Clustering errors; Validation indices BibRef

Edwards, D.C., Metz, C.E., Kupinski, M.A.,
Ideal Observers and Optimal ROC Hypersurfaces in N-Class Classification,
MedImg(23), No. 7, July 2004, pp. 891-895.
IEEE Abstract. IEEE Top Reference. 0407 See also Ideal observer approximation using bayesian classification neural networks. BibRef

Edwards, D.C., Metz, C.E., Nishikawa, R.M.,
The Hypervolume Under the ROC Hypersurface of 'Near-Guessing' and 'Near-Perfect' Observers in N-Class Classification Tasks,
MedImg(24), No. 3, March 2005, pp. 293-299.
IEEE Abstract. IEEE Top Reference. 0501 BibRef

Edwards, D.C., Metz, C.E.,
Restrictions on the three-class ideal observer's decision boundary lines,
MedImg(24), No. 12, December 2005, pp. 1566-1573.
IEEE DOI may work or IEEE-CS DOI may work. 0601 BibRef

Edwards, D.C., Metz, C.E.,
Optimization of Restricted ROC Surfaces in Three-Class Classification Tasks,
MedImg(26), No. 10, October 2007, pp. 1345-1356.
IEEE DOI may work or IEEE-CS DOI may work. 0711 BibRef

He, X., Metz, C.E., Tsui, B.M.W., Links, J.M., Frey, E.C.,
Three-Class ROC Analysis: A Decision Theoretic Approach Under the Ideal Observer Framework,
MedImg(25), No. 5, May 2006, pp. 571-581.
IEEE DOI may work or IEEE-CS DOI may work. 0605 BibRef

He, X., Frey, E.C.,
Three-Class ROC Analysis: The Equal Error Utility Assumption and the Optimality of Three-Class ROC Surface Using the Ideal Observer,
MedImg(25), No. 8, August 2006, pp. 979-986.
IEEE DOI may work or IEEE-CS DOI may work. 0608 BibRef

He, X.[Xin], Frey, E.C.,
The Meaning and Use of the Volume Under a Three-Class ROC Surface (VUS),
MedImg(27), No. 5, May 2008, pp. 577-588.
IEEE DOI may work or IEEE-CS DOI may work. 0711 BibRef

He, X., Frey, E.C.,
An Optimal Three-Class Linear Observer Derived From Decision Theory,
MedImg(26), No. 1, January 2007, pp. 77-83.
IEEE DOI may work or IEEE-CS DOI may work. 0701 BibRef

Baraldi, A., Bruzzone, L., Blonda, P.,
Quality Assessment of Classification and Cluster Maps Without Ground Truth Knowledge,
GeoRS(43), No. 4, April 2005, pp. 857-873.
IEEE Abstract. IEEE Top Reference. 0501 BibRef

Santos-Pereira, C.M.[Carla M.], Pires, A.M.[Ana M.],
On optimal reject rules and ROC curves,
PRL(26), No. 7, 15 May 2005, pp. 943-952.
WWW Version. 0506 BibRef

DeVore, M.D.,
Estimates of Error Probability for Complex Gaussian Channels with Generalized Likelihood Ratio Detection,
PAMI(27), No. 10, October 2005, pp. 1580-1591.
IEEE DOI may work or IEEE-CS DOI may work. 0509Two-class hypothesis testing. BibRef

Khurd, P., Gindi, G.,
Decision strategies that maximize the area under the LROC curve,
MedImg(24), No. 12, December 2005, pp. 1626-1636.
IEEE DOI may work or IEEE-CS DOI may work. 0601 BibRef

Ahlqvist, O.[Ola], Gahegan, M.[Mark],
Probing the Relationship Between Classification Error and Class Similarity,
PhEngRS(71), No. 12, December 2005, pp. 1365-1374.
WWW Version. 0602A method that predicts land-cover classification errors by using semantic similarity metrics derived from land-cover taxonomy definitions. BibRef

Landgrebe, T.C.W.[Thomas C.W.], Tax, D.M.J.[David M.J.], Paclík, P.[Pavel], Duin, R.P.W.[Robert P.W.],
The interaction between classification and reject performance for distance-based reject-option classifiers,
PRL(27), No. 8, June 2006, pp. 908-917.
WWW Version. Unseen classes; Reject-option; Model selection 0605 BibRef

Fawcett, T.[Tom],
ROC graphs with instance-varying costs,
PRL(27), No. 8, June 2006, pp. 882-891.
WWW Version. Cost-sensitive learning; Classifier evaluation 0605 BibRef

Everson, R.M.[Richard M.], Fieldsend, J.E.[Jonathan E.],
Multi-class ROC analysis from a multi-objective optimisation perspective,
PRL(27), No. 8, June 2006, pp. 918-927.
WWW Version. Evolutionary computation; Pareto optimality; Gini coefficient 0605 BibRef

Matei, B.C.[Bogdan C.], Meer, P.[Peter],
Estimation of Nonlinear Errors-in-Variables Models for Computer Vision Applications,
PAMI(28), No. 10, October 2006, pp. 1537-1552.
IEEE DOI may work or IEEE-CS DOI may work. 0609 BibRef
Earlier:
A General Method for Errors-in-Variables Problems in Computer Vision,
CVPR00(II: 18-25).
IEEE Abstract. IEEE Top Reference.
WWW Version. 0005All measurements are noisy. Related to Sampson, renormalization, numerical. BibRef

Landgrebe, T.C.W.[Thomas C.W.], Duin, R.P.W.[Robert P.W.],
Approximating the multiclass ROC by pairwise analysis,
PRL(28), No. 13, 1 October 2007, pp. 1747-1758.
WWW Version. 0709ROC analysis; Multiclass ROC; Cost sensitive; Threshold optimisation BibRef

Ericsson, A.[Anders], Karlsson, J.[Johan],
Measures for Benchmarking of Automatic Correspondence Algorithms,
JMIV(28), No. 3, July 2007, pp. 225-241.
WWW Version. 0709 BibRef
Earlier: A2, A1:
A Ground Truth Correspondence Measure for Benchmarking,
ICPR06(III: 568-573).
WWW Version. 0609 BibRef
And: A1, A2:
Benchmarking of algorithms for automatic correspondence localisation,
BMVC06(II:759).
PDF Version. 0609 BibRef

Waegeman, W.[Willem], De Baets, B.[Bernard], Boullart, L.[Luc],
ROC analysis in ordinal regression learning,
PRL(29), No. 1, 1 January 2008, pp. 1-9.
WWW Version. 0711ROC analysis; Ranking; Ordinal regression; Unbalanced learning problems; Performance measures; Machine learning BibRef

Gallas, B.D.[Brandon D.], Pennello, G.A.[Gene A.], Myers, K.J.[Kyle J.],
Multireader multicase variance analysis for binary data,
JOSA-A(24), No. 12, December 2007, pp. B70-B80.
WWW Version. 0801Analyzing ROC (receiver operating characteristic) curve data. BibRef

Marrocco, C.[Claudio], Duin, R.P.W., Tortorella, F.[Francesco],
Maximizing the area under the ROC curve by pairwise feature combination,
PR(41), No. 6, June 2008, pp. 1961-1974.
WWW Version. 0802Two-class problems; ROC curve; Ranking; AUC BibRef

El Ayadi, M.M.H.[Moataz M.H.], Kamel, M.S.[Mohamed S.], Karray, F.[Fakhri],
Toward a tight upper bound for the error probability of the binary Gaussian classification problem,
PR(41), No. 6, June 2008, pp. 2120-2132.
WWW Version. 0802Binary classification; Bayesian decision rule; Decision boundary; Error probability; Monte-Carlo simulations; Multivariate normal distribution; Quadratic surfaces BibRef

Fisher, R.B.,
An Empirical Model for Saturation and Capacity in Classifier Spaces,
ICPR06(IV: 189-193).
WWW Version. 0609Determine the achievable classification rate for a database given a level of noise. BibRef

Maloof, M.A.,
On machine learning, ROC analysis, and statistical tests of significance,
ICPR02(II: 204-207).
IEEE DOI may work or IEEE-CS DOI may work. 0211 BibRef

Johnson, A.Y., Bobick, A.F.,
Relationship between identification metrics: Expected confusion and area under a ROC curve,
ICPR02(III: 662-666).
IEEE DOI may work or IEEE-CS DOI may work. 0211 BibRef

Rees, G.S., Wright, W.A., Greenway, P.,
ROC Method for the Evaluation of Multi-class Segmentation/Classification Algorithms with Infrared Imagery,
BMVC02(Poster Session). 0208 BibRef

Ménard, M., Doget, T., Shahin, A.,
Ambiguity Concept and Switching Regression Models,
SCIA99(Pattern Recognition). BibRef 9900

Raudys, S.J., Diciunas, V.,
Expected Error of Minimum Empirical Error and Maximal Margin Classifiers,
ICPR96(II: 875-879).
IEEE DOI may work or IEEE-CS DOI may work. 9608(Institute of Mathematics and Informatics, LIT) BibRef

Kanungo, T., Gay, D.M., Haralick, R.M.,
Constrained monotone regression of ROC curves and histograms using splines and polynomials,
ICIP95(II: 292-295).
IEEE DOI may work or IEEE-CS DOI may work. 9510 BibRef

Grossman, T., Lapedes, A.,
Noise sensitivity signatures for model selection,
ICPR94(B:213-218).
IEEE DOI may work or IEEE-CS DOI may work. 9410 BibRef

Chapter on Pattern Recognition, Clustering, Statistics, Grammars, Learning, Neural Nets, Genetic Algorithms continues in
Multiple Classifiers, Combining Classifiers, Combinations .