14.1.5.2.2 Classifier Combination, Evaluation, Overview, Appliction Specific

Boland, P.J.,
Majority Systems and the Condorcet Jury Theorem,
Statistician(38), 1989, pp. 181-189. For independent classifiers, error eat less than .5, for an odd number of classifiers, majority voting increases the correct decision rate as the number of classifiers increases. BibRef 8900

Xu, L., Krzyzak, A., Suen, C.Y.,
Methods of Combining Multiple Classifiers and Their Applications to Handwriting Recognition,
SMC(22), No. 3, 1992, pp. 418-435. Majority Voting for combinations. Unanimous consensus. Threshold Voting. Averaged Bayes Classifier. Dempster-Shafer. BibRef 9200

Xu, L.[Lei], Krzyzak, A., Oja, E.,
Unsupervised and supervised classifications by rival penalized competitive learning,
ICPR92(II:496-499).
IEEE DOI Reference 9208 BibRef

Lam, L., Suen, C.Y.,
Application of Majority Voting to Pattern Recognition: An Analysis of Its Behavior and Performance,
SMC(27), No. 5, 1997, pp. 553-568. BibRef 9700
Earlier:
A Theoretical Analysis of the Application of Majority Voting to Pattern Recognition,
ICPR94(B:418-420).
IEEE DOI Reference BibRef

Smyth, P.,
Bounds on the Mean Classification Error Rate of Multiple Experts,
PRL(17), No. 12, October 25 1996, pp. 1253-1257. 9612 BibRef

Kittler, J.V.[Joseph V.],
Combining Classifiers: A Theoretical Framework,
PAA(1), No. 1, 1998, pp. 18-27. BibRef 9800

Kittler, J.V., Hatef, M., Duin, R.P.W., Matas, J.,
On Combining Classifiers,
PAMI(20), No. 3, March 1998, pp. 226-239.
IEEE Abstract. IEEE Top Reference.
WWW Version. 9805The combination rule using the most restrictive assumptions, the sum rule, did best. Compared versions of Max, Median, Majority vote rule dreived from See also Improving Model Accuracy Using Optimal Linear Combinations of Trained Neural Networks. Min rule derived from See also Multistage Algorithm for Fast Classification of Patterns, A. Also Sum and Product rules. BibRef

Kittler, J.V., Duin, R.P.W., Hatef, M.,
Combining Classifiers,
ICPR96(II: 897-901).
IEEE DOI Reference 9608(Univ. of Surrey, UK) BibRef

Wang, L.C., Der, S.Z., Nasrabadi, N.M.,
Composite Classifiers for Automatic Target Recognition,
OptEng(37), No. 3, March 1998, pp. 858-868. 9804 BibRef

Wang, L.C., Chan, L., Nasrabadi, N.M., and Der, S.Z.,
Combination of Two Learning Algorithms for Automatic Target Recognition,
ICIP97(I: 881-884).
IEEE DOI Reference BibRef 9700

Ng, G.S., Singh, H.,
Data Equalization with Evidence Combination for Pattern Recognition,
PRL(19), No. 3-4, March 1998, pp. 227-235. 9807 BibRef

Tax, D.M.J.[David M.J.], van Breukelen, M.[Martijn], Duin, R.P.W.[Robert P.W.], Kittler, J.V.[Josef V.],
Combining multiple classifiers by averaging or by multiplying?,
PR(33), No. 9, September 2000, pp. 1475-1485.
WWW Version. 0005 BibRef

van Breukelen, M.[Martijn], Duin, R.P.W.[Robert P.W.],
Neural Network Initialization by Combined Classifiers,
ICPR98(Vol I: 215-218).
IEEE DOI Reference 9808 BibRef

Pudil, P., Novovicova, J., Blaha, S., Kittler, J.V.,
Multistage Pattern Recognition with Reject Option,
ICPR92(II:92-95).
IEEE DOI Reference BibRef 9200

Kittler, J.V., Hojjatoleslami, A., Windeatt, T.,
Strategies for Combining Classifiers Employing Shared and Distinct Pattern Representations,
PRL(18), No. 11-13, November 1997, pp. 1373-1377. 9806 BibRef
Earlier:
Weighting Factors in Multiple Expert Fusion,
BMVC97(xx-yy).
HTML Version. 0209 BibRef

Hojjatoleslami, A., Kittler, J.V.[Josef V.],
Strategies for Weighted Combination of Classifiers Employing Shared and Distinct Pattern Representations,
ICPR98(Vol I: 338-340).
IEEE DOI Reference 9808 BibRef

Kittler, J.V.[Josef V.], Hojjatoleslami, A.[Ali],
A Weighted Combination of Classifiers Employing Shared and Distinct Representations,
CVPR98(924-929).
IEEE Abstract. IEEE Top Reference. BibRef 9800

Prior, M., Windeatt, T.,
Parameter Tuning using the Out-of-Bootstrap Generalisation Error Estimate for Stochastic Discrimination and Random Forests,
ICPR06(II: 498-501).
WWW Version. 0609 BibRef

Alkoot, F.M., Kittler, J.V.,
Experimental Evaluation of Expert Fusion Strategies,
PRL(20), 1999, pp. 1361-1369. BibRef 9900
And:
Improving the Performance of the Product Fusion Strategy,
ICPR00(Vol II: 164-167).
IEEE DOI Reference
HTML Version. 0009Compared Minimum, Maximum, Average, Median, Majority. Futher analysis in: See also Theoretical Study on Six Classifier Fusion Strategies, A. BibRef

Kittler, J.V., Alkoot, F.M.,
Sum versus vote fusion in multiple classifier systems,
PAMI(25), No. 1, January 2003, pp. 110-115.
IEEE Abstract. IEEE Top Reference.
WWW Version. 0301Analysis of fusion rules. BibRef

Martínez Trinidad, J.F.[José Francisco], Guzmán Arenas, A.[Adolfo],
The logical combinatorial approach to pattern recognition, an overview through selected works,
PR(34), No. 4, April 2001, pp. 741-751.
WWW Version. 0101 BibRef

Kuncheva, L.I.[Ludmila I.],
Switching between selection and fusion in combining classifiers: An Experiment,
SMC-B(32), No. 2, April 2002, pp. 146-156.
IEEE Top Reference. 0205 BibRef

Kuncheva, L.I.[Ludmila I.],
Using diversity measures for generating error-correcting output codes in classifier ensembles,
PRL(26), No. 1, 1 January 2005, pp. 83-90.
WWW Version. 0501 BibRef

Kuncheva, L.I.[Ludmila I.],
A Theoretical Study on Six Classifier Fusion Strategies,
PAMI(24), No. 2, February 2002, pp. 281-286.
IEEE Abstract. IEEE Top Reference.
WWW Version. 02022 classes and L classifiers. Minimum, Maximum, Average, Median, Majority, Product. Minimum/Maximum best. See also Experimental Evaluation of Expert Fusion Strategies. BibRef

Kuncheva, L.I., and Whitaker, C.J.,
Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy,
MachLearn(51), 2003, pp. 181-207 BibRef 0300

Kuncheva, L.I., Whitaker, C.J., Shipp, C.A., Duin, R.P.W.,
Limits on the Majority Vote Accuracy in Classifier Fusion,
PAA(6), No. 1, 2003, pp. 22-31. BibRef 0300
Earlier:
Is Independence Good for Combining Classifiers?,
ICPR00(Vol II: 168-171).
IEEE DOI Reference
HTML Version. 0009 BibRef

Kuncheva, L.I., Whitaker, C.J., Narasimhamurthy, A.,
A case-study on naive labelling for the nearest mean and the linear discriminant classifiers,
PR(41), No. 10, October 2008, pp. 3010-3020.
WWW Version. 0808Semi-supervised learning; Unlabelled data; On-line classifiers; Naive labelling BibRef

Alexandre, L.A.[Luís A.], Campilho, A.C.[Aurélio C.], Kamel, M.S.[Mohamed S.],
On combining classifiers using sum and product rules,
PRL(22), No. 12, October 2001, pp. 1283-1289.
HTML Version. 0108 BibRef
Earlier:
Combining Independent and Unbiased Classifiers Using Weighted Average,
ICPR00(Vol II: 495-498).
IEEE DOI Reference
HTML Version. 0009 BibRef

Priebe, C.E.[Carey E.],
Olfactory Classification via Interpoint Distance Analysis,
PAMI(23), No. 4, April 2001, pp. 404-413.
IEEE Abstract. IEEE Top Reference.
WWW Version. 0104Use a set of subsample classifiers and combine them. BibRef

Lu, Y.[Yue], Tan, C.L.[Chew Lim],
Combination of multiple classifiers using probabilistic dictionary and its application to postcode recognition,
PR(35), No. 12, December 2002, pp. 2823-2832.
WWW Version. 0209 BibRef

Oh, S.B.[Sang-Bong],
On the relationship between majority vote accuracy and dependency in multiple classifier systems,
PRL(24), No. 1-3, January 2003, pp. 359-363.
HTML Version. 0211 BibRef

Murua, A.[Alejandro],
Upper Bounds for Error Rates of Linear Combinations of Classifiers,
PAMI(24), No. 5, May 2002, pp. 591-602.
IEEE Abstract. IEEE Top Reference.
WWW Version. 0205Analyze classifiers constructed with same training data. Depending on dependence between them, linear combinations will achieve good error performance. BibRef

Giacinto, G.[Giorgio], Roli, F.[Fabio],
An approach to the automatic design of multiple classifier systems,
PRL(22), No. 1, January 2001, pp. 25-33.
HTML Version. 0105 See also Combination of neural and statistical algorithms for supervised classification of remote-sensing images. BibRef
Earlier:
A Theoretical Framework for Dynamic Classifier Selection,
ICPR00(Vol II: 8-11).
IEEE DOI Reference
HTML Version. 0009 BibRef
Earlier:
Methods for dynamic classifier selection,
CIAP99(659-664).
IEEE DOI Reference 9909 BibRef
Earlier:
Adaptive selection of image classifiers,
CIAP97(I: 38-45).
WWW Version. 9709 BibRef

Fumera, G.[Giorgio], Roli, F.[Fabio], Giacinto, G.[Giorgio],
Reject option with multiple thresholds,
PR(33), No. 12, December 2000, pp. 2099-2101.
WWW Version. 0401 BibRef

Giacinto, G.[Giorgio], Roli, F.[Fabio],
Dynamic classifier selection based on multiple classifier behaviour,
PR(34), No. 9, September 2001, pp. 1879-1881.
WWW Version. 0108 BibRef

Giacinto, G.[Giorgio], Roli, F.[Fabio], Didaci, L.[Luca],
Fusion of multiple classifiers for intrusion detection in computer networks,
PRL(24), No. 12, August 2003, pp. 1795-1803.
WWW Version. 0304 BibRef
Earlier: A1, A2 only:
Intrusion detection in computer networks by multiple classifier systems,
ICPR02(II: 390-393).
IEEE DOI Reference 0211 BibRef

Giacinto, G.[Giorgio], Perdisci, R.[Roberto], Roli, F.[Fabio],
Network Intrusion Detection by Combining One-Class Classifiers,
CIAP05(58-65).
Springer DOI Reference 0509 BibRef

Tronci, R.[Roberto], Giacinto, G.[Giorgio], Roli, F.[Fabio],
Combination of Experts by Classifiers in Similarity Score Spaces,
SSPR08(821-830).
Springer DOI Reference 0812 BibRef

Didaci, L.[Luca], Roli, F.[Fabio],
Using Co-training and Self-training in Semi-supervised Multiple Classifier Systems,
SSPR06(522-530).
Springer DOI Reference 0608 BibRef

Fumera, G.[Giorgio], Roli, F.[Fabio],
Analysis of error-reject trade-off in linearly combined multiple classifiers,
PR(37), No. 6, June 2004, pp. 1245-1265.
WWW Version. 0405 BibRef

Roli, F., Fumera, G., Vernazza, G.,
Analysis of error-reject trade-off in linearly combined classifiers,
ICPR02(II: 120-123).
IEEE DOI Reference 0211 BibRef
Earlier: A2, A1, A3:
A method for error rejection in multiple classifier systems,
CIAP01(454-458).
IEEE Top Reference. 0210 BibRef

Lin, X.[Xiaofan], Yacoub, S.[Sherif], Burns, J.[John], Simske, S.[Steven],
Performance analysis of pattern classifier combination by plurality voting,
PRL(24), No. 12, August 2003, pp. 1959-1969.
WWW Version. 0304 BibRef

Fumera, G.[Giorgio], Roli, F.[Fabio],
A Theoretical and Experimental Analysis of Linear Combiners for Multiple Classifier Systems,
PAMI(27), No. 6, June 2005, pp. 942-956.
IEEE Abstract. IEEE Top Reference. 0505Analysis follows on See also Analysis of decision boundaries in linearly combined neural classifiers. Performance depends on individual classifiers and correlation between them. BibRef

Fumera, G.[Giorgio], Fabio, R.[Roli], Alessandra, S.[Serrau],
A Theoretical Analysis of Bagging as a Linear Combination of Classifiers,
PAMI(30), No. 7, July 2008, pp. 1293-1299.
IEEE DOI Reference 0806Analysis derived from linear combinations to Bagging approaches. BibRef

Biggio, B.[Battista], Fumera, G.[Giorgio], Roli, F.[Fabio],
Adversarial Pattern Classification Using Multiple Classifiers and Randomisation,
SSPR08(500-509).
Springer DOI Reference 0812 BibRef

Didaci, L.[Luca], Giacinto, G.[Giorgio], Roli, F.[Fabio], Marcialis, G.L.[Gian Luca],
A study on the performances of dynamic classifier selection based on local accuracy estimation,
PR(38), No. 11, November 2005, pp. 2188-2191.
WWW Version. 0509 BibRef

Yildiz, O.T.[Olcay Taner], Alpaydin, E.[Ethem],
Ordering and Finding the Best of K>2 Supervised Learning Algorithms,
PAMI(28), No. 3, March 2006, pp. 392-402.
IEEE DOI Reference 0602Given a dataset and a set of algorithms, find the one that is best. BibRef

Vilariño, F.[Fernando], Kuncheva, L.I.[Ludmila I.], Radeva, P.I.[Petia I.],
ROC curves and video analysis optimization in intestinal capsule endoscopy,
PRL(27), No. 8, June 2006, pp. 875-881.
WWW Version. Classifiers ensemble; Imbalanced classes; Wireless capsule endoscopy 0605 BibRef

Rodriguez, J.J., Kuncheva, L.I.[Ludmila I.], Alonso, C.J.,
Rotation Forest: A New Classifier Ensemble Method,
PAMI(28), No. 10, October 2006, pp. 1619-1630.
IEEE DOI Reference 0609 BibRef

Rodríguez, J.J.[Juan J.], Kuncheva, L.I.[Ludmila I.],
Combining Online Classification Approaches for Changing Environments,
SSPR08(520-529).
Springer DOI Reference 0812 BibRef

Kuncheva, L.I.[Ludmila I.], Vetrov, D.P.,
Evaluation of Stability of k-Means Cluster Ensembles with Respect to Random Initialization,
PAMI(28), No. 11, November 2006, pp. 1798-1808.
IEEE DOI Reference 0609 BibRef

Cabrera, J.B.D.[João B.D.],
On the impact of fusion strategies on classification errors for large ensembles of classifiers,
PR(39), No. 11, November 2006, pp. 1963-1978.
WWW Version. 0608Classifier fusion; Asymptotic methods; Independent classifiers; Sensor networks BibRef

Canuto, A.M.P.[Anne M.P.], Abreu, M.C.C.[Marjory C.C.], de Melo Oliveira, L.[Lucas], Xavier, Jr., J.C.[João C.], de M. Santos, A.[Araken],
Investigating the influence of the choice of the ensemble members in accuracy and diversity of selection-based and fusion-based methods for ensembles,
PRL(28), No. 4, 1 March 2007, pp. 472-486.
WWW Version. 0701Diversity measures; Classifier ensembles; Selection-based combination methods; Fusion-based combination methods BibRef

Hu, R.[Roland], Damper, R.I.,
A 'No Panacea Theorem' for classifier combination,
PR(41), No. 8, August 2008, pp. 2665-2673.
WWW Version. 0805 BibRef
Earlier:
A 'No Panacea Theorem' for Multiple Classifier Combination,
ICPR06(II: 1250-1253).
WWW Version. 0609Probability density functions; Gaussian mixtures; `No Free Lunch' theorems BibRef

Freitas, C.O.A.[Cinthia O. A.], de Carvalho, J.M.[João M.], Oliveira, J.J.[José-Josemar], Aires, S.B.K.[Simone B. K.], Sabourin, R.[Robert],
Confusion Matrix Disagreement for Multiple Classifiers,
CIARP07(387-396).
Springer DOI Reference 0711 BibRef

Moreno-Seco, F.[Francisco], Iñesta, J.M.[José M.], Ponce de León, P.J.[Pedro J.], Micó, L.[Luisa],
Comparison of Classifier Fusion Methods for Classification in Pattern Recognition Tasks,
SSPR06(705-713).
Springer DOI Reference 0608 BibRef

Abdulkader, A.[Ahmad], Drakopoulos, J.A.[John A.], Zhang, Q.[Qi],
Comparative Classifier Aggregation,
ICPR06(III: 156-159).
WWW Version. 0609 BibRef

Bertolami, R.[Roman], Bunke, H.[Horst],
Early feature stream integration versus decision level combination in a multiple classifier system for text line recognition,
ICPR06(II: 845-848).
WWW Version. 0609 BibRef

Dietrich, C.[Christian], Schwenker, F.[Friedhelm], Palm, G.[Günther],
Multiple Classifier Systems for the Recognition of Orthoptera Songs,
DAGM03(474-481).
HTML Version. 0310 BibRef

Duin, R.P.W.,
The combining classifier: to train or not to train?,
ICPR02(II: 765-770).
IEEE DOI Reference 0211 BibRef

Schiele, B.,
How many classifiers do I need?,
ICPR02(II: 176-179).
IEEE DOI Reference 0211 BibRef

Chapter on Pattern Recognition, Clustering, Statistics, Grammars, Learning, Neural Nets, Genetic Algorithms continues in
Decision Fusion .