Abstract
Classifier combination methods need to make best use of the outputs of multiple, imperfect classifiers to enable higher accuracy classifications. In many situations, such as when human decisions need to be combined, the base decisions can vary enormously in reliability. A Bayesian approach to such uncertain combination allows us to infer the differences in performance between individuals and to incorporate any available prior knowledge about their abilities when training data is sparse. In this chapter we explore Bayesian classifier combination, using the computationally efficient framework of variational Bayesian inference. We apply the approach to real data from a large citizen science project, Galaxy Zoo Supernovae, and show that our method far outperforms other established approaches to imperfect decision combination.We go on to analyse the putative community structure of the decision makers, based on their inferred decision making strategies, and show that natural groupings are formed. Finally we present a dynamic Bayesian classifier combination approach and investigate the changes in base classifier performance over time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Abbeel, P., Ng, A.Y.: Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the Twenty-First International Conference on Machine Learning, ICML 2004, p. 1. ACM, New York (2004), http://doi.acm.org/10.1145/1015330.1015430
Abramowitz, M., Stegun, I.A. (eds.): Handbook of Mathematical Functions: with Formulas, Graphs, and Mathematical Tables. Dover Publications (1965)
Attias, H.: A Variational Bayesian Framework for Graphical Models. In: Advances in Neural Information Processing Systems 12, pp. 209–215 (2000)
Bierman, G.J.: Fixed interval smoothing with discrete measurements. International Journal of Control 18(1), 65–75 (1973), http://www.tandfonline.com/doi/abs/10.1080/00207177308932487
Bishop, C.M.: Pattern recognition and machine learning, 4th edn. Information Science and Statistics. Springer Science+Business Media, LLC (2006)
Bloodgood, M., Callison-Burch, C.: Using Mechanical Turk to build machine translation evaluation sets. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazons Mechanical Turk, Los Angeles, pp. 208–211 (2010)
Choudrey, R., Roberts, S.: Variational mixture of Bayesian independent component analysers. Neural Computation 15(1) (2003)
Dash, R.K., Jennings, N.R., Parkes, D.C.: Computational-mechanism design: a call to arms. IEEE Intelligent Systems 18(6), 40–47 (2003)
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer Error-Rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) 28(1), 20–28 (1979), http://www.jstor.org/stable/2346806
Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39(1), 1–38 (1977)
Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006), http://dx.doi.org/10.1016/j.patrec.2005.10.010 , doi:10.1016/j.patrec.2005.10.010
Fox, C., Roberts, S.: A tutorial on variational Bayesian inference. Artificial Intelligence Review 38(2), 85–95 (2011), http://www.springerlink.com/content/j5617xn756j27671/abstract/ , doi:10.1007/s10462-011-9236-8
Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-6(6), 721–741 (1984)
Ghahramani, Z., Kim, H.C.: Bayesian classifier combination. Gatsby Computational Neuroscience Unit Technical Report GCNU-T, London, UK (2003)
Gilks, W.R., Wild, P.: Adaptive rejection sampling for gibbs sampling. Journal of the Royal Statistical Society. Series C (Applied Statistics) 41(2), 337–348 (1992)
Girvan, M., Newman, M.E.J.: Community structure in social and biological networks. Proceedings of the National Academy of Sciences 99(12), 7821–7826 (2002), http://www.pnas.org/content/99/12/7821 , doi:10.1073/pnas.122653799
Kullback, S., Leibler, R.A.: On information and sufficiency. The Annals of Mathematical Statistics 22(1), 79–86 (1951), http://www.jstor.org/stable/2236703
Law, N.M., Kulkarni, S.R., Dekany, R.G., Ofek, E.O., Quimby, R.M., Nugent, P.E., Surace, J., Grillmair, C.C., Bloom, J.S., Kasliwal, M.M., et al.: The Palomar transient factory: System overview, performance, and first results. Publications of the Astronomical Society of the Pacific 121(886), 1395–1408 (2009)
Lee, S.M., Roberts, S.J.: Sequential dynamic classification using latent variable models. The Computer Journal 53(9), 1415–1429 (2010), http://comjnl.oxfordjournals.org/content/53/9/1415 , doi:10.1093/comjnl/bxp127
Lefkimmiatis, S., Maragos, P., Papandreou, G.: Bayesian inference on multiscale models for Poisson intensity estimation: Applications to Photon-Limited image denoising. IEEE Transactions on Image Processing 18(8), 1724–1741 (2009), doi:10.1109/TIP.2009.2022008
Littlestone, N., Warmuth, M.K.: The weighted majority algorithm. In: 30th Annual Symposium on Foundations of Computer Science 1989, pp. 256–261 (1989)
Monteith, K., Carroll, J.L., Seppi, K., Martinez, T.: Turning Bayesian model averaging into Bayesian model combination. In: The 2011 International Joint Conference on Neural Networks (IJCNN), pp. 2657–2663. IEEE (2011), doi:10.1109/IJCNN.2011.6033566
Parisi, G., Shankar, R.: Statistical field theory. Physics Today 41, 110 (1988)
Penny, W.D., Roberts, S.J.: Dynamic logistic regression. In: International Joint Conference on Neural Networks, IJCNN 1999, vol. 3, pp. 1562–1567 (2002)
Psorakis, I., Roberts, S.J., Ebden, M., Sheldon, B.: Overlapping community detection using Bayesian non-negative matrix factorization. Physical Review E 83(6) (2011), http://www.orchid.ac.uk/eprints/9/ , doi:10.1103/PhysRevE.83.066114
Quinn, A.J., Bederson, B.B., Yeh, T., Lin, J.: Crowdflow: Integrating machine learning with Mechanical Turk for speed-cost-quality flexibility. Tech. rep., HCIL-2010-09, University of Maryland, College Park (2010)
Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy, L.: Learning from crowds. J. Mach. Learn. Res. 11, 1297–1322 (2010), http://dl.acm.org/citation.cfm?id=1756006.1859894
Smith, A.M., Lynn, S., Sullivan, M., Lintott, C.J., Nugent, P.E., Botyanszki, J., Kasliwal, M., Quimby, R., Bamford, S.P., Fortson, L.F., et al.: Galaxy Zoo Supernovae. Monthly Notices of the Royal Astronomical Society (2010)
Tan, V.Y.F., Fvotte, C.: Automatic relevance determination in nonnegative matrix factorization. In: SPARS 2009 - Signal Processing with Adaptive Sparse Structured Representations, Inria Rennes-Bretagne Atlantique, Saint Malo, Royaume-Uni (2009), http://hal.inria.fr/view_by_stamp.php?&halsid=oerba5b5pkfqikvfb5e1ideks4&label=SPARS09&langue=en&action_todo=view&id=inria-00369376&version=1
Tulyakov, S., Jaeger, S., Govindaraju, V., Doermann, D.: Review of classifier combination methods. In: Machine Learning in Document Analysis and Recognition, pp. 361–386 (2008)
Weatherburn, C.E.: A First Course in Mathematical Statistics. CUP Archive (1949)
West, M., Harrison, J.: Bayesian forecasting and dynamic models. Springer (1997)
West, M., Harrison, P.J., Migon, H.S.: Dynamic generalized linear models and Bayesian forecasting. Journal of the American Statistical Association 80(389), 73–83 (1985), http://www.jstor.org/stable/2288042 , doi:10.2307/2288042
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Simpson, E., Roberts, S., Psorakis, I., Smith, A. (2013). Dynamic Bayesian Combination of Multiple Imperfect Classifiers. In: Guy, T., Karny, M., Wolpert, D. (eds) Decision Making and Imperfection. Studies in Computational Intelligence, vol 474. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36406-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-36406-8_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-36405-1
Online ISBN: 978-3-642-36406-8
eBook Packages: EngineeringEngineering (R0)