How to assess and report the performance of a stochastic algorithm on a benchmark problem: mean or best result on a number of runs? | Optimization Letters
Skip to main content
Springer Nature Link
Log in

How to assess and report the performance of a stochastic algorithm on a benchmark problem: mean or best result on a number of runs?

  • Short Communication
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

Some authors claim that reporting the best result obtained by a stochastic algorithm in a number of runs is more meaningful than reporting some central statistic. In this short note, we analyze and refute the main argument brought in favor of this statement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Eiben, A.E., Jelasity, M.: A critical note on experimental research methodology in EC. In: Proceedings of the 2002 Congress on Evolutionary Computation (CEC’2002), pp. 582–587. IEEE, New York (2002)

  2. Eiben A.E., Smith J.E. (2003). Introduction to Evolutionary Computing. Springer, Berlin Heidelberg New York

    MATH  Google Scholar 

  3. Fleming P.J., Wallace J.J. (1986). How not to lie with statistics: the correct way to summarize benchmark results. Commun. ACM 29(3):218–221

    Article  Google Scholar 

  4. Barr R.S., Golden B.L., Kelly J.P., Resende M.G.C., Stewart W.R. (1995). Designing and reporting computational experiments with heuristic methods. J. Heuristics 1(1):9–32

    Article  MATH  Google Scholar 

  5. Hooker J.N. (1995). Testing heuristics: We have it all wrong. J. Heuristics 1(1):33–42

    Article  MATH  MathSciNet  Google Scholar 

  6. Rardin R.R., Uzsoy R. (2001). Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7(2):261–304

    Article  MATH  Google Scholar 

  7. Stützle, T.G.: Local Search Algorithms for Combinatorial Problems – Analysis, Algorithms, and New Applications. PhD Thesis, Technische Universität Darmstadt (1999)

  8. Glover F., Kochenberger G. (eds) (2002). Handbook of Metaheuristics. Kluwer, Norwell

    Google Scholar 

  9. Hoos H.H., Stützle T. (2004). Stochastic Local Search. Foundations and Applications. Morgan Kaufmann, San Francisco

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IRIDIA-CoDE, Université Libre de Bruxelles, Brussels, Belgium

    Mauro Birattari & Marco Dorigo

Authors
  1. Mauro Birattari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Dorigo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mauro Birattari.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Birattari, M., Dorigo, M. How to assess and report the performance of a stochastic algorithm on a benchmark problem: mean or best result on a number of runs?. Optimization Letters 1, 309–311 (2007). https://doi.org/10.1007/s11590-006-0011-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-006-0011-8

Keywords

Search

Navigation