Evolution of human-competitive lossless compression algorithms with GP-zip2 | Genetic Programming and Evolvable Machines Skip to main content
Springer Nature Link
Log in

Evolution of human-competitive lossless compression algorithms with GP-zip2

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

We propose GP-zip2, a new approach to lossless data compression based on Genetic Programming (GP). GP is used to optimally combine well-known lossless compression algorithms to maximise data compression. GP-zip2 evolves programs with multiple components. One component analyses statistical features extracted by sequentially scanning the data to be compressed and divides the data into blocks. These blocks are projected onto a two-dimensional Euclidean space via two further (evolved) program components. K-means clustering is then applied to group similar data blocks. Each cluster is labelled with the optimal compression algorithm for its member blocks. After evolution, evolved programs can be used to compress unseen data. The compression algorithms available to GP-zip2 are: Arithmetic coding, Lempel-Ziv-Welch, Unbounded Prediction by Partial Matching, Run Length Encoding, and Bzip2. Experimentation shows that the results produced by GP-zip2 are human-competitive, being typically superior to well-established human-designed compression algorithms in terms of the compression ratios achieved in heterogeneous archive files.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. To be fair, in the bzip2 algorithm some regularities may be captured in particularly large files as a byproduct of its splitting files into large blocks for the purpose of keeping its memory footprint under control. For example, at the highest compression setting bzip2 splits files into 900 KB blocks. However, each block is processed via exactly the same stages of analysis and compression.

  2. One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

References

  1. Z. Arnavut, in Move-to-Front and Inversion Coding. Data Compression Conference, 2000. Proceedings. DCC 2000, pp. 193–202.

  2. R. Arnold, T. Bell, in A Corpus for the Evaluation of Lossless Compression Algorithms. Data Compression Conference, 1997. DCC’97. Proceedings, pp. 201–210

  3. J. Bezdek, N. Pal, Some new indexes of cluster validity. IEEE Trans. Syst. Man Cybern. B 28(3), 301–315 (1998)

    Article  Google Scholar 

  4. N. Boric, P.A. Estevez, in Genetic Programming-Based Clustering Using an Information Theoretic Fitness Measure, ed. by D. Srinivasan, L. Wang. 2007 IEEE Congress on Evolutionary Computation, Singapore, 25–28 Sept 2007 (IEEE Computational Intelligence Society, IEEE Press), pp. 31–38

  5. M. Burrows, D. Wheeler, A block-sorting lossless data compression algorithm. Digital SRC Research Report, 1994

  6. J. Cleary, I. Witten, Data compression using adaptive coding and partial string matching. IEEE Trans. Commun. 32(4), 396–402 (1984)

    Article  Google Scholar 

  7. J.G. Cleary, W.J. Teahan, I.H. Witten, in Unbounded length Contexts for PPM. Data Compression Conference (1995), pp. 52–61

  8. G.V. Cormack, R.N. Horspool, Data compression using dynamic markov modelling. Comput. J. 30(6), 541–550 (1987)

    MathSciNet  Google Scholar 

  9. A. Fukunaga, A. Stechert, in Evolving Nonlinear Predictive Models for Lossless Image Compression with Ggenetic Programming, ed. by J.R. Koza, W. Banzhaf, K. Chellapilla, K. Deb, M. Dorigo, D.B. Fogel, M.H. Garzon, D.E. Goldberg, H. Iba, R. Riolo. Genetic Programming 1998: Proceedings of the Third Annual Conference, University of Wisconsin, Madison, Wisconsin, USA, 22–25 July 1998 (Morgan Kaufmann), pp. 95–102.

  10. S.W. Golomb, Run-length encodings. IEEE Trans. Inf. Theory IT-12, 399–401 (1966)

    Article  MathSciNet  Google Scholar 

  11. T. Haynes, S. Sen, D. Schoenefeld, R. Wainwright, in Evolving a team, ed. by E.V. Siegel, J.R. Koza. Working Notes for the AAAI Symposium on Genetic Programming, MIT, Cambridge, MA, USA, 10–12 Nov 1995 (AAAI), pp. 23–30

  12. J. He, X. Wang, M. Zhang, J. Wang, Q. Fang, in New Research on Scalability of Lossless Image Compression by GP Engine, ed. by J. Lohn, D. Gwaltney, G. Hornby, R. Zebulum, D. Keymeulen, A. Stoica. Proceedings of the 2005 NASA/DoD Conference on Evolvable Hardware, Washington, DC, USA, 29 June–1 July 2005 (IEEE Press), pp. 160–164

  13. W.H. Hsu, A.E. Zwarico, Automatic synthesis of compression techniques for heterogeneous files. Softw. Pract. Exper. 25(10), 1097–1116 (1995)

    Article  Google Scholar 

  14. D.A. Huffman, A method for the construction of minimum-redundancy codes. Proc. Inst. Radio Eng. 40(9), 1098–1101 (1952)

    Google Scholar 

  15. A. Kattan, Universal Lossless Data Compression with Built in Encryption. Master’s thesis, School of Computer Science and Electronic Engineering, University of Essex, 2006

  16. A. Kattan, Evolutionary Synthesis of Lossless Compression Algorithms: The GP-zip Family. PhD thesis, School of Computer Science and Electronic Engineering, University of Essex, Oct 2010

  17. A. Kattan, R. Poli, in Evolutionary Lossless Compression with GP-ZIP, ed. by J. Wang. 2008 IEEE World Congress on Computational Intelligence, Hong Kong, 1–6 June 2008 (IEEE Computational Intelligence Society, IEEE Press)

  18. A. Kattan, R. Poli, Evolutionary Lossless Compression with GP-ZIP*, ed. by M. Keijzer, G. Antoniol, C.B. Congdon, K. Deb, B. Doerr, N. Hansen, J.H. Holmes, G.S. Hornby, D. Howard, J. Kennedy, S. Kumar, F.G. Lobo, J.F. Miller, J. Moore, F. Neumann, M. Pelikan, J. Pollack, K. Sastry, K. Stanley, A. Stoica, E.-G. Talbi, I. Wegener. GECCO ’08: Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation, Atlanta, GA, USA, 12–16 July 2008 (ACM), pp. 1211–1218

  19. A. Klappenecker, F.U. May, Evolving better wavelet compression schemes. in Wavelet Applications in Signal and Image Processing III, ed. by A.F. Laine, M.A. Unser, M.V. Wickerhauser, vol. 2569, San Diego, CA, USA, 9–14 July 1995 (SPIE)

  20. J. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection (The MIT press, Cambridge Massachusetts, 1992)

    MATH  Google Scholar 

  21. J.R. Koza, Genetic Programming II: Automatic Discovery of Reusable Programs (MIT Press, Cambridge Massachusetts, 1994), ISBN 0-262-11189-6

    MATH  Google Scholar 

  22. J.R. Koza, M.A. Keane, M.J. Streeter, W. Mydlowec, J. Yu, G. Lanza, Genetic Programming IV: Routine Human-Competitive Machine Intelligence (Kluwer Academic Publishers, Netherlands-Dordrecht, 2003)

    MATH  Google Scholar 

  23. T. Krantz, O. Lindberg, G. Thorburn, P. Nordin, in Programmatic Compression of Natural Video, ed. by E. Cantú-Paz. Late Breaking Papers at the Genetic and Evolutionary Computation Conference (GECCO-2002), New York, NY, July 2002 (AAAI)

  24. T. Kuthan, J. Lansky, in Genetic Algorithms in Syllable-Based Text Compression, ed. by J. Pokorný, V. Snásel, K. Richta. DATESO CEUR Workshop Proceedings, vol. 235. CEUR-WS.org, 2007

  25. G.G. Langdon, Arithmetic coding. IBM J. Res. Dev. 23, 149–162 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  26. S. Luke, L. Spector, in Evolving Teamwork and Ccoordination with Genetic Programming, ed. by J.R. Koza, D.E. Goldberg, D.B. Fogel, R.L. Riolo. Genetic Programming 1996: Proceedings of the First Annual Conference, Stanford University, CA, USA, 28–31 July 1996 (MIT Press), pp. 150–156

  27. E. Lutton, J. Levy-Vehel, G. Cretin, P. Glevarec, C. Roll, Mixed IFS: Resolution of the inverse problem using genetic programming. Complex Syst. 9, 375–398 (1995)

    MathSciNet  MATH  Google Scholar 

  28. E. Lutton, J. Levy-Vehel, G. Cretin, P. Glevarec, C. Roll, Mixed IFS: resolution of the inverse problem using genetic programming. Research Report No 2631, Inria, 1995

  29. D.P. Muni, N.R. Pal, J. Das, A novel approach to design classifier using genetic programming. IEEE Trans. Evol. Comput. 8(2), 183–196 (2004)

    Article  Google Scholar 

  30. P. Nordin, W. Banzhaf, in Programmatic compression of images and sound, ed. by J.R. Koza, D.E. Goldberg, D.B. Fogel, R.L. Riolo. Genetic Programming 1996: Proceedings of the First Annual Conference, Stanford University, CA, USA, 28–31 July 1996 (MIT Press), pp. 345–350

  31. F. Oroumchian, E. Darrudi, F. Taghiyareh, N. Angoshtari, in Experiments with Persian Text Compression for Web. WWW Alt. ’04: Proceedings of the 13th International World Wide Web Conference on Alternate Track Papers & Posters, New York, NY, USA, 2004 (ACM), pp. 478–479

  32. J. Parent, A. Nowe, in Evolving Compression Preprocessors with Genetic Programming, ed. by W.B. Langdon, E. Cantú-Paz, K. Mathias, R. Roy, D. Davis, R. Poli, K. Balakrishnan, V. Honavar, G. Rudolph, J. Wegener, L. Bull, M. A. Potter, A. C. Schultz, J. F. Miller, E. Burke, N. Jonoska. GECCO 2002: Proceedings of the Genetic and Evolutionary Computation Conference, New York, 9–13 July 2002 (Morgan Kaufmann Publishers), pp. 861–867

  33. I.M. Pu, Fundamental Data Compression (Butterworth-Heinemann, Newton, MA, USA, 2005)

    Google Scholar 

  34. D. Salomon, Data Compression: The Complete Reference, 2nd ed. (Springer-Verlag New York Inc, 2004)

  35. A. Sarafopoulos, in Automatic Generation of Affine IFS and Strongly Typed Genetic Programming, ed. by R. Poli, P. Nordin, W. B. Langdon, T.C. Fogarty. Genetic Programming, Proceedings of EuroGP’99 LNCS, vol. 1598, Goteborg, Sweden, 26–27 May 1999 (Springer), pp. 149–160

  36. K. Sayood, Introduction to Data Compression, 2nd ed. (Morgan Kaufmann Publishers, San Francisco, CA, USA, 2000)

    Google Scholar 

  37. J. Seward, Bzip2. Website, Nov 2009 http://www.bzip.org

  38. G. Ucoluk, I.H. Toroslu, A genetic algorithm approach for verification of the syllable based text compression technique. J. Inf. Sci. 23(5), 365–372 (1997)

    Article  Google Scholar 

  39. J.S. Vitter, Design and analysis of dynamic huffman codes. J. ACM 34(4), 825–845 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  40. T. Welch, A technique for high-performance data compression. IEEE Comput. 17(6), 8–19 (1984)

    Google Scholar 

  41. WinZip, The compression utility for windows. Website, Nov 2009 http://www.winzip.com

  42. I. Witten, R. Neal, J. Cleary, Arithmetic coding for data compression (1987)

  43. M.J. Zaki, M. Sayed, The use of genetic programming for adaptive text compression. Int. J. Inf. Coding Theory 1(1), 88–108 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  44. J. Ziv, A. Lempel, A universal algorithm for sequential data compression. IEEE Trans. Inf. Theory 23(3), 337–343 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  45. J. Ziv, A. Lempel, Compression of individual sequences via variable-rate coding. IEEE Trans. Inf. Theory 24(5), 530–536 (1978)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the editor-in-chief and the anonymous reviewers for their thoughtful, constructive and supportive comments. These have helped us enormously in improving this paper, including uncovering and correcting an important mistake.

Author information

Authors and Affiliations

  1. School of Computer Science and Electronic Engineering, University of Essex, Colchester, CO4 3SQ, UK

    Ahmed Kattan & Riccardo Poli

Authors
  1. Ahmed Kattan
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Riccardo Poli
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ahmed Kattan.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kattan, A., Poli, R. Evolution of human-competitive lossless compression algorithms with GP-zip2. Genet Program Evolvable Mach 12, 335–364 (2011). https://doi.org/10.1007/s10710-011-9133-6

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-011-9133-6

Keywords