FastMPJ: a scalable and efficient Java message-passing library | Cluster Computing Skip to main content
Springer Nature Link
Log in

FastMPJ: a scalable and efficient Java message-passing library

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The performance and scalability of communications are key for high performance computing (HPC) applications in the current multi-core era. Despite the significant benefits (e.g., productivity, portability, multithreading) of Java for parallel programming, its poor communications support has hindered its adoption in the HPC community. This paper presents FastMPJ, an efficient message-passing in Java (MPJ) library, boosting Java for HPC by: (1) providing high-performance shared memory communications using Java threads; (2) taking full advantage of high-speed cluster networks (e.g., InfiniBand) to provide low-latency and high bandwidth communications; (3) including a scalable collective library with topology aware primitives, automatically selected at runtime; (4) avoiding Java data buffering overheads through zero-copy protocols; and (5) implementing the most widely extended MPI-like Java bindings for a highly productive development. The comprehensive performance evaluation on representative testbeds (InfiniBand, 10 Gigabit Ethernet, Myrinet, and shared memory systems) has shown that FastMPJ communication primitives rival native MPI implementations, significantly improving the efficiency and scalability of Java HPC parallel applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Taboada, G.L., Ramos, S., Expósito, R.R., Touriño, J., Doallo, R.: Java in the high performance computing arena: research, practice and experience. Sci. Comput. Program. 78(5), 425–444 (2013)

    Article  Google Scholar 

  2. Blount, B., Chatterjee, S.: An evaluation of Java for numerical computing. Sci. Program. 7(2), 97–110 (1999)

    Google Scholar 

  3. Shafi, A., Carpenter, B., Baker, M., Hussain, A.: A comparative study of Java and C performance in two large-scale parallel applications. Concurr. Comput. Pract. Exp. 21(15), 1882–1906 (2009)

    Article  Google Scholar 

  4. Suganuma, T., Ogasawara, T., Takeuchi, M., Yasue, T., Kawahito, M., Ishizaki, K., Komatsu, H., Nakatani, T.: Overview of the IBM Java just-in-time compiler. IBM Syst. J. 39(1), 175–193 (2000)

    Article  Google Scholar 

  5. Message Passing Interface Forum: MPI: a message passing interface standard. http://www.mcs.anl.gov/research/projects/mpi/ (1995). Accessed Oct 2013

  6. IBTA: The InfiniBand Trade Association. http://www.infinibandta.org/. Accessed Oct 2013

  7. Carpenter, B., Getov, V., Judd, G., Skjellum, A., Fox, G.: MPJ: MPI-like message passing for Java. Concurr. Comput.: Pract. Exp. 12(11), 1019–1038 (2000)

    MATH  Google Scholar 

  8. Carpenter, B., Fox, G., Ko, S.H., Lim, S.: mpiJava 1.2: API specification. http://www.hpjava.org/reports/mpiJava-spec/mpiJava-spec/mpiJava-spec.html. Accessed Oct 2013

  9. Java Grande Forum. http://www.javagrande.org. Accessed Oct 2013

  10. Hongwei, Z., Wan, H., Jizhong, H., Jin, H., Lisheng, Z.: A performance study of Java communication stacks over InfiniBand and Gigabit Ethernet. In: Proceedings of the 4th IFIP International Conference on Network and Parallel Computing—Workshops (NPC’07), pp. 602–607. Dalian, China (2007)

  11. Thiruvathukal, G.K., Dickens, P.M., Bhatti, S.: Java on networks of workstations (JavaNOW): a parallel computing framework inspired by Linda and the message passing interface (MPI). Concurr. Comput.: Pract. Exp. 12(11), 1093–1116 (2000)

    MATH  Google Scholar 

  12. Baker, M., Carpenter, B., Fox, G., Ko, S.H., Lim, S.: mpiJava: an object-oriented Java interface to MPI. In: Proceedings of the 1st International Workshop on Java for Parallel and Distributed Computing (IWJPDC’99), pp. 748–762. San Juan, Puerto Rico (1999)

  13. Baker, M., Carpenter, B., Shafi, A.: MPJ express: towards thread safe Java HPC. In: Proceedings of 8th IEEE International Conference on Cluster Computing (CLUSTER’06), pp. 1–10. Barcelona, Spain (2006)

  14. Shafi, A., Manzoor, J., Hameed, K., Carpenter, B., Baker, M.: Multicore-enabling the MPJ express messaging library. In: Proceedings of 8th International Conference on the Principles and Practice of Programming in Java (PPPJ’10), pp. 49–58. Vienna, Austria (2010)

  15. Baker, M., Carpenter, B., Shafi, A.: A buffering layer to support derived types and proprietary networks for Java HPC. Scalable Comput. Pract. Exp. 8(4), 343–358 (2007)

    Google Scholar 

  16. Bornemann, M., van Nieuwpoort, R.V., Kielmann, T.: MPJ/Ibis: a flexible and efficient message passing platform for Java. In: Proceedings of 12th European PVM/MPI Users’ Group Meeting (EuroPVM/MPI’05), pp. 217–224. Sorrento, Italy (2005)

  17. van Nieuwpoort, R.V., Maassen, J., Wrzesinska, G., Hofman, R., Jacobs, C., Kielmann, T., Bal, H.E.: Ibis: a flexible and efficient Java-based grid programming environment. Concurr. Comput.: Pract. Exp. 17(7–8), 1079–1107 (2005)

    Article  Google Scholar 

  18. Myrinet Express (MX): A High Performance, Low-level, Message-Passing Interface for Myrinet, version 1.2, (2006)

  19. Taboada, G.L., Touriño, J., Doallo, R.: F-MPJ: scalable Java message-passing communications on parallel systems. J. Supercomput. 60(1), 117–140 (2012)

    Article  Google Scholar 

  20. Taboada, G.L., Touriño, J., Doallo, R.: Java fast sockets: enabling high-speed Java communications on high performance clusters. Comput. Commun. 31(17), 4049–4059 (2008)

    Article  Google Scholar 

  21. Taboada, G.L., Touriño, J., Doallo, R., Shafi, A., Baker, M., Carpenter, B.: Device level communication libraries for high-performance computing in Java. Concurr. Comput.: Pract. Exp. 23(18), 2382–2403 (2011)

    Article  Google Scholar 

  22. Bonachea, D., Dickens, P.M., Thakur, R.: High-performance file I/O in Java: existing approaches and bulk I/O extensions. Concurr. Comput.: Pract. Exp. 13(8–9), 713–736 (2001)

    Google Scholar 

  23. Dickens, P.M., Thakur, R.: An evaluation of Java’s I/O capabilities for high-performance computing. In: Proceedings of 1st ACM Java Grande Conference (JAVA’00), pp. 26–35. San Francisco, CA, USA (2000)

  24. Baker, M., Carpenter, B., Shafi, A.: A pluggable architecture for high-performance Java messaging. IEEE Distrib. Syst. Online 6(10), 1–4 (2005)

    Article  Google Scholar 

  25. Goglin, B.: High-performance message passing over generic Ethernet hardware with Open-MX. Parallel Comput. 37(2), 85–100 (2011)

    Google Scholar 

  26. Goglin, B.: High throughput intra-node MPI communication with Open-MX. In: Proceedings of 17th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP’09), pp. 173–180. Weimar, Germany (2009)

  27. OpenFabrics Alliance: http://www.openfabrics.org/. Accessed Oct 2013

  28. IETF RFC 4392: IP over InfiniBand (IPoIB) Architecture. http://www.ietf.org/rfc/rfc4392.txt.pdf. Accessed Oct 2013

  29. Expósito, R.R., Taboada, G.L., Touriño, J., Doallo, R.: Design of scalable Java message-passing communications over InfiniBand. J. Supercomput. 61(1), 141–165 (2012)

    Article  Google Scholar 

  30. Ramos, S., Taboada, G.L., Expósito, R.R., Touriño, J., Doallo, R.: Design of scalable Java communication middleware for multi-core systems. Comput. J. 56(2), 214–228 (2013)

    Article  Google Scholar 

  31. Chan, E., Heimlich, M., Purkayastha, A., van de Geijn, R.A.: Collective communication: theory, practice, and experience. Concurr. Comput.: Pract. Exp. 19(13), 1749–1783 (2007)

    Article  Google Scholar 

  32. Open MPI: Open Source High Performance Computing. http://www.open-mpi.org/. Accessed Oct 2013

  33. MVAPICH: MPI over InfiniBand, 10GigE/iWARP and RoCE. http://mvapich.cse.ohio-state.edu/. Accessed Oct 2013

  34. Portable MPI Model Implementation over MX. https://www.myricom.com/support/downloads/mx/mpich-mx.html. Accessed Oct 2013

  35. NAS Parallel Benchmarks. http://www.nas.nasa.gov/publications/npb.html. Accessed Oct 2013

  36. Bailey, D.H., et al.: The NAS parallel benchmarks. Int. J. High Perform. Comput. Appl. 5(3), 63–73 (1991)

    Article  Google Scholar 

  37. Advanced School for Computing and Imaging (ASCI): Distributed ASCI Supercomputer, Version 4 (DAS-4). http://www.cs.vu.nl/das4/. Accessed Oct 2013

  38. MareNostrum supercomputer in TOP500 List. http://www.top500.org/system/8242. Accessed Oct 2013

  39. TOP500 Org.: Top 500 Supercomputer Sites. http://www.top500.org/ Accessed Oct 2013

  40. Saini, S., et al.: Performance evaluation of supercomputers using HPCC and IMB benchmarks. J. Comput. Syst. Sci. 74(6), 965–982 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  41. Mallón, D.A., Taboada, G.L., Touriño, J., Doallo, R.: NPB-MPJ: NAS parallel benchmarks implementation for message-passing in Java. In: Proceedings of 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP’09), pp. 181–190. Weimar, Germany (2009)

Download references

Acknowledgments

This work has been funded by the Ministry of Education of Spain (FPU Grant AP2010-4348), the Ministry of Economy and Competitiviness (project TIN2010-16735) and the Galician Government (projects CN2012/211 and GRC2013/055), partially supported by FEDER funds. We thankfully acknowledge the computer resources, technical expertise and assistance provided by the Barcelona Supercomputing Center. We also gratefully thank the Advanced School for Computing and Imaging (ASCI) and the Vrije University Amsterdam for providing access to the DAS-4 cluster.

Author information

Authors and Affiliations

  1. Computer Architecture Group, Department of Electronics and Systems, University of A Coruña, La Coruña, Spain

    Roberto R. Expósito, Sabela Ramos, Guillermo L. Taboada, Juan Touriño & Ramón Doallo

Authors
  1. Roberto R. Expósito
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sabela Ramos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guillermo L. Taboada
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juan Touriño
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ramón Doallo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roberto R. Expósito.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Expósito, R.R., Ramos, S., Taboada, G.L. et al. FastMPJ: a scalable and efficient Java message-passing library. Cluster Comput 17, 1031–1050 (2014). https://doi.org/10.1007/s10586-014-0345-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-014-0345-4

Keywords

Search

Navigation