A Modular Traffic Sampling Architecture: Bringing Versatility and Efficiency to Massive Traffic Analysis | Journal of Network and Systems Management Skip to main content
Springer Nature Link
Log in

A Modular Traffic Sampling Architecture: Bringing Versatility and Efficiency to Massive Traffic Analysis

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

The massive traffic volumes and heterogeneity of services in today’s networks urge for flexible, yet simple measurement solutions to assist network management tasks, without impairing network performance. To turn treatable tasks requiring traffic analysis, sampling the traffic has become mandatory, triggering substantial research in the area. Despite that, there is still a lack of an encompassing solution able to support the flexible deployment of sampling techniques in production networks, adequate to diverse traffic scenarios and measurement activities. In this context, this article proposes a modular traffic sampling architecture able to foster the flexible design and deployment of efficient measurement strategies. The architecture is composed of three layers—management plane, control plane and data plane—covering key components to achieve versatile and lightweight measurements in diverse traffic scenarios and measurement activities. Each component of the architecture is described considering the different strategies, technologies and protocols that compose the several stages of a measurement process. Following the proposed architecture, a sampling framework prototype has been developed, providing a fair environment to assess and compare sampling techniques under distinct measurement scenarios, evaluating their performance in balancing computational burden and accuracy. The results have demonstrated the relevance and applicability of the proposed architecture, revealing that a modular and configurable approach to sampling is a step forward for improving sampling scope and efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. The framework is available for download at http://1drv.ms/1IggkCa as a Raspbian image ready to be deployed.

  2. Note that the evaluation of flow classification methodologies and tools is beyond the scope of this work, which resorts to a port-based classification technique for distinguishing flows.

References

  1. Zseby, T., Molina, M., Duffield, N.: Sampling and Filtering Techniques for IP Packet Selection RFC 5475. Technical report, IETF. http://datatracker.ietf.org/doc/rfc5475/ (2009)

  2. Silva, J.M.C., Carvalho, P., Rito Lima, S.: Analysing traffic flows through sampling: a comparative study. In: 20th IEEE Symposium on Computers and Communication (ISCC), Cyprus (2015)

  3. Jadwab, J., Phall, P., Pinna, B.: Traffic estimation for the largest sources on a network using packet sampling with limited storage. Technical report, Hewllet-Packard Laboratories, Bristol (1992)

  4. Claffy, K.C., Polyzos, G.C., Braun, H.W.: Application of sampling methodologies to network traffic characterization, SIGCOMM. Comput. Commun. Rev. 23(4), 194–203 (1993). doi:10.1145/167954.166256

    Article  Google Scholar 

  5. Cozzani, I., Giordano, S.: Traffic sampling methods for end-to-end QoS evaluation in large heterogeneous networks. Comput. Netw. ISDN Syst. 30(16–18), 1697–1706. http://www.sciencedirect.com/science/article/pii/S0169755298001986 (1998)

  6. Amer, P., Cassel, L.: Management of sampled real-time network measurements. In: Proceedings of 14th Conference on Local Computer Networks. IEEE Comput. Soc. Press, pp. 62–68. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=65244 (1989)

  7. Tammaro, D., Valenti, S., Rossi, D., Pescapé, A.: Exploiting packet-sampling measurements for traffic characterization and classification. Int. J. Netw. Manag. 22(6), 451–476 (2012). doi:10.1002/nem.1802

    Article  Google Scholar 

  8. Duffield, N.: Fair sampling across network flow measurements. ACM SIGMETRICS Perform. Eval. Rev. 40(1), 367 (2012). doi:http://dl.acm.org/citation.cfm?id=2318857.2254800

  9. Hernandez, E.A., Chidester, M.C., George, A.D.: Adaptive sampling for network management. J. Netw. Syst. Manag. 9(4), 409–434 (2001). doi:10.1023/A:1012980307500

    Article  Google Scholar 

  10. Silva, J.M.C., Carvalho, P., Rito Lima, S.: A multiadaptive sampling technique for cost-effective network measurements. Comput. Netw. 57(17), 3357–3369 (2013). doi:10.1016/j.comnet.2013.07.023

  11. Duffield, N.G., Grossglauser, M.: Trajectory sampling for direct traffic observation. ACM SIGCOMM Comput. Commun. Rev. 30(4), 271–282 (2000). doi:10.1145/347057.347555

    Article  Google Scholar 

  12. Estan, C., Varghese, G.: New directions in traffic measurement and accounting. SIGCOMM Comput. Commun. Rev. 32(4), 323–336 (2002). doi:10.1145/964725.633056

    Article  Google Scholar 

  13. Singh, R., Kumar, H., Singla, R.K.: Analyzing statistical effect of sampling on network traffic dataset. In: Satapathy, S.C., Avadhani, P.S., Udgata, S.K., Lakshminarayana, S. (eds.). ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India. Springer International Publishing, pp. 401–408. http://link.springer.com/chapter/10.1007/978-3-319-03107-1_43 (2014)

  14. Yang, L., Michailidis, G.: Sampled based estimation of network traffic flow characteristics. In: IEEE INFOCOM 2007---26th IEEE International Conference on Computer Communications, (IEEE) pp. 1775–1783. doi:10.1109/INFCOM.2007.207. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4215789 (2007)

  15. Carela-Español, V., Barlet-Ros, P., Cabellos-Aparicio, A., Solé-Pareta, J.: Analysis of the impact of sampling on NetFlow traffic classification. Comput. Netw. 55(5), 1083–1089 (2011). doi:10.1016/j.comnet.2010.11.002

    Article  Google Scholar 

  16. Lin, R., Li, O., Li, Q., Dai, K.: Exploiting adaptive packet-sampling measurements for multimedia traffic classification. J. Commun. 9(12) (2014). http://www.jocm.us/uploadfile/2014/1231/20141231030404520

  17. Kandula, S., Mahajan, R.: Sampling biases in network path measurements and what to do about it. In: Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement Conference IMC ’09 (ACM, New York, NY, USA) , pp. 156–169. doi:10.1145/1644893.1644912 (2009)

  18. Lee, M., Duffield, N., Kompella, R.: Two samples are enough: opportunistic flow-level latency estimation using NetFlow. In: 2010 Proceedings IEEE INFOCOM, pp. 1–9. doi:10.1109/INFCOM.2010.5462044. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5462044 (2010)

  19. Mahmood, A.N., Hu, J., Tari, Z., Leckie, C.: Critical infrastructure protection: resource efficient sampling to improve detection of less frequent patterns in network traffic. J. Netw. Comput. Appl. 33(4), 491–502 (2010). http://www.sciencedirect.com/science/article/B6WKB-4YBMFB6-1/2/9b91d8daa2364e0d025aed6088160da7

  20. Zhang, J., Luo, X., Perdisci, R., Gu, G., Lee, W., Feamster, N.: Boosting the scalability of botnet detection using adaptive traffic sampling. In: Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security, Ser. (ACM, New York, NY, USA), ASIACCS ’11, pp. 124–134. doi:10.1145/1966913.1966930 (2011)

  21. Huang, Y., Pullen, J.: Countering denial-of-service attacks using congestion triggered packet sampling and filtering. In: Proceedings Tenth International Conference on Computer Communications and Networks (Cat. No.01EX495), (IEEE), pp. 490–494 (2001). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=956309

  22. Brauckhoff, D., Tellenbach, B., Wagner, A., May, M., Lakhina, A.: Impact of packet sampling on anomaly detection metrics. In: Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement, Ser. (ACM, New York, NY, USA) IMC ’06, pp. 159–164. doi:10.1145/1177080.1177101 (2006)

  23. Paredes-Oliva, I., Barlet-Ros, P., Solé-Pareta, J.: Portscan detection with sampled Netflow. In: Traffic Monitoring and Analysis (Springer), pp. 26–33. http://link.springer.com/chapter/10.1007/978-3-642-01645-5_4 (2009)

  24. Mai, J., Chuah, C.N., Sridharan, A., Ye, T., Zang, H.: Is sampled data sufficient for anomaly detection? In: Proceedings of the 6th ACM SIGCOMM on Internet measurement—IMC’06, Ser. (ACM Press, New York, NY, USA) p. 165 (2006). http://portal.acm.org/citation.cfm?doid=1177080.1177102

  25. Jae-Hyun, J., Cheol-Woong, A., Dongjoon, L., Sung-Ho, K.: DDoS attack detection using flow entropy and packet sampling on huge networks. In: ICN 2014 : The Thirteenth International Conference on Networks (IARIA), pp. 183–190 (2014)

  26. Zseby, T.: Deployment of sampling methods for SLA validation with non-intrusive measurements. In: Proceedings of Passive and Active Measurements Conference (Fort Collins) (2002)

  27. Zseby, T.: Comparison of sampling methods for non-intrusive SLA validation. In: Proceedings of the Second Workshop on End-to-End Monitoring Techniques and Services (E2EMon) (2004)

  28. Serral-Gracia, R., Cabellos-Aparicio, A., Domingo-Pascual, J.: Packet loss estimation using distributed adaptive sampling. In: Network Operations and Management Symposium Workshops, 2008. NOMS Workshops 2008. IEEE (IEEE), pp. 124–131 (2008). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4509938

  29. Sommers, J., Barford, P., Duffield, N., Ron, A.: Improving accuracy in end-to-end packet loss measurement. In: Proceedings of the 2005 conference on Applications, Technologies, Architectures, and Protocols for Computer Communications—SIGCOMM ’05, (ACM Press, New York, New York, USA), vol. 35, p. 157 (2005). http://dl.acm.org/citation.cfm?id=1080091.1080111

  30. Dogman, A., Saatchi, R., Al-Khayatt, S.: An adaptive statistical sampling technique for computer network traffic. In: 7th International Symposium on Communication Systems Networks and Digital Signal Processing (CSNDSP, 2010), pp. 479–483 (2010)

  31. Gu, Y., Breslau, L., Duffield, N., Sen, S.: On passive one-way loss measurements using sampled flow statistics. In: IEEE INFOCOM 2009—The 28th Conference on Computer Communications (IEEE), pp. 2946–2950 (2009). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5062264

  32. Androulidakis, G., Chatzigiannakis, V., Papavassiliou, S.: Network anomaly detection and classification via opportunistic sampling. IEEE Netw. 23(1), 6–12 (2009). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4804318

  33. Choi, B.Y., Bhattacharyya, S.: Observations on cisco sampled netflow. ACM SIGMETRICS Perform. Eval. Rev. 33(3), p. 18 (2005). http://portal.acm.org/citation.cfm?doid=1111572.1111579

  34. Zseby, T., Hirsch, T., Claise, B.: Packet sampling for flow accounting: challenges and limitations. In: Claypool, M., Uhlig, S. (eds.) Passive and Active Network Measurement, Ser. Lecture Notes in Computer Science, vol. 4979, (Springer Berlin / Heidelberg), pp. 61–71 (2008). doi:10.1007/978-3-540-79232-1_7

  35. Pescape, A., Rossi, D., Tammaro, D., Valenti, S.: On the impact of sampling on traffic monitoring and analysis. In: 2010 22nd International Teletraffic Congress (lTC 22) (IEEE), pp. 1–8. (2010). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5608718

  36. Chabchoub, Y., Fricker, C., Guillemin, F., Robert, P.: Deterministic versus probabilistic packet sampling in the internet. In: Mason, L., Drwiega, T., Yan, J. (eds.) Managing Traffic Performance in Converged Networks, Lecture Notes in Computer Science, vol. 4516, Springer, Berlin, Heidelberg, pp. 678–689 (2007). http://link.springer.com/chapter/10.1007/978-3-540-72990-7_60

  37. Castro, V., Carvalho, P., Lima, S.R., In: A cooperative network monitoring overlay. Smart Spaces and Next Generation Wired/Wireless Networking, Springer, pp. 475–486 (2011). http://link.springer.com/chapter/10.1007/978-3-642-22875-9_43

  38. Schad, J., Dittrich, J., Quiané-Ruiz, J.A.: Runtime measurements in the cloud: observing, analyzing, and reducing variance. Proc. VLDB Endow. 3(1–2), 460–471 (2010). doi:10.14778/1920841.1920902

    Article  Google Scholar 

  39. Pras, A., Schoenwaelder, J.: On the difference between information models and data models—RFC 3444. Technical Report, IETF (2003). https://datatracker.ietf.org/doc/rfc3444/

  40. Claise, B., Trammell, B.: Specification of the IP Flow Information eXport (IPFIX) Protocol for the Exchange of Flow Information. RFC 7011 (2013). http://datatracker.ietf.org/doc/draft-ietf-ipfix-protocol-rfc5101bis/

  41. Claise, B., Trammel, B.: Information Model for IP Flow Information Export (IPFIX)—RFC 7012. Technical Report IETF (2013). https://datatracker.ietf.org/doc/rfc7012/

  42. Dietz, T., Claise, B., Aitken, P., Dressler, F., Carle, G.: Information Model for Packet Sampling Exports. Technical Report, IETF RFC 5477 (2009). https://datatracker.ietf.org/doc/rfc5477/

  43. IP Flow Information Export (IPFIX).: Entities (2015).http://www.iana.org/assignments/ipfix/ipfix.xhtml

  44. Dietz, T., Claise, B., Quittek. J.: Definitions of Managed Objects for Packet Sampling. RFC 6727 (2012). http://datatracker.ietf.org/doc/rfc6727/

  45. Case, J., Mundy, R., Partain, D., Stewart, B.: Introduction and Applicability Statements for Internet-Standard Management Framework—RFC 3410. Technical Report, IETF (2002). https://datatracker.ietf.org/doc/rfc3410/

  46. Aitken, P., Claise, B., McDowall, C., Schoenwaelder, J.: Exporting MIB Variables using the IPFIX Protocol draft-ietf-ipfix-mib-variable-export-09. Technical Report, IETF (2015). https://datatracker.ietf.org/doc/draft-ietf-ipfix-mib-variable-export/

  47. McCloghrie, K., Seligson, J., Reichmeyer, F., Smith, A., Sahita, R.: Structure of policy provisioning information (SPPI)—RFC 3159. Technical Report, IETF (2001). https://datatracker.ietf.org/doc/rfc3159/

  48. Uslar, M., Specht, M., Rohjans, S., Trefke, J., González, J.M.: The Common Information Model CIM: IEC 61968/61970 and 62325—A Practical Introduction to the CIM, vol. 66. Springer, New York (2012)

    Google Scholar 

  49. Silva, J.M.C., Carvalho, P., Rito Lima, S.: Enhancing traffic sampling scope and efficiency. In: 2013 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) (IEEE), pp. 71–72 (2013). http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6562848

  50. Hofstede, R., Celeda, P.,  Trammell, B., Drago, I., Sadre, R., Sperotto, A., Pras, A.: Flow monitoring explained: from packet capture to data analysis With NetFlow and IPFIX. IEEE Commun. Surv. Tutor. 16(4), 2037–206 (2014). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6814316

  51. Claise, B., Johnson, A., Quittek. J.: Packet Sampling (PSAMP) Protocol Specifications. RFC 5476 (2009). http://datatracker.ietf.org/doc/rfc5476/

  52. Orebaugh, A., Ramirez, G., Beale, J.: Wireshark and Ethereal Network Protocol Analyzer Toolkit. Syngress, Rockland (2006)

    Google Scholar 

  53. Jacobson, V., McCanne, S.: Lawrence Berkeley Laboratory, Berkeley, CA (2009)

  54. Alcock, S., Lorier, P., Nelson, R.: ACM SIGCOMM Comput. Commun. Rev. 42(2), 42 (2012). http://dl.acm.org/citation.cfm?doid=2185376.2185382

  55. Silva, J.M.C., Carvalho, P., Lima, S.R.: Computational weight of network traffic sampling techniques. In: 2014 IEEE Symposium on Computers and Communications (ISCC) (IEEE, Madeira, Portugal), pp. 1–6 (2014). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6912467

  56. Shannon, C., Aben, E., Claffy, K., Andersen, D., Brownlee, N.: The CAIDA UCSD Anonymized Internet Traces 2008—equinix-chicago.dirA.20080430-170200.UTC.anon. Downloaded from http://www.caida.org/data/passive/passive_2008_dataset.xml (2008)

  57. Shannon, C., Aben, E., Claffy, K., Andersen, D., Brownlee, N.: The CAIDA UCSD Anonymized Internet Traces 2014—-equinix-chicago.dirA.20140619-131100.UTC.anon. Downloaded from http://www.caida.org/data/passive/passive_2014_dataset.xml (2014)

  58. Krishnan, R., Yong, L., Ghanwani, A., So, N., Khasnabish, B.: Mechanisms for Optimizing Link Aggregation Group (LAG) and Equal-Cost Multipath (ECMP) Component Link Utilization in Networks—RFC 7424. Technical Report, IETF (2015). https://datatracker.ietf.org/doc/rfc7424/

  59. Silverman, B.W.: Density Estimation for Statistics and Data Analysis, vol. 26. CRC Press, Boca Raton (1986)

    Book  MATH  Google Scholar 

Download references

Acknowledgements

This work has been supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/ 00319/2013.

Author information

Authors and Affiliations

  1. Departamento de Informática, Centro Algoritmi, Universidade do Minho, Campus de Gualtar, 4710-057, Braga, Portugal

    João Marco C. Silva, Paulo Carvalho & Solange Rito Lima

Authors
  1. João Marco C. Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paulo Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Solange Rito Lima
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Paulo Carvalho.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Silva, J.M.C., Carvalho, P. & Lima, S.R. A Modular Traffic Sampling Architecture: Bringing Versatility and Efficiency to Massive Traffic Analysis. J Netw Syst Manage 25, 643–668 (2017). https://doi.org/10.1007/s10922-017-9404-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10922-017-9404-5

Keywords

Search

Navigation