NetStore: An efficient storage infrastructure for network forensics and monitoring

Paul Giura, Nasir Memon

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the increasing sophistication of attacks, there is a need for network security monitoring systems that store and examine very large amounts of historical network flow data. An efficient storage infrastructure should provide both high insertion rates and fast data access. Traditional row-oriented Relational Database Management Systems (RDBMS) provide satisfactory query performance for network flow data collected only over a period of several hours. In many cases, such as the detection of sophisticated coordinated attacks, it is crucial to query days, weeks or even months worth of disk resident historical data rapidly. For such monitoring and forensics queries, row oriented databases become I/O bound due to long disk access times. Furthermore, their data insertion rate is proportional to the number of indexes used, and query processing time is increased when it is necessary to load unused attributes along with the used ones. To overcome these problems we propose a new column oriented storage infrastructure for network flow records, called NetStore. NetStore is aware of network data semantics and access patterns, and benefits from the simple column oriented layout without the need to meet general purpose RDBMS requirements. The prototype implementation of NetStore can potentially achieve more than ten times query speedup and ninety times less storage size compared to traditional row-stores, while it performs better than existing open source column-stores for network flow data.

Original languageEnglish (US)
Title of host publicationRecent Advances in Intrusion Detection - 13th International Symposium, RAID 2010, Proceedings
Pages277-296
Number of pages20
DOIs
StatePublished - 2010
Event13th International Symposium on Recent Advances in Intrusion Detection Systems, RAID 2010 - Ottawa, ON, Canada
Duration: Sep 15 2010Sep 17 2010

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume6307 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other13th International Symposium on Recent Advances in Intrusion Detection Systems, RAID 2010
CountryCanada
CityOttawa, ON
Period9/15/109/17/10

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'NetStore: An efficient storage infrastructure for network forensics and monitoring'. Together they form a unique fingerprint.

Cite this