Machine learning on NetFlow/IPFIX with ELK for free

Abstract

The ELK (Elasitsearch, Logstash and Kibana) stack provide also a Machine Learning engine, included in the X-Pack plugin. Let's see how to install it and how to detect anomalies in our network using NetFlow/IPFIX.

Install X-Pack for a RPM installation

# /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
# service elasticsearch restart
# /usr/share/logstash/bin/logstash-plugin install x-pack
# service logstash restart
# /usr/share/kibana/bin/kibana-plugin install x-pack
# service kibana restart

Login to Kibana usint elastic username with password changeme. If you cannot login to Kibana (Unknown provider: sessionTimeoutProvider), stop Kibana, be sure no Kibana process is running, than start Kibana again (oro just reboot your server).

Remember you have to append username and password for every command:

# curl -XGET 'http://elastic:changeme@localhost:9200/_cat/indices?v'

And also update the Logstash configuration:

output {
  if "port_9996" in [tags] {
    elasticsearch {
        hosts => "127.0.0.1"
        index => "logstash-netflow-9996-%{+YYYY.MM.dd}"
        user => "elastic"
        password => "changeme"
    }
  } else if "port_9995" in [tags] {
    elasticsearch {
        hosts => "127.0.0.1"
        index => "logstash-netflow-9995-%{+YYYY.MM.dd}"
        user => "elastic"
        password => "changeme"
    }
  }
}

Analyze incoming HTTP/HTTPS requests (rate)

This job will analyze the HTTP and HTTPS request rate:

Analyze incoming HTTP/HTTPS requests (rate)

The graph evidences peaks.

Open Kibana and go trough Machine Learning -> Create new job -> Create an advanced job:

  • set logstash-netflow-9996-* as input index;
  • set web-requests as name;
  • under “Analysis Configuration” add a detector:
    • function count;
  • set the following query under datafeed:
{
	"bool": {
		"should": [{
				"match": {
					"netflow.l4_dst_port": {
						"query": 80,
						"operator": "OR"
					}
				}
			},
			{
				"match": {
					"netflow.l4_dst_port": {
						"query": 443,
						"operator": "OR"
					}
				}
			}
		]
	}
}

Analyze DNS requests (size and rate)

This job will analyze the DNS rate and size:

Analyze DNS requests (size and rate)

The above graph analyzes the size of each DNS requests.

Open Kibana and go trough Machine Learning -> Create new job -> Create an advanced job:

  • set logstash-netflow-9996-* as input index;
  • set dns-requests as name;
  • under “Analysis Configuration” add two detectors:
    • function count;
    • function mean, field_name netflow.in_bytes;
  • set the following query under datafeed:
{
	"match": {
		"netflow.l4_dst_port": 53
	}
}

Vew and compare anomalies

Different jobs can be compared int a single timeline. Go to Machine Learning -> Anomaly Explorer and select all required jobs:

Anomaly timeline

Posted on 19 Jul 2017 by Andrea.
  • Gmail icon
  • Twitter icon
  • Facebook icon
  • LinkedIN icon
  • Google+ icon