Send VPC flow logs to Sumologic


VPC flow is a AWS feature that allows you to capture the inbound and outbound traffics of all network interfaces in your VPC. To get the information,  you have to go to the Cloudwatch to read the raw logs line by line which is not really efficient. If you happen to use Sumologic, you can send VPC flow logs to it, then define some queries to create a nice dashboard.

This the one that I setup for my VPC. It shows the top 10 source ip by hits or size, top 10 destination ip by hits or size, top accepted or rejected ip…

Selection_001

 
Here are the how-to steps:

  1. Enable VPC flow, please refer AWS doc.
  2. Setup a collector in your VPC to get the logs then send to Sumologic, please refer Sumologic doc.
  3. Setup the dashboard, here are some sample of the queries:

Top 10 source ip by hits (excluding local ip e.g 172.31.*)

_sourceCategory=vpc <your_vpc_log_group>
| json "message","logStream","logGroup"
| parse field=message "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status
| where !(src_ip matches "172.31.*")
| count as Records by src_ip | top 10 src_ip by Records

Top 10 accepted source ip from external that hit network interface eni-1234abcd or eni-5678efgh

_sourceCategory=vpc <your_vpc_log_group> ACCEPT
| json "message","logStream","logGroup"
| parse field=message "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status
| where !(src_ip matches "172.31.*") and interfaceid in ("eni-1234abcd", "eni-5678efgh")
| count as Records by src_ip | top 20 src_ip by Records

Map

_sourceCategory=vpc <your_vpc_log_group>
| json "message","logStream","logGroup"
| parse field=message "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status
| where !(src_ip matches "172.31.*") and interfaceid in ("eni-1234abcd", "eni-5678efgh")
| lookup latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code from geo://default on ip = src_ip
| count by latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code

*Note*: The private IP address of ELB can not be found in the AWS console, you have to run CLI to get it. Here is the command:

aws ec2 describe-network-interfaces --filters "Name=description,Values=ELB*" |  jq -r '.NetworkInterfaces[] | "NAME: \(.Description) INTERFACE: \(.NetworkInterfaceId) IP: \(.PrivateIpAddress)"'
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s