AWS security group limits Q&A



Here are a few questions that I asked AWS regarding the security group limits and their answers. Just like to share it with more people here:

Screen Shot 2016-11-24 at 8.59.19 AM.png

1) Q: By default, it is 50 limit for both inbound and outbound (giving 100 rules in total). Is it possible to set a different limit to inbound and outbound. For example, 80 limit for inbound, 20 limit for outbound (still giving a total 100 combined rules).

A: Unfortunately no, inbound and outbound traffic are processed separately, therefore the limit is set for both of them separately, the limits for inbound and outbound rules are always the same, when you update the limit, it applies to both inbound and outbound.

2) Q: Is the limit a global setting? or it can be set on a particular security group?

A: The limit is a regional setting, it will apply to all the security groups in the same region.

3) Q:  If it is global settings, will it impact the existing security groups? For example, I decrease the inbound limit to 30, but there is already a security group with 40 inbound rules. What will happen?

A: Before applying to decrease the inbound limit to 30, you need to make sure you don’t have any security groups at the moment which have more than 30 rules. If you have security groups with 40 rules, the change cannot be made, you’ll be asked to delete/modify the security groups which do not meet the requirement.

4) Q: Each NIC has maximum 250 rules (the multiple of the limit of security group per NIC and the limit of the rules per security group). Is it a global setting as well? If so, will the change impact the existing ones which violate the limits.

A: The maximum 250 rules limit is a global hard limit, exceeding the 250 rules per interface limit can have a negative impact on performance, not only for your instances, but also for any other customers’ instances running on the same underlying hardware. And similar to what has been discussed in question 3, you have to make sure all your existing security groups do not violate the limit after the change before applying for the change on limits. I hope this helps, please do feel free to come back to us at any time if you need further assistance.



Elasticache Redis Unreachable Issue


, ,

We have a Elasticache Redis replication group, it has two nodes: one primary and one replica. Last week, we noticed that the primary redis node suddenly stops working – any connections to the primary node timed out eventually.

According to the log, there was a load burst and following that the redis reboot itself.



Unfortunately, the redis node stops responding after that. The weird thing is the replication between nodes still works. So I promoted the original replica to primary, and login into it. The ‘role’ or ‘info’ commands tells me the replication is working fine, and the slave ip is 10.0.x.x. Ah, that’s interesting as my VPC network is 172.31.x.x. So it means there is something wrong with the instance’s 172.31.x.x NIC. Contacted AWS and their service team restart that NIC, then things are back to normal.

The ironical thing is that the AWS console still shows everything is green while the the 172.31.x.x NIC is not functional. Looks to me that AWS only monitor their internal network (in this case it is 10.0.x.x network). I have submitted a feature request to suggest them to improve the monitoring.

CloudFormation takes 8 hours to complete



I used CloudFormation to restore a RDS snapshot to a new instance, and it took 8 hours to complete!!

Screen Shot 2016-11-21 at 11.36.58 AM.png

The original instance (where the snapshot was taken) disk size is 45G, and the new instance is 50G which I specified in the CloudFormation template. According to AWS, that is the reason why it took so long to complete. So the best practice is not to resize the DB size when restoring the snapshot. Re-run the restoration without resize the disk took only 20 minutes to finish.

If time is an issue, then creating a new empty DB instance, then use mysqldump should be a better option. Also I have submitted a feature request to make EBS resizable like how we used to increase virtual disk.

Quote from AWS support:

"When a DB instance is created from a snapshot, the initial db instance that is created has the same storage size/type as the snapshot. This process is fairly quick, but once the instance is stabilized, the storage is scaled out to your desired size, which takes some time to copy all of your data to another set of disks. After the scaling is complete, the database is configured as multi-az, which will launch another instance and go through the same steps to scale out the storage again."
        "Engine": "mysql",
        "EngineVersion": "5.6.21",
        "DBInstanceClass": "db.m1.medium",
        "DBSnapshotIdentifier": "arn:aws:rds:ap-southeast-2:1234567890:snapshot:rds:mydb01-2016-11-17-14-04",
        "MultiAZ": "true",
        "AllocatedStorage": "50",
        "DBInstanceIdentifier": "mydb02",
        "Port": "3306",
        "MasterUsername": "root",
        "MasterUserPassword": "**********",
        "VPCSecurityGroups": [{"Fn::ImportValue": "RDSProdId"}],
        "DBSubnetGroupName": {"Ref": "DBSubnetGroup"},
        "PubliclyAccessible": "false",
        "BackupRetentionPeriod": "3",
        "AutoMinorVersionUpgrade": "false",
        "StorageType": "gp2",

Double SSH Hops example


Client –ssh only–> Jumpbox00 –ssh only–> Jumpbox01 —http only-> Internal network

Here is how to ssh to jumpbox01, and visit websites in internal networks from Client.

1) Ensure you have a private key that is trusted by both jumpbox00 and jumpbox01. For example, jb.pem under ~/.ssh/, then run the following command:

ssh-add ~/.ssh/jb.pem 

2) Add the following two lines in ~/.ssh/config

Host jumpbox01
ForwardAgent yes

3) For convenience, create a command or alias. For example, I created a command /usr/local/bin/double_jump

ssh -i ~/.ssh/jb.pem -A -t jchen@jumpbox00 -L 12345:localhost:12345 ssh -A -t jchen@jumpbox01 -D 12345

4) Make the above command executable

sudo chmod a+x /usr/local/bin/double_jump

Now run double_jump, it will open a ssh session to jumpbox01. Set socks proxy in your browser to localhost:12345, it allows client to visit the websites in the internal network.

Avoid Elastic Beanstalk to create security group for ELB


Just found out that there is a Elastic Beanstalk option named ‘ManagedSecurityGroup‘ to allow you to use an exisiting security group for the ELB. Note the security group must be in the aws:elb:loadbalancer SecurityGroups list. Here is a sample:

“Namespace”: “aws:elb:loadbalancer”,
“OptionName”: “SecurityGroups“,
“Value”: “sg-1111111,sg-222222”
“Namespace”: “aws:elb:loadbalancer”,
“OptionName”: “ManagedSecurityGroup“,
“Value”: “sg-1111111”

Unfortunately, there is no such option for EC2. But you are able to control the SSH access in the EB managed EC2 security group with option ‘SSHSourceRestriction’

“Namespace”: “aws:autoscaling:launchconfiguration”,
“OptionName”: “SSHSourceRestriction“,
“Value”: “tcp, 22, 22, sg-222222”