A wrong way of using Vault


We recently had a Vault issue. It is Vault with HA, the data store is Consul. One day the Consul cluster went offline, and we noticed there are a few OOM errors in the logs. The servers have been running fine and no stress on the memory utilisation for quite a while. There must be a dramatic change somewhere.

After some analysis of the audit logs, it turns out that a developer uses Vault in a wrong (lazy) way – In stead of creating a token with TTL,  and renew it or get a new one before the TTL expires. He just creates a new token each time when he needs to retrieve a credential from Vault, and set TTL to 0 which makes things worse. The consequence is that the keys stored in Consul increased 16 times within 3 days!! No wonder the servers are running out of memory.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s