Hourly Intermittent Elastic Beanstalk Service Interruption


We noticed that one of our Java applications that is deployed on Elastic Beanstalk has hourly service interruption. Each outage only lasts for about 40 – 60 seconds. By looking at the CloudWatch, the ELB has intermittent 5xx errors every one hour, like 09:01, 10:01, 11:01 …

After some investigations, I found it out that it is by httpd logrotation. More specifically, it is the post script in the logrotation configuration (/etc/logrotate.d/logrotate.elasticbeanstalk.httpd.conf)

/sbin/service httpd reload > /dev/null 2>/dev/null || true

Under the hood ‘service httpd reload‘ runs as ‘/etc/init.d/httpd reload’ which send HUP signal to httpd parent process to tell it kill all child processes immediately then reload the http config file. That’s exactly how the service interruption happens. As it is not a graceful reload. Graceful reload signal should be USR1 which let all the child processes to finish the in flight requests before exit. Once I changed HUP to USR1 in the init script, the issues did not happen again (starting 21:00 in the above graph).

reload() {
    echo -n $"Reloading $prog: "
    if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then
        RETVAL=6
        echo $"not reloading due to configuration syntax error"
        failure $"not reloading $httpd due to configuration syntax error"
    else
        # Force LSB behaviour from killproc
        LSB=1 killproc -p ${pidfile} $httpd -HUP
        RETVAL=$?
        if [ $RETVAL -eq 7 ]; then
            failure $"httpd shutdown"
        fi
    fi
    echo
}

References:
https://httpd.apache.org/docs/2.4/stopping.html
https://www.datadoghq.com/blog/top-elb-health-and-performance-metrics/

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s