Monitor V7000 active alerts in Xymon


Image

Image

Image

 

Continue with my previous post on installing the Xymon, you can write your own monitor scripts. I wrote a script named v7000alerts to check whether there are active alerts in my IBM StorWize V7000.

1) Download the script v7000alerts to /home/xymon/server/ext, and run ‘chmod a+x v7000alerts’ and ‘chown xymon:xymon v7000alerts’.

2) Add the following into /home/xymon/server/etc/tasks.cfg

[v7000alerts]
ENVFILE /home/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD $XYMONHOME/ext/v7000alerts
LOGFILE $XYMONSERVERLOGS/v7000alerts.log
INTERVAL 5m

3) Add your V7000 into the /home/xymon/server/etc/hosts.cfg file, here is a example:

# STORAGES PAGE
page storages STORAGES
group-compress <font size=”+1″>IBM StorWize V7000</font>
192.168.1.100 SYD-V7000 # v7000alerts sslcert https://192.168.1.100

The v7000alerts script can be downloaded from here:
Image
 

 

Setup Xymon monitor


Selection_007

Xymon is an easy and nice and systems and network monitor tool. It was inspired by the Big Brother system and was called Hobbit. I have set it up in my environment (CentOS 6.5) to monitor some critical web sites and services. Here are my how-to notes:

1) Install Xymon from source code.

2) Change the fping permission.

chown root:root /usr/local/sbin/fping
chmod u+s /usr/local/sbin/fping

3) (Optional) If the Xymon is behide a proxy server, you need to add the –bb-proxy-syntax in the xymonet configuration in the file /home/xymon/server/etc/tasks.cfg.

[xymonnet]
ENVFILE /home/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymonnet –report –ping –checkresponse –bb-proxy-syntax
LOGFILE $XYMONSERVERLOGS/xymonnet.log
INTERVAL 5m

4) (Optional) If you want to use Xymon to monitor the Domain Name validation, you need to:

- Install whois: yum install bind-utils jwhois

- Create a file /home/xymon/server/ext/dnsreg

- Add the following configurations to /home/xymon/server/etc/tasks.cfg

[dnsreg]
ENVFILE /home/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD $XYMONHOME/ext/dnsreg
LOGFILE $XYMONSERVERLOGS/dnsreg.log
INTERVAL 1d

5) Add the targets in the file /home/xymon/server/etc/hosts.cfg.

The format is: <IP Address> <Hostname> # <Services>

There are so many Services you can monitor, check the help file to find out more. Here are just some simple examples:

http: monitor the web server. Add the proxy in front of the web if Xymon works behide the proxy.
noconn: ping check is enabled by default. noconn is to disable it.
sslcert: monitor the SSL certificate validation.
smtp: monitor the smtp service.
dns: monitor the dns service.
ldap: monitor the ldap service.
dnsreg: monitor the public DNS registration expiration date.

Here is the sample:

# WEB PAGE
page webserver WEB

group-compress <font size=”+1″>INTERNAL</font>
192.168.1.1 web-01.my.lab # http://web-01.my.lab/
192.168.1.2 web-02.my.lab # http://web-02.my.lab/

group-compress <font size=”+1″>EXTERNAL</font>
2XX.1XX.1XX.1XX http://www.mypublicweb.com # noconn http://proxy-server:8080/http://www.mypublicweb.com/
2XX.1XX.1XX.1XX www.mysslpublicweb.com # noconn sslcert http://proxy-server:8080/http://www.mysslpublicweb.com/

# SERVICES PAGE
page services SERVICES

subpage internal-service INTERNAL
group-compress <font size=”+1″>XYMON</font>
192.168.1.100 xymon.my.lab # bbd http://xymon.my.lab/

group-compress <font size=”+1″>SMTP</font>
192.168.1.200 smtp.my.lab # smtp

group-compress <font size=”+1″>DNS LDAP</font>
0.0.0.0 ad.my.lab # DNS ldap

subpage external-service EXTERNAL

group-compress <font size=”+1″>DNS</font>
2XX.1XX.1XX.1XX dns.mypublic.com # noconn DNS

page domain DOMAINS

0.0.0.0 mypublic.com # noconn dnsreg

Setup GitHub in Linux


I should have done this earlier – to use GitHub to manage my scripts/configs/documentations. So I can have them managed and tracked. Later better than never, I got them setup Today in my Ubuntu box. Here is the how to:

1) Go to the GitHub.com to create an account.

2) Install git client.

jchen@mylaptop:~$ sudo apt-get install git -y

3) Configure the git user info and password caching.

jchen@mylaptop:~$ git config –global user.name “jchen”
jchen@mylaptop:~$ git config –global user.email “xxx@xxx.xxx”
jchen@mylaptop:~$ git config –global credential.helper cache
jchen@mylaptop:~$ git config –global credential.helper ‘cache –timeout=3600′

4) Go to GitHub.com to create a repository, my first repository is bash-scripts

5) Initialize the Git local repository in ~/Google Drive/scripts/bash-scripts

jchen@mylaptop:~/Google Drive/scripts$ mkdir bash-scripts
jchen@mylaptop:~/Google Drive/scripts$ cd bash-scripts
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git init
Initialized empty Git repository in /home/jchen/Google Drive/scripts/bash-scripts/.git/

6) Create README file in local repository, and add some descriptions.
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ touch README
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ vi README

7) Add README file to the commit list, then commit it.
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git add README
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git commit -m ‘general info’

8) Create a remote repository pointing to GitHub.com, and push my commits to it.
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git remote add origin https://github.com/jc1518/bash-scripts.git
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git push origin master
Username for ‘https://github.com': jc1518
Password for ‘https://jc1518@github.com':
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 252 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To https://github.com/jc1518/bash-scripts.git
* [new branch] master -> master

9) Add/Copy my first code to the local repository, and commit it, then push it to GitHub repository.
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ mkdir wallpaper-of-the-day
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ cp /usr/bin/wod wallpaper-of-the-day/
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git add wallpaper-of-the-day/
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git commit -m ‘wod first submit’
[master 73319e7] wod first submit
1 file changed, 27 insertions(+)
create mode 100755 wallpaper-of-the-day/wod
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git push origin master
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 723 bytes | 0 bytes/s, done.
Total 4 (delta 0), reused 0 (delta 0)
To https://github.com/jc1518/bash-scripts.git
6dfe1e4..73319e7 master -> master

RecordRoutingInfo: Unable to collect IPv4 routing table


One of my VMware Linux virtual machine takes a long time to boot. And there are couple warnings on the screen complaining RecordRoutingInfo: Unable to collect IPv4 routing table.

image

According to VMware KB2048572, this issue occurs when the Linux iputilis package causes a delay in the boot process. A warning message appears when the guestinfo plugin tool fails to parse the content from the /proc/net/route file. The guest operating system’s clock is ahead of the host on which it is running, causing the arping process to become unresponsive during boot. This results in the delayed boot times.

To work around this issue, add rtc.diffFromUTC=0 to the .vmx file of the affected virtual machine. This forces a time sync by setting the clock to UTC at power on, which enables the arping process to complete as expected leading to normal boot times.

Find out open files of a process


Some one asked me yesterday how to find out the open files of a process. I could not answer it. But I knew it has something to do with the /proc folder, as I restored the apache access log once from the /proc folder after a user deleted the access log in the /var folder before. After a quick search, I find two ways to do it. I still take apache as a example. First, check the pid of it. Say it is 16138

Method 1: Check the /pro/$pid/fd folder

[root@centos-01 ~]# ls -l /proc/16138/fd
total 0
lr-x——. 1 root root 64 Aug 28 21:27 0 -> /dev/null
l-wx——. 1 root root 64 Aug 28 21:27 1 -> /dev/null
l-wx——. 1 root root 64 Aug 28 21:27 2 -> /var/log/httpd/error_log
lrwx——. 1 root root 64 Aug 28 21:27 3 -> socket:[57650]
lrwx——. 1 root root 64 Aug 28 21:27 4 -> socket:[57651]
lr-x——. 1 root root 64 Aug 28 21:27 5 -> pipe:[57673]
l-wx——. 1 root root 64 Aug 28 21:27 6 -> pipe:[57673]
l-wx——. 1 root root 64 Aug 28 21:27 7 -> /var/log/httpd/access_log
lr-x——. 1 root root 64 Aug 28 21:27 8 -> /dev/urandom

Method 2: Use the lsof command

[root@centos-01 ~]# lsof -p 16138
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
httpd   16138 root  cwd    DIR    8,2     4096      2 /
httpd   16138 root  rtd    DIR    8,2     4096      2 /
httpd   16138 root  txt    REG    8,2   354816 399332 /usr/sbin/httpd
httpd   16138 root  mem    REG    8,2     9488 399251 /usr/lib64/apr-util-1/apr_ldap-1.so
httpd   16138 root  mem    REG    8,2    27424 260638 /lib64/libnss_dns-2.12.so
httpd   16138 root  mem    REG    8,2    65928 260640 /lib64/libnss_files-2.12.so
httpd   16138 root  mem    REG    8,2    10416 399328 /usr/lib64/httpd/modules/mod_version.so
httpd   16138 root  mem    REG    8,2    27312 399287 /usr/lib64/httpd/modules/mod_cgi.so
httpd   16138 root  mem    REG    8,2    22992 399294 /usr/lib64/httpd/modules/mod_disk_cache.so
httpd   16138 root  mem    REG    8,2    10384 399324 /usr/lib64/httpd/modules/mod_suexec.so
httpd   16138 root  mem    REG    8,2    39664 399285 /usr/lib64/httpd/modules/mod_cache.so
httpd   16138 root  mem    REG    8,2    14648 399314 /usr/lib64/httpd/modules/mod_proxy_connect.so
httpd   16138 root  mem    REG    8,2    39632 399312 /usr/lib64/httpd/modules/mod_proxy_ajp.so
httpd   16138 root  mem    REG    8,2    31472 399316 /usr/lib64/httpd/modules/mod_proxy_http.so
httpd   16138 root  mem    REG    8,2    35600 399315 /usr/lib64/httpd/modules/mod_proxy_ftp.so
httpd   16138 root  mem    REG    8,2    27160 399313 /usr/lib64/httpd/modules/mod_proxy_balancer.so
httpd   16138 root  mem    REG    8,2    85328 399311 /usr/lib64/httpd/modules/mod_proxy.so
httpd   16138 root  mem    REG    8,2    60464 399319 /usr/lib64/httpd/modules/mod_rewrite.so
httpd   16138 root  mem    REG    8,2    14608 399323 /usr/lib64/httpd/modules/mod_substitute.so
httpd   16138 root  mem    REG    8,2    14608 399267 /usr/lib64/httpd/modules/mod_alias.so
httpd   16138 root  mem    REG    8,2    10448 399326 /usr/lib64/httpd/modules/mod_userdir.so
httpd   16138 root  mem    REG    8,2    14608 399321 /usr/lib64/httpd/modules/mod_speling.so
httpd   16138 root  mem    REG    8,2    10416 399266 /usr/lib64/httpd/modules/mod_actions.so
httpd   16138 root  mem    REG    8,2    10416 399293 /usr/lib64/httpd/modules/mod_dir.so
httpd   16138 root  mem    REG    8,2    31384 399310 /usr/lib64/httpd/modules/mod_negotiation.so
httpd   16138 root  mem    REG    8,2    10424 399329 /usr/lib64/httpd/modules/mod_vhost_alias.so
httpd   16138 root  mem    REG    8,2    52016 399290 /usr/lib64/httpd/modules/mod_dav_fs.so
httpd   16138 root  mem    REG    8,2    19032 399303 /usr/lib64/httpd/modules/mod_info.so
httpd   16138 root  mem    REG    8,2    35472 399284 /usr/lib64/httpd/modules/mod_autoindex.so
httpd   16138 root  mem    REG    8,2    22864 399322 /usr/lib64/httpd/modules/mod_status.so
httpd   16138 root  mem    REG    8,2    93328 399289 /usr/lib64/httpd/modules/mod_dav.so
httpd   16138 root  mem    REG    8,2    18800 399308 /usr/lib64/httpd/modules/mod_mime.so
httpd   16138 root  mem    REG    8,2    14576 399320 /usr/lib64/httpd/modules/mod_setenvif.so
httpd   16138 root  mem    REG    8,2    14608 399327 /usr/lib64/httpd/modules/mod_usertrack.so
httpd   16138 root  mem    REG    8,2    18896 399300 /usr/lib64/httpd/modules/mod_headers.so
httpd   16138 root  mem    REG    8,2    22960 399292 /usr/lib64/httpd/modules/mod_deflate.so
httpd   16138 root  mem    REG    8,2    10512 399297 /usr/lib64/httpd/modules/mod_expires.so
httpd   16138 root  mem    REG    8,2    27672 399309 /usr/lib64/httpd/modules/mod_mime_magic.so
httpd   16138 root  mem    REG    8,2    23120 399298 /usr/lib64/httpd/modules/mod_ext_filter.so
httpd   16138 root  mem    REG    8,2    10416 399296 /usr/lib64/httpd/modules/mod_env.so
httpd   16138 root  mem    REG    8,2    10440 399307 /usr/lib64/httpd/modules/mod_logio.so
httpd   16138 root  mem    REG    8,2    27200 399305 /usr/lib64/httpd/modules/mod_log_config.so
httpd   16138 root  mem    REG    8,2    43600 399302 /usr/lib64/httpd/modules/mod_include.so
httpd   16138 root  mem    REG    8,2    31192 399277 /usr/lib64/httpd/modules/mod_authnz_ldap.so
httpd   16138 root  mem    REG    8,2    88520 260674 /lib64/libz.so.1.2.3
httpd   16138 root  mem    REG    8,2   106160 393220 /usr/lib64/libsasl2.so.2.0.23
httpd   16138 root  mem    REG    8,2   240592 260681 /lib64/libnspr4.so
httpd   16138 root  mem    REG    8,2    18720 260682 /lib64/libplc4.so
httpd   16138 root  mem    REG    8,2    14560 260683 /lib64/libplds4.so
httpd   16138 root  mem    REG    8,2   154456 392670 /usr/lib64/libnssutil3.so
httpd   16138 root  mem    REG    8,2  1283920 393943 /usr/lib64/libnss3.so
httpd   16138 root  mem    REG    8,2   181136 393946 /usr/lib64/libsmime3.so
httpd   16138 root  mem    REG    8,2   242112 393947 /usr/lib64/libssl3.so
httpd   16138 root  mem    REG    8,2   110960 260650 /lib64/libresolv-2.12.so
httpd   16138 root  mem    REG    8,2    60512 260919 /lib64/liblber-2.4.so.2.5.6
httpd   16138 root  mem    REG    8,2   305984 260921 /lib64/libldap-2.4.so.2.5.6
httpd   16138 root  mem    REG    8,2    56144 399304 /usr/lib64/httpd/modules/mod_ldap.so
httpd   16138 root  mem    REG    8,2     6264 399279 /usr/lib64/httpd/modules/mod_authz_default.so
httpd   16138 root  mem    REG    8,2    10448 399278 /usr/lib64/httpd/modules/mod_authz_dbm.so
httpd   16138 root  mem    REG    8,2    10520 399280 /usr/lib64/httpd/modules/mod_authz_groupfile.so
httpd   16138 root  mem    REG    8,2    10424 399282 /usr/lib64/httpd/modules/mod_authz_owner.so
httpd   16138 root  mem    REG    8,2    10384 399283 /usr/lib64/httpd/modules/mod_authz_user.so
httpd   16138 root  mem    REG    8,2    10448 399281 /usr/lib64/httpd/modules/mod_authz_host.so
httpd   16138 root  mem    REG    8,2     6264 399275 /usr/lib64/httpd/modules/mod_authn_default.so
httpd   16138 root  mem    REG    8,2    10416 399274 /usr/lib64/httpd/modules/mod_authn_dbm.so
httpd   16138 root  mem    REG    8,2    10352 399272 /usr/lib64/httpd/modules/mod_authn_anon.so
httpd   16138 root  mem    REG    8,2    10456 399271 /usr/lib64/httpd/modules/mod_authn_alias.so
httpd   16138 root  mem    REG    8,2    10416 399276 /usr/lib64/httpd/modules/mod_authn_file.so
httpd   16138 root  mem    REG    8,2    31256 399270 /usr/lib64/httpd/modules/mod_auth_digest.so
httpd   16138 root  mem    REG    8,2    10448 399269 /usr/lib64/httpd/modules/mod_auth_basic.so
httpd   16138 root  mem    REG    8,2   383504 260616 /lib64/libfreebl3.so
httpd   16138 root  mem    REG    8,2    16304 260706 /lib64/libuuid.so.1.3.0
httpd   16138 root  mem    REG    8,2    19536 260630 /lib64/libdl-2.12.so
httpd   16138 root  mem    REG    8,2  1912432 260624 /lib64/libc-2.12.so
httpd   16138 root  mem    REG    8,2   142464 260648 /lib64/libpthread-2.12.so
httpd   16138 root  mem    REG    8,2   181792 399247 /usr/lib64/libapr-1.so.0.3.9
httpd   16138 root  mem    REG    8,2  1522840 260690 /lib64/libdb-4.7.so
httpd   16138 root  mem    REG    8,2   165264 260828 /lib64/libexpat.so.1.5.2
httpd   16138 root  mem    REG    8,2    40400 260628 /lib64/libcrypt-2.12.so
httpd   16138 root  mem    REG    8,2   146080 399250 /usr/lib64/libaprutil-1.so.0.3.9
httpd   16138 root  mem    REG    8,2   122040 260694 /lib64/libselinux.so.1
httpd   16138 root  mem    REG    8,2   181464 260826 /lib64/libpcre.so.0.0.1
httpd   16138 root  mem    REG    8,2   595800 260632 /lib64/libm-2.12.so
httpd   16138 root  mem    REG    8,2   154464 260617 /lib64/ld-2.12.so
httpd   16138 root  DEL    REG    0,4           57681 /dev/zero
httpd   16138 root  DEL    REG    0,4           57679 /dev/zero
httpd   16138 root    0r   CHR    1,3      0t0   4021 /dev/null
httpd   16138 root    1w   CHR    1,3      0t0   4021 /dev/null
httpd   16138 root    2w   REG    8,2      452 781949 /var/log/httpd/error_log
httpd   16138 root    3u  sock    0,6      0t0  57650 can’t identify protocol
httpd   16138 root    4u  IPv6  57651      0t0    TCP *:http (LISTEN)
httpd   16138 root    5r  FIFO    0,8      0t0  57673 pipe
httpd   16138 root    6w  FIFO    0,8      0t0  57673 pipe
httpd   16138 root    7w   REG    8,2        0 782554 /var/log/httpd/access_log
httpd   16138 root    8r   CHR    1,9      0t0   4026 /dev/urandom

By Jackie Chen Posted in Linux Tagged