Setup GitHub in Linux


I should have done this earlier – to use GitHub to manage my scripts/configs/documentations. So I can have them managed and tracked. Later better than never, I got them setup Today in my Ubuntu box. Here is the how to:

1) Go to the GitHub.com to create an account.

2) Install git client.

jchen@mylaptop:~$ sudo apt-get install git -y

3) Configure the git user info and password caching.

jchen@mylaptop:~$ git config –global user.name “jchen”
jchen@mylaptop:~$ git config –global user.email “xxx@xxx.xxx”
jchen@mylaptop:~$ git config –global credential.helper cache
jchen@mylaptop:~$ git config –global credential.helper ‘cache –timeout=3600′

4) Go to GitHub.com to create a repository, my first repository is bash-scripts

5) Initialize the Git local repository in ~/Google Drive/scripts/bash-scripts

jchen@mylaptop:~/Google Drive/scripts$ mkdir bash-scripts
jchen@mylaptop:~/Google Drive/scripts$ cd bash-scripts
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git init
Initialized empty Git repository in /home/jchen/Google Drive/scripts/bash-scripts/.git/

6) Create README file in local repository, and add some descriptions.
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ touch README
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ vi README

7) Add README file to the commit list, then commit it.
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git add README
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git commit -m ‘general info’

8) Create a remote repository pointing to GitHub.com, and push my commits to it.
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git remote add origin https://github.com/jc1518/bash-scripts.git
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git push origin master
Username for ‘https://github.com': jc1518
Password for ‘https://jc1518@github.com':
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 252 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To https://github.com/jc1518/bash-scripts.git
* [new branch] master -> master

9) Add/Copy my first code to the local repository, and commit it, then push it to GitHub repository.
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ mkdir wallpaper-of-the-day
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ cp /usr/bin/wod wallpaper-of-the-day/
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git add wallpaper-of-the-day/
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git commit -m ‘wod first submit’
[master 73319e7] wod first submit
1 file changed, 27 insertions(+)
create mode 100755 wallpaper-of-the-day/wod
jchen@mylaptop:~/Google Drive/scripts/bash-scripts$ git push origin master
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 723 bytes | 0 bytes/s, done.
Total 4 (delta 0), reused 0 (delta 0)
To https://github.com/jc1518/bash-scripts.git
6dfe1e4..73319e7 master -> master

RecordRoutingInfo: Unable to collect IPv4 routing table


One of my VMware Linux virtual machine takes a long time to boot. And there are couple warnings on the screen complaining RecordRoutingInfo: Unable to collect IPv4 routing table.

image

According to VMware KB2048572, this issue occurs when the Linux iputilis package causes a delay in the boot process. A warning message appears when the guestinfo plugin tool fails to parse the content from the /proc/net/route file. The guest operating system’s clock is ahead of the host on which it is running, causing the arping process to become unresponsive during boot. This results in the delayed boot times.

To work around this issue, add rtc.diffFromUTC=0 to the .vmx file of the affected virtual machine. This forces a time sync by setting the clock to UTC at power on, which enables the arping process to complete as expected leading to normal boot times.

Find out open files of a process


Some one asked me yesterday how to find out the open files of a process. I could not answer it. But I knew it has something to do with the /proc folder, as I restored the apache access log once from the /proc folder after a user deleted the access log in the /var folder before. After a quick search, I find two ways to do it. I still take apache as a example. First, check the pid of it. Say it is 16138

Method 1: Check the /pro/$pid/fd folder

[root@centos-01 ~]# ls -l /proc/16138/fd
total 0
lr-x——. 1 root root 64 Aug 28 21:27 0 -> /dev/null
l-wx——. 1 root root 64 Aug 28 21:27 1 -> /dev/null
l-wx——. 1 root root 64 Aug 28 21:27 2 -> /var/log/httpd/error_log
lrwx——. 1 root root 64 Aug 28 21:27 3 -> socket:[57650]
lrwx——. 1 root root 64 Aug 28 21:27 4 -> socket:[57651]
lr-x——. 1 root root 64 Aug 28 21:27 5 -> pipe:[57673]
l-wx——. 1 root root 64 Aug 28 21:27 6 -> pipe:[57673]
l-wx——. 1 root root 64 Aug 28 21:27 7 -> /var/log/httpd/access_log
lr-x——. 1 root root 64 Aug 28 21:27 8 -> /dev/urandom

Method 2: Use the lsof command

[root@centos-01 ~]# lsof -p 16138
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
httpd   16138 root  cwd    DIR    8,2     4096      2 /
httpd   16138 root  rtd    DIR    8,2     4096      2 /
httpd   16138 root  txt    REG    8,2   354816 399332 /usr/sbin/httpd
httpd   16138 root  mem    REG    8,2     9488 399251 /usr/lib64/apr-util-1/apr_ldap-1.so
httpd   16138 root  mem    REG    8,2    27424 260638 /lib64/libnss_dns-2.12.so
httpd   16138 root  mem    REG    8,2    65928 260640 /lib64/libnss_files-2.12.so
httpd   16138 root  mem    REG    8,2    10416 399328 /usr/lib64/httpd/modules/mod_version.so
httpd   16138 root  mem    REG    8,2    27312 399287 /usr/lib64/httpd/modules/mod_cgi.so
httpd   16138 root  mem    REG    8,2    22992 399294 /usr/lib64/httpd/modules/mod_disk_cache.so
httpd   16138 root  mem    REG    8,2    10384 399324 /usr/lib64/httpd/modules/mod_suexec.so
httpd   16138 root  mem    REG    8,2    39664 399285 /usr/lib64/httpd/modules/mod_cache.so
httpd   16138 root  mem    REG    8,2    14648 399314 /usr/lib64/httpd/modules/mod_proxy_connect.so
httpd   16138 root  mem    REG    8,2    39632 399312 /usr/lib64/httpd/modules/mod_proxy_ajp.so
httpd   16138 root  mem    REG    8,2    31472 399316 /usr/lib64/httpd/modules/mod_proxy_http.so
httpd   16138 root  mem    REG    8,2    35600 399315 /usr/lib64/httpd/modules/mod_proxy_ftp.so
httpd   16138 root  mem    REG    8,2    27160 399313 /usr/lib64/httpd/modules/mod_proxy_balancer.so
httpd   16138 root  mem    REG    8,2    85328 399311 /usr/lib64/httpd/modules/mod_proxy.so
httpd   16138 root  mem    REG    8,2    60464 399319 /usr/lib64/httpd/modules/mod_rewrite.so
httpd   16138 root  mem    REG    8,2    14608 399323 /usr/lib64/httpd/modules/mod_substitute.so
httpd   16138 root  mem    REG    8,2    14608 399267 /usr/lib64/httpd/modules/mod_alias.so
httpd   16138 root  mem    REG    8,2    10448 399326 /usr/lib64/httpd/modules/mod_userdir.so
httpd   16138 root  mem    REG    8,2    14608 399321 /usr/lib64/httpd/modules/mod_speling.so
httpd   16138 root  mem    REG    8,2    10416 399266 /usr/lib64/httpd/modules/mod_actions.so
httpd   16138 root  mem    REG    8,2    10416 399293 /usr/lib64/httpd/modules/mod_dir.so
httpd   16138 root  mem    REG    8,2    31384 399310 /usr/lib64/httpd/modules/mod_negotiation.so
httpd   16138 root  mem    REG    8,2    10424 399329 /usr/lib64/httpd/modules/mod_vhost_alias.so
httpd   16138 root  mem    REG    8,2    52016 399290 /usr/lib64/httpd/modules/mod_dav_fs.so
httpd   16138 root  mem    REG    8,2    19032 399303 /usr/lib64/httpd/modules/mod_info.so
httpd   16138 root  mem    REG    8,2    35472 399284 /usr/lib64/httpd/modules/mod_autoindex.so
httpd   16138 root  mem    REG    8,2    22864 399322 /usr/lib64/httpd/modules/mod_status.so
httpd   16138 root  mem    REG    8,2    93328 399289 /usr/lib64/httpd/modules/mod_dav.so
httpd   16138 root  mem    REG    8,2    18800 399308 /usr/lib64/httpd/modules/mod_mime.so
httpd   16138 root  mem    REG    8,2    14576 399320 /usr/lib64/httpd/modules/mod_setenvif.so
httpd   16138 root  mem    REG    8,2    14608 399327 /usr/lib64/httpd/modules/mod_usertrack.so
httpd   16138 root  mem    REG    8,2    18896 399300 /usr/lib64/httpd/modules/mod_headers.so
httpd   16138 root  mem    REG    8,2    22960 399292 /usr/lib64/httpd/modules/mod_deflate.so
httpd   16138 root  mem    REG    8,2    10512 399297 /usr/lib64/httpd/modules/mod_expires.so
httpd   16138 root  mem    REG    8,2    27672 399309 /usr/lib64/httpd/modules/mod_mime_magic.so
httpd   16138 root  mem    REG    8,2    23120 399298 /usr/lib64/httpd/modules/mod_ext_filter.so
httpd   16138 root  mem    REG    8,2    10416 399296 /usr/lib64/httpd/modules/mod_env.so
httpd   16138 root  mem    REG    8,2    10440 399307 /usr/lib64/httpd/modules/mod_logio.so
httpd   16138 root  mem    REG    8,2    27200 399305 /usr/lib64/httpd/modules/mod_log_config.so
httpd   16138 root  mem    REG    8,2    43600 399302 /usr/lib64/httpd/modules/mod_include.so
httpd   16138 root  mem    REG    8,2    31192 399277 /usr/lib64/httpd/modules/mod_authnz_ldap.so
httpd   16138 root  mem    REG    8,2    88520 260674 /lib64/libz.so.1.2.3
httpd   16138 root  mem    REG    8,2   106160 393220 /usr/lib64/libsasl2.so.2.0.23
httpd   16138 root  mem    REG    8,2   240592 260681 /lib64/libnspr4.so
httpd   16138 root  mem    REG    8,2    18720 260682 /lib64/libplc4.so
httpd   16138 root  mem    REG    8,2    14560 260683 /lib64/libplds4.so
httpd   16138 root  mem    REG    8,2   154456 392670 /usr/lib64/libnssutil3.so
httpd   16138 root  mem    REG    8,2  1283920 393943 /usr/lib64/libnss3.so
httpd   16138 root  mem    REG    8,2   181136 393946 /usr/lib64/libsmime3.so
httpd   16138 root  mem    REG    8,2   242112 393947 /usr/lib64/libssl3.so
httpd   16138 root  mem    REG    8,2   110960 260650 /lib64/libresolv-2.12.so
httpd   16138 root  mem    REG    8,2    60512 260919 /lib64/liblber-2.4.so.2.5.6
httpd   16138 root  mem    REG    8,2   305984 260921 /lib64/libldap-2.4.so.2.5.6
httpd   16138 root  mem    REG    8,2    56144 399304 /usr/lib64/httpd/modules/mod_ldap.so
httpd   16138 root  mem    REG    8,2     6264 399279 /usr/lib64/httpd/modules/mod_authz_default.so
httpd   16138 root  mem    REG    8,2    10448 399278 /usr/lib64/httpd/modules/mod_authz_dbm.so
httpd   16138 root  mem    REG    8,2    10520 399280 /usr/lib64/httpd/modules/mod_authz_groupfile.so
httpd   16138 root  mem    REG    8,2    10424 399282 /usr/lib64/httpd/modules/mod_authz_owner.so
httpd   16138 root  mem    REG    8,2    10384 399283 /usr/lib64/httpd/modules/mod_authz_user.so
httpd   16138 root  mem    REG    8,2    10448 399281 /usr/lib64/httpd/modules/mod_authz_host.so
httpd   16138 root  mem    REG    8,2     6264 399275 /usr/lib64/httpd/modules/mod_authn_default.so
httpd   16138 root  mem    REG    8,2    10416 399274 /usr/lib64/httpd/modules/mod_authn_dbm.so
httpd   16138 root  mem    REG    8,2    10352 399272 /usr/lib64/httpd/modules/mod_authn_anon.so
httpd   16138 root  mem    REG    8,2    10456 399271 /usr/lib64/httpd/modules/mod_authn_alias.so
httpd   16138 root  mem    REG    8,2    10416 399276 /usr/lib64/httpd/modules/mod_authn_file.so
httpd   16138 root  mem    REG    8,2    31256 399270 /usr/lib64/httpd/modules/mod_auth_digest.so
httpd   16138 root  mem    REG    8,2    10448 399269 /usr/lib64/httpd/modules/mod_auth_basic.so
httpd   16138 root  mem    REG    8,2   383504 260616 /lib64/libfreebl3.so
httpd   16138 root  mem    REG    8,2    16304 260706 /lib64/libuuid.so.1.3.0
httpd   16138 root  mem    REG    8,2    19536 260630 /lib64/libdl-2.12.so
httpd   16138 root  mem    REG    8,2  1912432 260624 /lib64/libc-2.12.so
httpd   16138 root  mem    REG    8,2   142464 260648 /lib64/libpthread-2.12.so
httpd   16138 root  mem    REG    8,2   181792 399247 /usr/lib64/libapr-1.so.0.3.9
httpd   16138 root  mem    REG    8,2  1522840 260690 /lib64/libdb-4.7.so
httpd   16138 root  mem    REG    8,2   165264 260828 /lib64/libexpat.so.1.5.2
httpd   16138 root  mem    REG    8,2    40400 260628 /lib64/libcrypt-2.12.so
httpd   16138 root  mem    REG    8,2   146080 399250 /usr/lib64/libaprutil-1.so.0.3.9
httpd   16138 root  mem    REG    8,2   122040 260694 /lib64/libselinux.so.1
httpd   16138 root  mem    REG    8,2   181464 260826 /lib64/libpcre.so.0.0.1
httpd   16138 root  mem    REG    8,2   595800 260632 /lib64/libm-2.12.so
httpd   16138 root  mem    REG    8,2   154464 260617 /lib64/ld-2.12.so
httpd   16138 root  DEL    REG    0,4           57681 /dev/zero
httpd   16138 root  DEL    REG    0,4           57679 /dev/zero
httpd   16138 root    0r   CHR    1,3      0t0   4021 /dev/null
httpd   16138 root    1w   CHR    1,3      0t0   4021 /dev/null
httpd   16138 root    2w   REG    8,2      452 781949 /var/log/httpd/error_log
httpd   16138 root    3u  sock    0,6      0t0  57650 can’t identify protocol
httpd   16138 root    4u  IPv6  57651      0t0    TCP *:http (LISTEN)
httpd   16138 root    5r  FIFO    0,8      0t0  57673 pipe
httpd   16138 root    6w  FIFO    0,8      0t0  57673 pipe
httpd   16138 root    7w   REG    8,2        0 782554 /var/log/httpd/access_log
httpd   16138 root    8r   CHR    1,9      0t0   4026 /dev/urandom

By Jackie Chen Posted in Linux Tagged

Encrypt disk in Ubuntu


I installed  a Ubuntu 12.04 desktop by using the 12.04.2 alternate CD. After generated a distribution iso, I found the disk encryption feature was not included in the custom install process. To make the new iso support disk encryption, I wrote a script. Here is how it works:

1) Boot from the live CD.

2) Run the script (the codes can be found at the bottom).

image

3) The script will set up the partitions and ask you to provide the encryption passphrase.

image

4) Choose the language.

image

5) Choose Continue.

image

6) Choose ‘Something else’.

image

7) Choose /dev/mapper/sda5_crypt as the mount point for / and /dev/sda1 as the mount point for /boot, then click ‘Install Now’.

image

8) Click ‘Continue’.

image

9) Choose location.

image

10) Choose Keyboard

image

11) Choose ‘Continue testing’.

image

12) The script will automatically configure the newly installed system and reboot.

13) The system will ask for the encryption passphrase after reboot.

image

image

image

#!/bin/bash

# Wipe existing partition
dd if=/dev/zero of=/dev/sda bs=512 count=1

# Set up partition
(echo n; echo p; echo; echo; echo +300M; echo n; echo p; echo; echo; echo +2G; echo n; echo e; echo; echo; echo; echo n; echo l; echo; echo; echo t; echo 2; echo 82; echo w) | sudo fdisk /dev/sda

sudo fdisk -l

# Encrypt disk
sudo cryptsetup -y -v luksFormat /dev/sda5
sudo cryptsetup luksOpen /dev/sda5 sda5_crypt

# Format disk
sudo mkfs.ext2 /dev/sda1
sudo mkswap /dev/sda2
sudo mkfs.ext4 /dev/mapper/sda5_crypt

# Install Ubuntu
ubiquity –desktop %k gtk_ui

# Configure the new system
sudo mount /dev/mapper/sda5_crypt /mnt
sudo chroot /mnt mount /proc
sudo mount –bind /dev /mnt/dev
sudo chroot /mnt mount /boot

echo “sda5_crypt UUID=`sudo blkid -s UUID -o value /dev/sda5` none luks” | sudo tee -a /mnt/etc/crypttab

sudo chroot /mnt update-initramfs -u
sudo umount /mnt/proc /mnt/dev /mnt/boot /mnt
echo rebooting…
sudo reboot

download_icon

Auto deploy Puppet Agent


Continue with my previous Puppet and Puppet dashboard installation, now I start to deploy the agent to all my Ubuntu desktops. I wrote a script to do the dirty work.
Pre-requisite package: expect (no need if use passwordless ssh, but you need to change the script slightly).
How to use:
1) Create the following 4 scripts (sshlogin.exp, scplogin.exp, install_puppet and deploy_puppet), and make them executable.
2) Add all IP address into the IP.list file, only one IP in each line.
3) Run ./deploy_puppet, then go to get a cup of coffee :)
+++++++++++++++++++++++++++++
#!/usr/bin/expect -f
# sshlogin.exp
# syntax:  ./sshlogin.exp password ip command
# set Variables
set password [lrange $argv 0 0]
set ipaddr [lrange $argv 1 1]
set scriptname [lrange $argv 2 2]
set arg1 [lrange $argv 3 3]
set timeout -1
# now connect to remote UNIX box (ipaddr) with given script to execute
# replace jchen with your username
spawn ssh -o StrictHostKeyChecking=no jchen@$ipaddr $scriptname $arg1
match_max 100000
# Look for passwod prompt
expect “*?assword:*”
# Send password aka $password
send — “$password”
# send blank line () to make sure we get back to gui
send — “”
expect eof
+++++++++++++++++++++++++++++
#!/usr/bin/expect -f
# scplogin.exp
# syntax: ./sshlogin.exp password ip filename
# set Variables
set password [lrange $argv 0 0]
set ipaddr [lrange $argv 1 1]
set filename [lrange $argv 2 2]
set timeout -1
# now connect to remote UNIX box (ipaddr) with given script to execute
# replace jchen with your username
spawn scp $filename jchen@$ipaddr:~/
match_max 100000
# Look for passwod prompt
expect “*?assword:*”
# Send password aka $password
send — “$password”
# send blank line () to make sure we get back to gui
send — “”
expect eof
+++++++++++++++++++++++++++++
#!/bin/bash
# install_puppet
#install puppet
echo N  | sudo apt-get install -y puppet
#start puppet on boot
sudo puppet resource service puppet ensure=running enable=true
sudo puppet agent –test
sudo chmod 666 /etc/default/puppet
sudo echo START=yes > /etc/default/puppet
sudo echo DAEMON_OPTS=”” >> /etc/default/puppet
#configure the server in the conf file
agent=$(hostname | tr ‘[A-Z]‘ ‘[a-z]‘)
sudo touch /etc/puppet/puppet.conf
sudo chmod 666 /etc/puppet/puppet.conf
sudo echo [main] > /etc/puppet/puppet.conf
# replace CentOS.my.lab with your Master server name
sudo echo server=CentOS.my.lab >> /etc/puppet/puppet.conf
sudo echo certname=$agent >> /etc/puppet/puppet.conf
sudo echo logdir=/var/log/puppet >> /etc/puppet/puppet.conf
sudo echo vardir=/var/lib/puppet >> /etc/puppet/puppet.conf
sudo echo ssldir=/var/lib/puppet/ssl >> /etc/puppet/puppet.conf
sudo echo rundir=/var/run/puppet >> /etc/puppet/puppet.conf
sudo echo factpath=$vardir/lib/facter >> /etc/puppet/puppet.conf
sudo echo templatedir=$confdir/templates >> /etc/puppet/puppet.conf
sudo echo prerun_command=/etc/puppet/etckeeper-commit-pre >> /etc/puppet/puppet.conf
sudo echo postrun_command=/etc/puppet/etckeeper-commit-post >> /etc/puppet/puppet.conf
#restart puppet
sudo puppet agent
+++++++++++++++++++++++++++++++
#!/bin/bash
# deploy_puppet
# check if IP pingable
for ip in $(cat IP.list)
do
if  ping -c 4 $ip > /dev/null 2>&1
then
echo $ip ‘is pingable’
#check /etc/puppet exisitence
 if ./sshlogin.exp password $ip ls /etc | grep puppet > /dev/null 2>&1
 then
 echo ‘Puppet is already installed on’ $ip
 else
 #copy install script to remote host
 ./scplogin.exp password $ip install_puppet
 #install puppet on remote host
 ./sshlogin.exp password $ip /home/jchen/install_puppet
 echo $ip >> IP.success
 fi
else
echo $ip ‘is NOT pingable’
echo $ip >> IP.not_pingable
fi
done
download_icon