If you md5sum a bunch of files, you get this kind of output:

[ben@minitron ~]$ md5sum *.txt
d41d8cd98f00b204e9800998ecf8427e  myfile.txt
cdbd2b665e14f8803d6bddc80bff1402  omg.txt
6353f9b2e7c47f73b48eef4544968bcd  otherfile.txt

I wanted to iterate through a list of files and their md5sums using a for loop, run a test and then take some action based on the result. But my script kept throwing unexpected output. I realised the for loop was parsing each word, rather than each line because of the spaces.

[ben@minitron ~]$ for line in $(md5sum *.txt); do
echo $line; done
d41d8cd98f00b204e9800998ecf8427e
myfile.txt
cdbd2b665e14f8803d6bddc80bff1402
omg.txt
6353f9b2e7c47f73b48eef4544968bcd
otherfile.txt

Sometimes putting spaces around variables can help bash to resolve values in the right way, but here it caused all line breaks to be removed:

[ben@minitron ~]$ for line in "$(md5sum *.txt)"; do echo $line; done
d41d8cd98f00b204e9800998ecf8427e myfile.txt cdbd2b665e14f8803d6bddc80bff1402 omg.txt 6353f9b2e7c47f73b48eef4544968bcd otherfile.txt

I needed to preserve the original line breaks in order for my script to make sense of which md5sum matched up to which file. There are a couple of ways to do this. One possible solution is to use tr to translate the spaces into a different character:

[ben@minitron ~]$ for line in $(md5sum *.txt | tr " " :); do
echo $line; done
d41d8cd98f00b204e9800998ecf8427e::myfile.txt
cdbd2b665e14f8803d6bddc80bff1402::omg.txt
6353f9b2e7c47f73b48eef4544968bcd::otherfile.txt

But for some reason md5sum uses 2 spaces for a field separator, if you wanted just one character between the md5sum and the filename, you could go a little more advanced:

[ben@minitron ~]$ for file in *.txt; do
echo -n "$file:";md5sum $file | cut -d " " -f1 ; done
myfile.txt:d41d8cd98f00b204e9800998ecf8427e
omg.txt:cdbd2b665e14f8803d6bddc80bff1402
otherfile.txt:6353f9b2e7c47f73b48eef4544968bcd

This does give the output in the other way round but was good enough for me to make my script work.

Notes
One thing I have noticed with command line Java utilities, is they always output to stdErr. So if you want to grep or awk the output of the commands you have to redirect that to stdIn first, hence the "2>&1".

• Replace user:pass with the JMX username and password set in jmx.properties
• Replace server with the first part of your server name
• Replace the port numbers with the JMX ports your servers are using.
• Finally you'll need to change the paths to the log file and the JMX client.
• The variables "one","three", "five" and "seven" can be changed to anything you want. It makes sense to call them this in my environment.

#!/bin/bash
while true; do

 export now=$(date +%Y-%m-%d)
 export log=/path/to/ajp_all_tomcat_${now}.log
  if [ ! -f $log ]; then
   echo -n "Start AJP thread count log
Time" > $log;
   for s in $(seq -w 1 30); do for i in 1 3 5 7; do echo -n ",server${s}tc0${i}" >> $log; done; done
   echo >> $log
  fi

 echo -n $(date +%H:%M:%S) >> $log
 for s in $(seq -w 1 30); do
  export one=$(/usr/bin/java -jar /path/to/cmdline-jmxclient-0.10.3.jar user:pass server${s}:7601 "Catalina:type=ThreadPool,name=ajp-7501" currentThreadsBusy 2>&1 | awk '{print $6}')
  export three=$(/usr/bin/java -jar /path/to/cmdline-jmxclient-0.10.3.jar user:pass server${s}:7603 "Catalina:type=ThreadPool,name=ajp-7503" currentThreadsBusy 2>&1 | awk '{print $6}')
  export five=$(/usr/bin/java -jar /path/to/cmdline-jmxclient-0.10.3.jar user:pass server${s}:7605 "Catalina:type=ThreadPool,name=ajp-7505" currentThreadsBusy 2>&1 | awk '{print $6}')
  export seven=$(/usr/bin/java -jar /path/to/cmdline-jmxclient-0.10.3.jar user:pass server${s}:7607 "Catalina:type=ThreadPool,name=ajp-7507" currentThreadsBusy 2>&1 | awk '{print $6}')
  echo -n ",$one,$three,$five,$seven" >> $log
 done
 echo >> $log

done

Add the following to the end of your .bash_profile:

export DISPLAY=$(who am i | awk -F "(" {'print $2'} | awk -F "." {'print $1'}):0.0

Then on my Windows 7 laptop, I installed xming and configured xlaunch to start automatically when the computer is started (with the "no access control" box ticked).

Now I can just SSH to servers and launch X apps, and they appear on my desktop with no configuration required.

• Send advance warning to users
• Log on to site, edit params and enter some text under 'shutdown site'
• Backup db using this commmand

mysqldump --opt -u bugs -p bugs > bugs_backup_$(date +%d%m%y).sql

• Download and extract bugzilla 3.4.6

cd ~
wget http://ftp.mozilla.org/pub/mozilla.org/webtools/bugzilla-3.4.6.tar.gz
tar fvzx bugzilla-3.4.6.tar.gz
cd /var/www/html/
mv bugzilla bugzilla.old
cp -r ~/bugzilla-3.4.6 ./bugzilla
cp -r bugzilla.old/localconfig* bugzilla

Later you will install the DBD::mysql perl module which requires mysql header files. So if you don't have mysql-devel, install it now. Also the Image::Magick perl module may fail to build from cpan (this is what happened to me), this can be installed by yum as an alternative.

yum install mysql-devel ImageMagick-perl.i386
cd bugzilla
./checksetup.pl

checksetup.pl will tell you to install missing perl modules using cpan, however this requires an internet connection. I used ntlmaps to proxy through our MS ISA server as my bugzilla server doesn't have Internet access. I got ntlmaps from another server on the network:

scp -r root@bugtest:/root/ntlm* .
cd ntlmaps-0.9.9.0.1
nohup python main.py
export http_proxy=http://localhost:5865
export ftp_proxy=ftp://localhost:5865
cd /var/www/html/bugzilla

Install optional updates:

# failed  --> /usr/bin/perl install-module.pl Image::Magick
# skipped --> /usr/bin/perl install-module.pl Authen::Radius
/usr/bin/perl install-module.pl SOAP::Lite
/usr/bin/perl install-module.pl TheSchwartz
/usr/bin/perl install-module.pl Daemon::Generic
/usr/bin/perl install-module.pl SOAP::Lite

Install compulsary updates:

/usr/bin/perl install-module.pl DBD::mysql
/usr/bin/perl install-module.pl CGI
/usr/bin/perl install-module.pl Digest::SHA
/usr/bin/perl install-module.pl DateTime
/usr/bin/perl install-module.pl DateTime::TimeZone
/usr/bin/perl install-module.pl Template
/usr/bin/perl install-module.pl Email::MIME
/usr/bin/perl install-module.pl Email::MIME::Encodings

I also needed these perl modules, even though they weren't listed by checksetup.pl:

/usr/bin/perl install-module.pl DateTime::Locale
/usr/bin/perl install-module.pl List::MoreUtils

Finished with ntlmaps now, it can be stopped

kill $(ps x | grep main.py | grep -v grep | awk {'print $1'})
export http_proxy=
export ftp_proxy=

Run checksetup again, this bit can take some time while the db is upgraded:

./checksetup.pl

• Fix any more problems, run again if required.
• Once finished log into bugzilla, put maintenance page back up. This option has now moved to Administration
• Run sanity check, fix any problems.
• Change any css and templates that you lost in the upgrade.
• Reopen the site for logons when you are ready.

#!/bin/sh

echo -n "Enter command to run (on one line): "
read STRING

for SERVER in server2 server3 server4 server5 server6 etc; do

echo -e "\033[1;31m$SERVER says:\033[m"
ssh $SERVER "$STRING"

done

echo -n "Do you want to run the command locally? (y/n) :"
read ANSWER

case "$ANSWER" in

y|Y) echo -e "\033[1;31mlocalhost says:\033[m"; $STRING ;;
N|n) exit 1 ;;
esac

For this to work I had to create ssh keys on each server using:

ssh-keygen -t dsa

And then install the newly generated key (~/ssh/id_dsa.pub) into the authorized_keys file on server1. This works for running simple commands one at a time.

For other tasks its sometimes necessary to manage multiple ssh sessions at once, for example to monitor resources using htop or tailing log files. On Linux you can use ClusterSSH (cssh) but this turned out to be a royal pain in the butt to get working on CentOS or RHEL. It worked OK in an Ubuntu VM but was a bit clunky and I felt there would be a better way of managing multiple PuTTy windows, since I am using a Windows 7 laptop for my day-to-day stuff.

There are quite a few goodies for this on the Links page on the Putty website and this is where I found Putty Command Sender. Quite simply you type your command into the command sender, and much like clusterssh, it sends it to all the putty windows you have open.

Putty Command Sender

It's not so great for editing files in Vi on 12 servers at once but it is possible - you can send cursor movements as well as lines of code or single commands. The only thing you have to watch out for in PuttyCS is that all Putty windows have updated before you start typing your next command, otherwise the last window to update may miss the first few characters of the next command.

Finally to launch my sessions in groups I'm using Putty Session Manager. Other alternatives are available but I found this one to be the best. It's lightweight nature fits in with Putty nicely.

boot: linux ks=http://server/kickstart.cfg ksdevice=eth0 ip=xx.xx.xx.xx netmask=xx.xx.xx.xx gateway=xx.xx.xx.xx dns=xx.xx.xx.xx

DNS servers can be comma separated if you want to specify more than one.

kern.*        @xx.xx.xx.xx

in the syslog.conf. In our case we're logging to our Cacti server and using the Cacti syslog plugin (from here) to view our Linux server and network switch syslogs in a central location.

iptables configuration
In a default installation of CentOS, iptables is already installed and running by default. There are only 2 changes to make; The first is to bind your additional IP address to your network card (usually eth0). This is done by creating a virtual device called eth0:1, by inserting this text into new file etc/sysconfig/network-scripts/ifcfg-eth0:1 (10.10.2.17 is the destination address of the incoming traffic that I want to log):

IPADDR=10.10.2.17
NETMASK=255.255.0.0

And then bringing up the virtual device using:

ifconfig eth0:1 up

In the iptables config file (etc/sysconfig/iptables) I added these lines in the RH-Firewall-1-INPUT chain:

-A RH-Firewall-1-INPUT -d 10.10.2.17 -j LOG --log-level 4 --log-prefix "OLD SRV1"
-A RH-Firewall-1-INPUT -d 10.10.2.17 -j DROP

Any traffic to 10.10.2.17 will now be logged in messages and the lines will be prefixed with "OLD SRV1".

Note about file paths: I'm getting an odd error when posting Linux file paths in WordPress - putting a leading / in the path gives a 404 error. The paths I mentioned above should have a leading forwardslash "/".

This runs every 5 minutes as a cron job and updates the file /etc/motd (Message Of The Day) which is shown when a user logs in. Our servers already have a 5 line banner message with information including the server name and purpose of the server, hence the first 5 lines being saved and readded into a new /etc/motd each time.

#!/bin/bash
cat /etc/motd | head -n 5 > /tmp/file && cat /tmp/file > /etc/motd

CPUTIME=$(ps -eo pcpu | awk 'NR>1' | awk '{tot=tot+$1} END {print tot}')
CPUCORES=$(cat /proc/cpuinfo | grep -c processor)

echo "System summary (collected `date`)

 - CPU Usage (total average) = `echo $CPUTIME / $CPUCORES | bc`%
 - Memory free (real)        = `free -m | head -n 2 | tail -n 1 | awk {'print $4'}` Mb
 - Memory free (cache)       = `free -m | head -n 3 | tail -n 1 | awk {'print $3'}` Mb
 - Swap in use               = `free -m | tail -n 1 | awk {'print $3'}` Mb
" >> /etc/motd

Now when we log in we get a summary like this:

I recently was confronted with a VM that was out of space on /usr/local. The filesystem was already using LVM so I just added a new virtual disk and stretched the /usr/local volume over the new disk. The whole process is even easier in VMware as you can add the new disk while the machine is running and run through the whole process without a reboot, providing you don't have daemons or processes running in /usr/local that stop it from being unmounted. Here's how I did it:

init 1
umount /usr/local

Going to runlevel 1 may not always be necessary but was in my case. pvcreate enables the new physical disk for use with LVM. Then vgextend extends the volume group, and lvresze resizes the logical volume. In my case the new disk that was added became known to the system as /dev/sdc.

pvcreate /dev/sdc
vgextend VolGroup00 /dev/sdc
lvresize /dev/VolGroup/lvol0 -L 12.7G

Then use resize2fs to extend the file system into the free space. You are required to fun a filesystem check first.

e2fsck -f /dev/VolGroup/lvol0
resize2fs /dev/VolGroup00/lvol0 12700M
mount -a
init 3

In my example the previous size was 7.7Gb, I added a 5Gb disk and extended to 12.7Gb.

Resize swap partition in Red Hat Linux

• Is your swap on a logical volume (LVM)? If so then skip below to the LVM section. If not then read on:
• Download gparted (Gnome Partition Manager), burn the iso and then boot into it.
• Gparted will identify the filesystems on each partition so your target will be clearly labelled as swap.
• Reize your partitions as required. Try to minimise the overall number of resize and move operations as this can take several hours to complete.

When you reboot swap not be enabled - if you check using top (or htop) or free -m, you will see 0 mb of swap available. The reason for this is the UUID of the partition changed when it was resized by GParted, and this confuses the system when it tries to mount the volumes in /etc/fstab. The solution is to relabel your swap partition, by reformatting it as swap and specifying the correct label.

fdisk -l | grep swap

Note down the device name of your swap partition. e.g:

/dev/cciss/c0d0p3 1926 3837 15358140 82 Linux swap

Make a note of the label for the swap partition from fstab:

cat /etc/fstab | grep swap

e.g.

LABEL=SW-cciss/c0d0p3 swap

Now format your partition as swap, specifiying the label exactly as shown in the fstab.

mkswap /dev/cciss/c0d0p3 -L SW-cciss/c0d0p3

You can then enable the swap space straight away by using swapon devicename, or just reboot and check with free -m again, and all should be ok.

Resizing swap on LVM
If your swap partition is on a logical volume it can be resized without rebooting your system. However you will need free space to extend into, if you do not have free space you will need to shrink another volume or add another physical disk into your volume group (see this post which explains how to do this).

cat /etc/fstab | grep swap
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

lvdisplay will show you the size of your logical volume:

lvdisplay /dev/VolGroup00/LogVol01

--- Logical volume ---
LV Name                /dev/VolGroup00/LogVol01
VG Name                VolGroup00
LV UUID                RVIFz3-B8kp-z9KV-JYtG-N997-JOQ6-ETaJaJ
LV Write Access        read/write
LV Status              available
# open                 1
LV Size                512.00 MB
Current LE             24
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           253:1

To resize an LVM you need to unmount it, or in this case swapoff.

swapoff /dev/VolGroup00/LogVol01

Resizing the volume to 768Mb (assuming you have the space to extend into)

lvresize /dev/VolGroup00/LogVol01 -L 768M
Extending logical volume LogVol01 to 768.00 MB
Logical volume LogVol01 successfully resized

swapon /dev/VolGroup00/LogVol01
free -m
total       used       free     shared    buffers     cached
Mem:           375        343         32          0         48        120
-/+ buffers/cache:        174        201
Swap:          767          0        767