• Send advance warning to users
• Log on to site, edit params and enter some text under 'shutdown site'
• Backup db using this commmand

mysqldump --opt -u bugs -p bugs > bugs_backup_$(date +%d%m%y).sql

• Download and extract bugzilla 3.4.6

cd ~
wget http://ftp.mozilla.org/pub/mozilla.org/webtools/bugzilla-3.4.6.tar.gz
tar fvzx bugzilla-3.4.6.tar.gz
cd /var/www/html/
mv bugzilla bugzilla.old
cp -r ~/bugzilla-3.4.6 ./bugzilla
cp -r bugzilla.old/localconfig* bugzilla

Later you will install the DBD::mysql perl module which requires mysql header files. So if you don't have mysql-devel, install it now. Also the Image::Magick perl module may fail to build from cpan (this is what happened to me), this can be installed by yum as an alternative.

yum install mysql-devel ImageMagick-perl.i386
cd bugzilla
./checksetup.pl

checksetup.pl will tell you to install missing perl modules using cpan, however this requires an internet connection. I used ntlmaps to proxy through our MS ISA server as my bugzilla server doesn't have Internet access. I got ntlmaps from another server on the network:

scp -r root@bugtest:/root/ntlm* .
cd ntlmaps-0.9.9.0.1
nohup python main.py
export http_proxy=http://localhost:5865
export ftp_proxy=ftp://localhost:5865
cd /var/www/html/bugzilla

Install optional updates:

# failed  --> /usr/bin/perl install-module.pl Image::Magick
# skipped --> /usr/bin/perl install-module.pl Authen::Radius
/usr/bin/perl install-module.pl SOAP::Lite
/usr/bin/perl install-module.pl TheSchwartz
/usr/bin/perl install-module.pl Daemon::Generic
/usr/bin/perl install-module.pl SOAP::Lite

Install compulsary updates:

/usr/bin/perl install-module.pl DBD::mysql
/usr/bin/perl install-module.pl CGI
/usr/bin/perl install-module.pl Digest::SHA
/usr/bin/perl install-module.pl DateTime
/usr/bin/perl install-module.pl DateTime::TimeZone
/usr/bin/perl install-module.pl Template
/usr/bin/perl install-module.pl Email::MIME
/usr/bin/perl install-module.pl Email::MIME::Encodings

I also needed these perl modules, even though they weren't listed by checksetup.pl:

/usr/bin/perl install-module.pl DateTime::Locale
/usr/bin/perl install-module.pl List::MoreUtils

Finished with ntlmaps now, it can be stopped

kill $(ps x | grep main.py | grep -v grep | awk {'print $1'})
export http_proxy=
export ftp_proxy=

Run checksetup again, this bit can take some time while the db is upgraded:

./checksetup.pl

• Fix any more problems, run again if required.
• Once finished log into bugzilla, put maintenance page back up. This option has now moved to Administration
• Run sanity check, fix any problems.
• Change any css and templates that you lost in the upgrade.
• Reopen the site for logons when you are ready.

#!/bin/sh

echo -n "Enter command to run (on one line): "
read STRING

for SERVER in server2 server3 server4 server5 server6 etc; do

echo -e "\033[1;31m$SERVER says:\033[m"
ssh $SERVER "$STRING"

done

echo -n "Do you want to run the command locally? (y/n) :"
read ANSWER

case "$ANSWER" in

y|Y) echo -e "\033[1;31mlocalhost says:\033[m"; $STRING ;;
N|n) exit 1 ;;
esac

For this to work I had to create ssh keys on each server using:

ssh-keygen -t dsa

And then install the newly generated key (~/ssh/id_dsa.pub) into the authorized_keys file on server1. This works for running simple commands one at a time.

For other tasks its sometimes necessary to manage multiple ssh sessions at once, for example to monitor resources using htop or tailing log files. On Linux you can use ClusterSSH (cssh) but this turned out to be a royal pain in the butt to get working on CentOS or RHEL. It worked OK in an Ubuntu VM but was a bit clunky and I felt there would be a better way of managing multiple PuTTy windows, since I am using a Windows 7 laptop for my day-to-day stuff.

There are quite a few goodies for this on the Links page on the Putty website and this is where I found Putty Command Sender. Quite simply you type your command into the command sender, and much like clusterssh, it sends it to all the putty windows you have open.

Putty Command Sender

It's not so great for editing files in Vi on 12 servers at once but it is possible - you can send cursor movements as well as lines of code or single commands. The only thing you have to watch out for in PuttyCS is that all Putty windows have updated before you start typing your next command, otherwise the last window to update may miss the first few characters of the next command.

Finally to launch my sessions in groups I'm using Putty Session Manager. Other alternatives are available but I found this one to be the best. It's lightweight nature fits in with Putty nicely.

boot: linux ks=http://server/kickstart.cfg ksdevice=eth0 ip=xx.xx.xx.xx netmask=xx.xx.xx.xx gateway=xx.xx.xx.xx dns=xx.xx.xx.xx

DNS servers can be comma separated if you want to specify more than one.

kern.*        @xx.xx.xx.xx

in the syslog.conf. In our case we're logging to our Cacti server and using the Cacti syslog plugin (from here) to view our Linux server and network switch syslogs in a central location.

iptables configuration
In a default installation of CentOS, iptables is already installed and running by default. There are only 2 changes to make; The first is to bind your additional IP address to your network card (usually eth0). This is done by creating a virtual device called eth0:1, by inserting this text into new file etc/sysconfig/network-scripts/ifcfg-eth0:1 (10.10.2.17 is the destination address of the incoming traffic that I want to log):

IPADDR=10.10.2.17
NETMASK=255.255.0.0

And then bringing up the virtual device using:

ifconfig eth0:1 up

In the iptables config file (etc/sysconfig/iptables) I added these lines in the RH-Firewall-1-INPUT chain:

-A RH-Firewall-1-INPUT -d 10.10.2.17 -j LOG --log-level 4 --log-prefix "OLD SRV1"
-A RH-Firewall-1-INPUT -d 10.10.2.17 -j DROP

Any traffic to 10.10.2.17 will now be logged in messages and the lines will be prefixed with "OLD SRV1".

Note about file paths: I'm getting an odd error when posting Linux file paths in WordPress - putting a leading / in the path gives a 404 error. The paths I mentioned above should have a leading forwardslash "/".

This runs every 5 minutes as a cron job and updates the file /etc/motd (Message Of The Day) which is shown when a user logs in. Our servers already have a 5 line banner message with information including the server name and purpose of the server, hence the first 5 lines being saved and readded into a new /etc/motd each time.

#!/bin/bash
cat /etc/motd | head -n 5 > /tmp/file && cat /tmp/file > /etc/motd

CPUTIME=$(ps -eo pcpu | awk 'NR>1' | awk '{tot=tot+$1} END {print tot}')
CPUCORES=$(cat /proc/cpuinfo | grep -c processor)

echo "System summary (collected `date`)

 - CPU Usage (total average) = `echo $CPUTIME / $CPUCORES | bc`%
 - Memory free (real)        = `free -m | head -n 2 | tail -n 1 | awk {'print $4'}` Mb
 - Memory free (cache)       = `free -m | head -n 3 | tail -n 1 | awk {'print $3'}` Mb
 - Swap in use               = `free -m | tail -n 1 | awk {'print $3'}` Mb
" >> /etc/motd

Now when we log in we get a summary like this:

I recently was confronted with a VM that was out of space on /usr/local. The filesystem was already using LVM so I just added a new virtual disk and stretched the /usr/local volume over the new disk. The whole process is even easier in VMware as you can add the new disk while the machine is running and run through the whole process without a reboot, providing you don't have daemons or processes running in /usr/local that stop it from being unmounted. Here's how I did it:

init 1
umount /usr/local

Going to runlevel 1 may not always be necessary but was in my case. pvcreate enables the new physical disk for use with LVM. Then vgextend extends the volume group, and lvresze resizes the logical volume. In my case the new disk that was added became known to the system as /dev/sdc.

pvcreate /dev/sdc
vgextend VolGroup00 /dev/sdc
lvresize /dev/VolGroup/lvol0 -L 12.7G

Then use resize2fs to extend the file system into the free space. You are required to fun a filesystem check first.

e2fsck -f /dev/VolGroup/lvol0
resize2fs /dev/VolGroup00/lvol0 12700M
mount -a
init 3

In my example the previous size was 7.7Gb, I added a 5Gb disk and extended to 12.7Gb.

Resize swap partition in Red Hat Linux

• Is your swap on a logical volume (LVM)? If so then skip below to the LVM section. If not then read on:
• Download gparted (Gnome Partition Manager), burn the iso and then boot into it.
• Gparted will identify the filesystems on each partition so your target will be clearly labelled as swap.
• Reize your partitions as required. Try to minimise the overall number of resize and move operations as this can take several hours to complete.

When you reboot swap not be enabled - if you check using top (or htop) or free -m, you will see 0 mb of swap available. The reason for this is the UUID of the partition changed when it was resized by GParted, and this confuses the system when it tries to mount the volumes in /etc/fstab. The solution is to relabel your swap partition, by reformatting it as swap and specifying the correct label.

fdisk -l | grep swap

Note down the device name of your swap partition. e.g:

/dev/cciss/c0d0p3 1926 3837 15358140 82 Linux swap

Make a note of the label for the swap partition from fstab:

cat /etc/fstab | grep swap

e.g.

LABEL=SW-cciss/c0d0p3 swap

Now format your partition as swap, specifiying the label exactly as shown in the fstab.

mkswap /dev/cciss/c0d0p3 -L SW-cciss/c0d0p3

You can then enable the swap space straight away by using swapon devicename, or just reboot and check with free -m again, and all should be ok.

Resizing swap on LVM
If your swap partition is on a logical volume it can be resized without rebooting your system. However you will need free space to extend into, if you do not have free space you will need to shrink another volume or add another physical disk into your volume group (see this post which explains how to do this).

cat /etc/fstab | grep swap
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

lvdisplay will show you the size of your logical volume:

lvdisplay /dev/VolGroup00/LogVol01

--- Logical volume ---
LV Name                /dev/VolGroup00/LogVol01
VG Name                VolGroup00
LV UUID                RVIFz3-B8kp-z9KV-JYtG-N997-JOQ6-ETaJaJ
LV Write Access        read/write
LV Status              available
# open                 1
LV Size                512.00 MB
Current LE             24
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           253:1

To resize an LVM you need to unmount it, or in this case swapoff.

swapoff /dev/VolGroup00/LogVol01

Resizing the volume to 768Mb (assuming you have the space to extend into)

lvresize /dev/VolGroup00/LogVol01 -L 768M
Extending logical volume LogVol01 to 768.00 MB
Logical volume LogVol01 successfully resized

swapon /dev/VolGroup00/LogVol01
free -m
total       used       free     shared    buffers     cached
Mem:           375        343         32          0         48        120
-/+ buffers/cache:        174        201
Swap:          767          0        767

The hardware I am using is 2 HP DL580 G4 servers with 12GB of RAM, Qlogic HBA's, and an MSA1000 SAN. I'm using this guide on Red Hat's website and have been making my own notes along the way. I've got as far as installing cluster suite on one node but I'm unable to choose the grand unified lock manager as it doesn't appear as an option. I expect this may take a bit of reading to get working ... I'll post a mini howto here when I make some decent progress.

A while ago I undertook a similar exercise and had problems getting the HP ACU to see the SAN under Red Hat, I had to get HP involved as there were specific versions of the kernel and drivers required for it to work properly. I've noticed I have the same problem time time, but now I am on CentOS and it's the x64bit version, so it looks like I will have to experiment with driver versions and see what I can get to work without any support this time.

If I had to summarise the summit I would do so using the following 5 buzzwords:

1. Virtualisation. Guess what ...? Red Hat has their own virtualisation product. It looks very promising but if I am brutally honest (and Red Hat would kill me for saying this), they are pretty much in the same boat as Microsoft. By that I mean they are leagues behind what VMWare are offering right now. But it wasn't just Red Hat banging on about virtualisation. All the vendors in the exhibition hall were touting their products with the big V in mind.

2. Green/Energy saving. Everyone is working towards making their products consume less energy, while giving the user much more back. Red Hat and intel are working towards more closely unifying hardware and software, so that their OS can have much greater control of how much power a device on the system should consume.

3. Cloud computing. Everyone was talking about this. Offload your computing power to other computers in the cloud when you have extreme amounts of processing to do. It's a bit like clustering except you only dip into the cloud when you need to, and the computers in the cloud don't have to be on your network.

4. Red Hat Satellite. A lot of the speakers talked about how they use Satellite to administer their Red Hat servers. And of course there was the big announcement about Spacewalk.

5. Open Source. Red Hat love talking about it and they are very proud about being open source. They even talked about the importance of CentOS and Ubuntu, which surprised me! Moreover, I was astonished that CentOS will be a supported product by Spacewalk. When questioned about Ubuntu, the reply was Ubuntu is not a rival of Red Hat ... as it's going to take more than one player to topple Microsoft!

I'm off home now, but first I'm due a visit to the duty free shop for an extremely large bottle of whisky. Thank you very much HP (but sorry, I'm not keeping the t-shirt)

• Red Hat customers can save hundreds of dollars as it's now free - you don't have to buy an RHAS licence for a RH Satellite (that is, unless you want support for it).
• You can run it on Fedora!
• Community users can add to the code and make it better. By this I mean they can (and will) fix bugs and add enhancements.
• This is beneficial for Red Hat as these improvements can (and will in due course) be loaded back into the paid RHAS Satellite packages.

BUT .... what you can't do with Spacewalk is update RHEL systems with it - you can only use it to manage Fedora and CentOS systems for now (according to the FAQ). So it's no use to me at the moment as we're a Red Hat shop only .... I guess I'll be waiting for my RHAS system after all.