Archive for the 'Enterprise Linux' Category

Managing multiple PuTTys

Published
by
ben
on September 9, 2009
in Enterprise Linux
. 0 Comments

I don't have any Linux cluster servers to look after but do manage 2 Linux server farms. They are similar to clusters in that they are both groups of RHEL servers that all have to have an identical config. The larger of these farms is 12 RHEL 5.3 servers, and to roll out changes to them all, I wanted to be able to make the change once, and after it was verified, make the same change on the other 11 servers. I started off by writing this script on server1:

#!/bin/sh

echo -n "Enter command to run (on one line): "
read STRING

for SERVER in server2 server3 server4 server5 server6 etc; do

echo -e "\033[1;31m$SERVER says:\033[m"
ssh $SERVER "$STRING"

done

echo -n "Do you want to run the command locally? (y/n) :"
read ANSWER

case "$ANSWER" in

y|Y) echo -e "\033[1;31mlocalhost says:\033[m"; $STRING ;;
N|n) exit 1 ;;
esac

For this to work I had to create ssh keys on each server using:

ssh-keygen -t dsa

And then install the newly generated key (~/ssh/id_dsa.pub) into the authorized_keys file on server1. This works for running simple commands one at a time.

For other tasks its sometimes necessary to manage multiple ssh sessions at once, for example to monitor resources using htop or tailing log files. On Linux you can use ClusterSSH (cssh) but this turned out to be a royal pain in the butt to get working on CentOS or RHEL. It worked OK in an Ubuntu VM but was a bit clunky and I felt there would be a better way of managing multiple PuTTy windows, since I am using a Windows 7 laptop for my day-to-day stuff.

There are quite a few goodies for this on the Links page on the Putty website and this is where I found Putty Command Sender. Quite simply you type your command into the command sender, and much like clusterssh, it sends it to all the putty windows you have open.

puttys
Putty Command Sender

It's not so great for editing files in Vi on 12 servers at once but it is possible - you can send cursor movements as well as lines of code or single commands. The only thing you have to watch out for in PuttyCS is that all Putty windows have updated before you start typing your next command, otherwise the last window to update may miss the first few characters of the next command.

Finally to launch my sessions in groups I'm using Putty Session Manager. Other alternatives are available but I found this one to be the best. It's lightweight nature fits in with Putty nicely.

Install RHEL with Kickstart and no DHCP

Published
by
ben
on June 25, 2009
in Enterprise Linux
. 0 Comments

I've been installing 16 RHEL 5.3 servers on a DR site, owned by our sister company. There is no DHCP server on the subnet which has stopped us using HP Rapid Deployment Pack to roll out the servers, but it was still possible to use Kickstart, by specifying the IP information as a parameter when booting from the RHEL CD (disk 1). The syntax goes like this:

boot: linux ks=http://server/kickstart.cfg ksdevice=eth0 ip=xx.xx.xx.xx netmask=xx.xx.xx.xx gateway=xx.xx.xx.xx dns=xx.xx.xx.xx

DNS servers can be comma separated if you want to specify more than one.

Set up a ‘honeypot’ router using Netfilter (iptables)

Published
by
ben
on June 4, 2009
in Enterprise Linux
. 0 Comments

We're in the process of moving all servers from a legacy flat network to a new server VLAN. Each time a server is moved and is assigned a new IP address, there is a risk that some applications in the building might still try and connect to the server's old IP address rather than the DNS name. To log any attempts to reach the old IP address I've set up a 'honeypot' router using iptables and CentOS (as a small VMware machine). Each time a server is moved, the old server IP is added to the CentOS machine and and 2 rules are added in the iptables firewall to drop and log the incoming IP connections. The failed connections would usually logged to the console and the 'messages' log file, but alternatively these could be sent to a remote syslog server by specifying:

kern.*        @xx.xx.xx.xx

in the syslog.conf. In our case we're logging to our Cacti server and using the Cacti syslog plugin (from here) to view our Linux server and network switch syslogs in a central location.

iptables configuration
In a default installation of CentOS, iptables is already installed and running by default. There are only 2 changes to make; The first is to bind your additional IP address to your network card (usually eth0). This is done by creating a virtual device called eth0:1, by inserting this text into new file etc/sysconfig/network-scripts/ifcfg-eth0:1 (10.10.2.17 is the destination address of the incoming traffic that I want to log):

IPADDR=10.10.2.17
NETMASK=255.255.0.0

And then bringing up the virtual device using:

ifconfig eth0:1 up

In the iptables config file (etc/sysconfig/iptables) I added these lines in the RH-Firewall-1-INPUT chain:

-A RH-Firewall-1-INPUT -d 10.10.2.17 -j LOG --log-level 4 --log-prefix "OLD SRV1"
-A RH-Firewall-1-INPUT -d 10.10.2.17 -j DROP

Any traffic to 10.10.2.17 will now be logged in messages and the lines will be prefixed with "OLD SRV1".

Note about file paths: I'm getting an odd error when posting Linux file paths in Wordpress - putting a leading / in the path gives a 404 error. The paths I mentioned above should have a leading forwardslash "/".

Get system stats automatically on SSH login

Published
by
ben
on April 10, 2009
in Enterprise Linux and Scripting
. 0 Comments

I noticed in recent versions of Ubuntu you get some system stats in a banner message when you connect using SSH. I thought this was pretty useful so have implemented my own version on our Red Hat servers at work.

This runs every 5 minutes as a cron job and updates the file /etc/motd (Message Of The Day) which is shown when a user logs in. Our servers already have a 5 line banner message with information including the server name and purpose of the server, hence the first 5 lines being saved and readded into a new /etc/motd each time.

#!/bin/bash
cat /etc/motd | head -n 5 > /tmp/file && cat /tmp/file > /etc/motd

CPUTIME=$(ps -eo pcpu | awk 'NR>1' | awk '{tot=tot+$1} END {print tot}')
CPUCORES=$(cat /proc/cpuinfo | grep -c processor)

echo "System summary (collected `date`)

 - CPU Usage (total average) = `echo $CPUTIME / $CPUCORES | bc`%
 - Memory free (real)        = `free -m | head -n 2 | tail -n 1 | awk {'print $4'}` Mb
 - Memory free (cache)       = `free -m | head -n 3 | tail -n 1 | awk {'print $3'}` Mb
 - Swap in use               = `free -m | tail -n 1 | awk {'print $3'}` Mb
" >> /etc/motd

Now when we log in we get a summary like this:

SSH Server Stats

Working with Logical Volumes

Published
by
ben
on April 1, 2009
in Enterprise Linux and VMware
. 0 Comments

I'm actually coming around to Linux LVM - once you get the hang of the concepts and the associated commands it can be a straightforward exercise to extend your existing volumes after adding new physical disks. This differs from software RAID, as you have the ability to lay an LVM filesystem over a single disk and later take advantage of the LVM commands to resize your volumes if you so desire.

I recently was confronted with a VM that was out of space on /usr/local. The filesystem was already using LVM so I just added a new virtual disk and stretched the /usr/local volume over the new disk. The whole process is even easier in VMware as you can add the new disk while the machine is running and run through the whole process without a reboot, providing you don't have daemons or processes running in /usr/local that stop it from being unmounted. Here's how I did it:

init 1
umount /usr/local

Going to runlevel 1 may not always be necessary but was in my case. pvcreate enables the new physical disk for use with LVM. Then vgextend extends the volume group, and lvresze resizes the logical volume. In my case the new disk that was added became known to the system as /dev/sdc.

pvcreate /dev/sdc
vgextend VolGroup00 /dev/sdc
lvresize /dev/VolGroup/lvol0 -L 12.7G

Then use resize2fs to extend the file system into the free space. You are required to fun a filesystem check first.

e2fsck -f /dev/VolGroup/lvol0
resize2fs /dev/VolGroup00/lvol0 12700M
mount -a
init 3

In my example the previous size was 7.7Gb, I added a 5Gb disk and extended to 12.7Gb.

Resize swap partitions on Red Hat Linux

Published
by
ben
on August 12, 2008
in Enterprise Linux
. 2 Comments

First of all a little disclaimer - although I've done this a few times and never had any problems, one should never consider resizing partitions a completely foolproof exercise. Things can go wrong. With regards to swap though, don't worry if your swap partition is destroyed as this will not harm your system - most healthy systems will not ever need swap and unlike Windows, Linux only starts using swap when it needs it and can quite easily survive without it. You can trash it and format it as many times as you like as long as it's not in use. And of course take backups of any important data first!

Resize swap partition in Red Hat Linux

  • Is your swap on a logical volume (LVM)? If so then skip below to the LVM section. If not then read on:
  • Download gparted (Gnome Partition Manager), burn the iso and then boot into it.
  • Gparted will identify the filesystems on each partition so your target will be clearly labelled as swap.
  • Reize your partitions as required. Try to minimise the overall number of resize and move operations as this can take several hours to complete.

When you reboot swap not be enabled - if you check using top (or htop) or free -m, you will see 0 mb of swap available. The reason for this is the UUID of the partition changed when it was resized by GParted, and this confuses the system when it tries to mount the volumes in /etc/fstab. The solution is to relabel your swap partition, by reformatting it as swap and specifying the correct label.

fdisk -l | grep swap

Note down the device name of your swap partition. e.g:

/dev/cciss/c0d0p3 1926 3837 15358140 82 Linux swap

Make a note of the label for the swap partition from fstab:

cat /etc/fstab | grep swap

e.g.

LABEL=SW-cciss/c0d0p3 swap

Now format your partition as swap, specifiying the label exactly as shown in the fstab.

mkswap /dev/cciss/c0d0p3 -L SW-cciss/c0d0p3

You can then enable the swap space straight away by using swapon devicename, or just reboot and check with free -m again, and all should be ok.

Resizing swap on LVM
If your swap partition is on a logical volume it can be resized without rebooting your system. However you will need free space to extend into, if you do not have free space you will need to shrink another volume or add another physical disk into your volume group (see this post which explains how to do this).

cat /etc/fstab | grep swap
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

lvdisplay will show you the size of your logical volume:

lvdisplay /dev/VolGroup00/LogVol01

--- Logical volume ---
LV Name                /dev/VolGroup00/LogVol01
VG Name                VolGroup00
LV UUID                RVIFz3-B8kp-z9KV-JYtG-N997-JOQ6-ETaJaJ
LV Write Access        read/write
LV Status              available
# open                 1
LV Size                512.00 MB
Current LE             24
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           253:1

To resize an LVM you need to unmount it, or in this case swapoff.

swapoff /dev/VolGroup00/LogVol01

Resizing the volume to 768Mb (assuming you have the space to extend into)

lvresize /dev/VolGroup00/LogVol01 -L 768M
Extending logical volume LogVol01 to 768.00 MB
Logical volume LogVol01 successfully resized

swapon /dev/VolGroup00/LogVol01
free -m
total       used       free     shared    buffers     cached
Mem:           375        343         32          0         48        120
-/+ buffers/cache:        174        201
Swap:          767          0        767

Oracle RAC, CentOS and GFS

Published
by
ben
on June 24, 2008
in Enterprise Linux
. 0 Comments

One thing I'm currently working on is a 2 node CentOS cluster using GFS as the shared file system, with the storage on a fibre attached HP MSA SAN. This is a proof of concept before investing in RHEL AS licences (which will be the better part of £2000). CentOS is free and is intended to clone RHEL as closly as possible, removing the Red Hat branding and of course professional support. The cluster suite and GFS are available in CentOS as they are both open source.

The hardware I am using is 2 HP DL580 G4 servers with 12GB of RAM, Qlogic HBA's, and an MSA1000 SAN. I'm using this guide on Red Hat's website and have been making my own notes along the way. I've got as far as installing cluster suite on one node but I'm unable to choose the grand unified lock manager as it doesn't appear as an option. I expect this may take a bit of reading to get working ... I'll post a mini howto here when I make some decent progress.

A while ago I undertook a similar exercise and had problems getting the HP ACU to see the SAN under Red Hat, I had to get HP involved as there were specific versions of the kernel and drivers required for it to work properly. I've noticed I have the same problem time time, but now I am on CentOS and it's the x64bit version, so it looks like I will have to experiment with driver versions and see what I can get to work without any support this time.

Red Hat Summit 2008 – closing post

Published
by
ben
on June 20, 2008
in Enterprise Linux
. 0 Comments

In short - it's been good, a valuable use of my time. The presentation throughout the summit was of a very high standard, and this included the catering too.

If I had to summarise the summit I would do so using the following 5 buzzwords:

1. Virtualisation. Guess what ...? Red Hat has their own virtualisation product. It looks very promising but if I am brutally honest (and Red Hat would kill me for saying this), they are pretty much in the same boat as Microsoft. By that I mean they are leagues behind what VMWare are offering right now. But it wasn't just Red Hat banging on about virtualisation. All the vendors in the exhibition hall were touting their products with the big V in mind.

2. Green/Energy saving. Everyone is working towards making their products consume less energy, while giving the user much more back. Red Hat and intel are working towards more closely unifying hardware and software, so that their OS can have much greater control of how much power a device on the system should consume.

3. Cloud computing. Everyone was talking about this. Offload your computing power to other computers in the cloud when you have extreme amounts of processing to do. It's a bit like clustering except you only dip into the cloud when you need to, and the computers in the cloud don't have to be on your network.

4. Red Hat Satellite. A lot of the speakers talked about how they use Satellite to administer their Red Hat servers. And of course there was the big announcement about Spacewalk.

5. Open Source. Red Hat love talking about it and they are very proud about being open source. They even talked about the importance of CentOS and Ubuntu, which surprised me! Moreover, I was astonished that CentOS will be a supported product by Spacewalk. When questioned about Ubuntu, the reply was Ubuntu is not a rival of Red Hat ... as it's going to take more than one player to topple Microsoft!

I'm off home now, but first I'm due a visit to the duty free shop for an extremely large bottle of whisky. Thank you very much HP (but sorry, I'm not keeping the t-shirt) :)

Spacewalk – Red Hat Satellite for free!

Published
by
ben
on June 19, 2008
in Enterprise Linux
. 3 Comments

Red Hat have made another important announcement at this years Summit - RH Satellite has been made open source! At first I was excited, because:

  • Red Hat customers can save hundreds of dollars as it's now free - you don't have to buy an RHAS licence for a RH Satellite (that is, unless you want support for it).
  • You can run it on Fedora!
  • Community users can add to the code and make it better. By this I mean they can (and will) fix bugs and add enhancements.
  • This is beneficial for Red Hat as these improvements can (and will in due course) be loaded back into the paid RHAS Satellite packages.

BUT .... what you can't do with Spacewalk is update RHEL systems with it - you can only use it to manage Fedora and CentOS systems for now (according to the FAQ). So it's no use to me at the moment as we're a Red Hat shop only .... I guess I'll be waiting for my RHAS system after all.

Red Hat Summit – Day 2

Published
by
ben
on June 19, 2008
in Enterprise Linux
. 0 Comments

Red Hat logoThere were 2 seminars that stood out for me today after Joel Cowen's (Executive producer of The Simpsons) amusing Keynote speech - did you know The Simpsons is rendered on Red Hat?

The first of the noteworthy seminars was Michael Stahnkage on how to manage RHEL updates. He's one of the few speakers who kept it human and maintained an interesting session throughout with a mix of his own experiences, some real world scenarios, and a bit of dry humour to keep the audience awake. I made plenty of notes which I will be taking with me to begin planning an update strategy for my company's Red Hat servers. For my own reference - the slides are here (in Open Office format).

The other session of interest was a business pitch from IBM (IMO, to a room full of the wrong people) on Red Hat as a desktop alternative for the enterprise. The cost savings are incredible particularly when you replace Microsoft Office with Open Office (or Symphony, as IBM call it). It started to drag towards the end but got more interesting when someone asked about the new Office 2007 formats and compatibility with Symphony, just because we got to see some very senior people from IBM squirming and getting very defensive. They also didn't have an answer for any decent Visio alternative on Linux. I thought their strategy was very interesting and an honourable thing to do, but I think the average consumer will be difficult to migrate from Windows to something alien, especially while sharing documents with Windows users is not straightforward, and application compatibility with browsers other than IE is still an important issue.

Bounty!
I have accumulated a lot of free stuff over the last 2 days:

  • A Tux foam penguin
  • 1GB HP Branded USB key
  • Flashing bouncy ball and stress ball
  • Red Hat baseball cap
  • HP Lip ice sunscreen!
  • HP T-shirt (XL - looks like a tent on me)
  • r1soft.com T-Shirt (XXL - blimey, how big??)
  • zenoss T-Shirt (Medium - which is actually alright!)
  • Red Hat bag and USB key which all attendees received

The HP T-Shirt also had a secondary purpose. If you wear one around the Summit there is a chance to win iPods and iPhones (and other Apple goodies), if you get stopped by HP and correctly answer some questions about ICE-Linux - an HP SIM plugin for managing Linux servers. I opted into this and wore my huge T-Shirt, and when I was stopped and answered my question correctly .... I won $50 in Amex vouchers! Nice - free duty free alcohol for me on the way home!

ZenOss was an interesting product as well, it's a monitoring solution based on Cacti and RRD but with some extra features and paid support thrown in (like viewing your servers on a Google Maps mashup). Really I'm not sure if it's worth it - if you already use Cacti and have time to invest in beefing up your existing environment with all the best plugins available (like threshold monitoring, alerting, syslogs), then you will get pretty close to a ZenOss install out of the box. But that said, if you don't have time or want professional and friendly assistance to set up a comprehensive monitoring solution, then ZenOss could be for you.

In general the exhibition part of the Summit was a bit small, but I've found all of the vendors very friendly and they wanted to discuss how I used Open Source (and Red Hat) in my own workplace, avoiding the hard sell of their own products.