There are many competing mantras about how to best deploy a ruby web application into a ‘production’ state. Some people just say, ‘start up your framework’s WEBrick server on 80 and go for it!’ Others, like myself want more stability and security — for that we turn to Passenger. Passenger provides support for Rackable ruby apps on Apache and Nginx.

I’ve always liked the Apache+Passenger configuration when installing on CentOS — I can schedule regular yum updates to upgrade Apache and gem updates to upgrade Passenger. This way, I know that I’ve got the latest and most secure versions running on my systems. With monitoring and testing, I know if something breaks and can fix it promptly.

After setting up my fresh CentOS 6 machine, I found the Nginx yum repos which will happily keep Nginx updated. Great. I then went about doing the usual rvm+ruby+gem+passenger dance. I ran passenger-install-nginx-module to initiate the passenger+nginx config and ran into this:

Nginx doesn't support loadable modules such as some other web servers do,
so in order to install Nginx with Passenger support, it must be recompiled.

Do you want this installer to download, compile and install Nginx for you?

 1. Yes: download, compile and install Nginx for me. (recommended)
    The easiest way to get started. A stock Nginx 1.0.15 with Passenger
    support, but with no other additional third party modules, will be
    installed for you to a directory of your choice.

 2. No: I want to customize my Nginx installation. (for advanced users)
    Choose this if you want to compile Nginx with more third party modules
    besides Passenger, or if you need to pass additional options to Nginx's
    'configure' script. This installer will  1) ask you for the location of
    the Nginx source code,  2) run the 'configure' script according to your
    instructions, and  3) run 'make install'.

Whichever you choose, if you already have an existing Nginx configuration file,
then it will be preserved.

Enter your choice (1 or 2) or press Ctrl-C to abort: 

‘Doesn’t support loadable modules’… ‘stock Nginx 1.0.15′… ‘compile’. Nope.

So… I can allow passenger to build itself into an old version of nginx (1.0.15) or I can download the Nginx source and recompile it with Passenger support. This is where Nginx lost me.

If I want to run a ruby app through Nginx using Passenger (or any other module that integrates Ruby support) I’ll have to recompile Nginx every time there’s an update to either system. Judging by the release frequency of Nginx, that requires recompiling at least once per month. Or, of course, I could just let my system languish and not upgrade regularly. That’ll only leave me open to vulnerabilities in either system, which is not cool.

So I’ll return to Apache+Passenger which supports dynamic module loading and independently updating components. A secure system is a happy system.

When I would try to login, it would return an error saying I specified invalid credentials. Looking at my maillog I saw the connection failing with the username “user@hostname.tld”. My mailserver is configured to use local accounts which means that authentication is done without having to specify the hostname in the address. Checking my other mail clients confirmed that I only have to specify my username to authenticate. AtMail uses the combination of username and hostname when it attempts to authenticate.

I dug around the install for a bit and found a little hidden setting in [AtMailHome]/libs/Atmail/Config.php. To configure AtMail to only send the username to the mailserver for authentication, change this line:

'mailserver_auth' => '1',

to this

'mailserver_auth' => '0',

.

A lovely undocumented setting.

Here are a list of the tricks I’ve learned so far:

1. Firewall rule numbering: The firewall documentation shows that in order to enforce rule ordering, you have to preface your declaration names with a number (e.g. 001, 002, 003, …, 999, etc). Since my firewall rules selectively allow a few ports to be open and deny all other requests, it is easier to number the declarations based off the port they declare.

   firewall { "00000 accept icmp":
     proto => "icmp",
     action => "accept"
   }
   
   firewall { "00001 accept established, related":
     state  => ['ESTABLISHED', 'RELATED'],
     proto  => 'all',
     action => 'accept',
   }
   
   firewall { "00002 accept localhost":
     source => '127.0.0.1',
     proto  => 'all',
     action => 'accept',
   }
   
   firewall { "00080 http on port 80":
     proto => "tcp",
     dport => "80",
     action => "accept"
   }
   
   firewall { "65536 drop incoming packets":
     action => 'drop'
   }

   This way the rules are loaded in port order. The first rules that setup iptables are loaded in the lowest numbers which are reserved ports and won’t likely be used by me anyway. The last rules are numbered after 65535 — the last port.

2. Auto-save configuration: I’ve also found it useful to configured every declaration to notify puppet that it should save the iptables configuration. This way, when one rule gets changed, it automatically saves the new config.

   firewall { "00080 http on port 80":
     proto => "tcp",
     dport => "80",
     action => "accept",
     notify => Exec["iptables-save"]
   }
   
   exec { "iptables-save":
     command => $operatingsystem ? {
       "debian" => "/sbin/iptables-save > /etc/iptables/rules.v4",
       /(RedHat|CentOS)/ => "/sbin/iptables-save > /etc/sysconfig/iptables",
     },
     refreshonly => true,
     notify => Service["iptables"],
   }

3. Overriding declared rules: Probably the most useful thing I figured out was how to remove firewall rules from the saved ruleset. For example, by default my httpd rule allows requests from any IP on the Internet. There is one server that should only allow requests from specific IP addresses. To configure this, I have to remove the default rule and add in new rules for port 80 in iptables. This is done using inheritence.

   class special-http-rules inherits httpd {
   
     Firewall["00080 http on port 80"] { ensure => absent }
   
     firewall { "00080 http on port 80 from special server (123.45.67.89)":
       proto => "tcp",
       dport => "80",
       source => '123.45.67.89',
       action => "accept",
       notify => Exec["iptables-save"]
     }
   }

   Using the inherits keyword on the parent class, I can call the “00080 http on port 80″ firewall declaration and ensure that it is absent from the rules applied to the node. Any server that I assign the “special-http-rules” class to will only respond to requests from the special server on port 80.

1. Using your vSphere client, increase the size of the disk attached to the VM.
2. Boot into the CentOS 6 installation disk and select ‘Rescue Mode’ from the list of options.
3. When prompted to search for LVM partitions on the disk, select ‘Skip’.
4. Type the following commands at the prompt. The following assume you have a standard CentOS LVM configuration.

   fdisk /dev/sda
   p  # Print partition table
   n  # New partition
   p  # Primary partition
   3  # ID = 3
   # When prompted, add 1 to the end block value for the sda2 partition and use it as the start of the sda3 partition.
   # Use the default for the size which should be the rest of the free space on the disk.
   t  # Change partition type
   3  # Change partition 3
   8e # Type = Linux LVM
   p  # Print partition table
   w  # Write partition table
   
   # Create a new LVM physical volume from the new partition
   lvm pvcreate /dev/sda3
   lvm pvdisplay
   
   # Mount the volume group
   lvm vgscan
   lvm vgchange -ay
   # You will see the name of the volume group that is activated.  Usually something like 'vg_hostname'.
   
   # Extend the volume group with the new physical volume.  Be sure to substitute the name of your volume group in the command below.
   lvm vgextend /dev/vg_hostname /dev/sda3
   
   # Extend the logical volume to include 100% of the free space on the volume group.
   lvm lvextend /dev/vg_hostname/lv_root /dev/sda3
   
   # Mount the volume group
   lvm vgscan
   lvm vgchange -ay
   
   # Run a filesystem check on the newly expanded disk
   e2fsck -f /dev/vg_hostname/lv_root
   
   # Resize the filesystem to use the entire disk
   resize2fs /dev/vg_hostname/lv_root

5. After rebooting, you can confirm the final size of your disk using:

   df -h

As always, have a backup of your data.

Note: I spent a long time trying to figure this out. The key is that the ‘banner’ property is the String that the monitor daemon looks for to confirm that it’s communicating with an SSH server. I erroneously put ‘SSH-1920′ which would never match the SSH server’s response banner.

<!-- capsd-configuration.xml -->
<protocol-plugin protocol="SSH-1920" class-name="org.opennms.netmgt.capsd.plugins.SshPlugin" scan="on">
  <property key="banner" value="SSH" />
  <property key="port" value="1920" />
  <property key="timeout" value="3000" />
  <property key="retry" value="1" />
</protocol-plugin>

<!-- poller-configuration.xml -->
<service name="SSH-1920" interval="300000" user-defined="false" status="on">
  <parameter key="retry" value="1"/>
  <parameter key="banner" value="SSH"/>
  <parameter key="port" value="1920"/>
  <parameter key="timeout" value="3000"/>
  <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"/>
  <parameter key="rrd-base-name" value="ssh-1920"/>
  <parameter key="ds-name" value="ssh-1920"/>
</service>

<!-- ... and farther down below ...  -->

<monitor service="SSH-1920" class-name="org.opennms.netmgt.poller.monitors.SshMonitor"/>

Message: dualvar is only available with the XS version of Scalar::Util at /usr/lib/perl5/vendor_perl/5.8.8/IO/Socket/SSL.pm line 19

This error was caused by an upgrade to the Compress::Zlib package. The newer version of the package isn’t compiled with XS support. I found some information about this error from this OTRS mailing list post from 2010 regarding RHEL. Looks like it takes about a year for RHEL packages to make it downstream to CentOS.

To fix the error, I ran the following from the command line:

perl -MCPAN -e "CPAN::Shell->force(qw(install Scalar::Util));"

After letting the cronjobs rerun, the error was gone and tickets successfully processed again.

When trying to install the latest nVidia drivers on my Fedora 15 workstation, I kept getting errors about the default nouveau video driver being loaded into the kernel. The nVidia installer creates a modprobe config file that is supposed to prevent that module from being loaded but it doesn’t work fully.

To really disable the nouveau driver, you need to edit the grub config file and add the following to the end of the kernel init line:

rdblacklist=nouveau nouveau.modeset=0

For example, your resulting grub.conf file will look like this:

title Fedora (2.6.40.6-0.fc15.x86_64)
        root (hd0,1)
        kernel /vmlinuz-2.6.40.6-0.fc15.x86_64 ro root=/dev/mapper/vg_cline-lv_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb vga=794 quiet rdblacklist=nouveau nouveau.modeset=0
        initrd /initramfs-2.6.40.6-0.fc15.x86_64.img

One thing that is available for Macs is called ‘DVD or CD Sharing’, also called ‘Remote Disk’. You can find the default instructions on how to set it up over at Apple’s support article. The instructions include how to install the DVD or CD Sharing client on a Windows PC for sharing to a Mac.

What isn’t included in those instructions is the fact that by default, only Macbook Airs and Mac Minis support Remote Disk out of the box. I guess this is because they are the only two Mac models that don’t have optical drives.

Once DVD or CD Sharing has been enabled on a remote system (and the firewalls between the two systems correctly configured), the following two lines will activate the ‘Remote Disk’ option in the Finder window. Run them in a terminal window, then restart your computer.

defaults write com.apple.NetworkBrowser EnableODiskBrowsing -bool true
defaults write com.apple.NetworkBrowser ODSSupported -bool true

After boot, you should see the following options in your Finder.

Note: My PC has two disk drives shared in the above picture.

(* Munin Node module for Augeas *)

module MuninNode =
  autoload xfm

  let record =
    let value = store /[^ \t\n]+([ \t]+[^ \t\n]+)*/ in
      [ key Rx.word . Sep.space . value . Util.eol ]

  let lns = (record | Util.comment | Util.empty) *

  let filter = incl "/etc/munin/munin-node.conf" . Util.stdexcl

  let xfm = transform lns filter

1. First, you’ll want to create the new VM that you’ll be migrating to; we’ll call this the DestinationVM. Just configure the hardware — don’t install an OS.
2. Next, boot the VM from the CentOS installation disk and enter rescue mode. At the prompt, type

   linux rescue

3. Configure the network interfaces and when it asks to search for installations, allow it to initialize the disks in the VM. There isn’t an install present, but we need to setup the disks to perform the copy.
4. When the VM has booted run

   nc -l -p 6501 | dd of=/dev/sda

   This will start the nc daemon and output the data to the /dev/sda disk. Make sure to change the destination disk if it is different than /dev/sda.

5. On the physical machine run

   dd if=/dev/sda | nc <ip-of-VM> 6501

   Tip: Start the command in a screen session if you might be disconnected from the server during the transfer. It may take a while.

6. You won’t see anything until the transfer completes. It took 6 hours for a 250GB drive to copy for me. YMMV
7. Once the transfer is complete, reboot the VM and you should be good to go!

Sadly, when I rebooted the VM, I encountered the following error:

  Reading all physical volumes. This may take awhile...
  Volume group "VolGroup00" not found
Unable to access resume device (/dev/VolGroup00/LogVol01)
mount: could not find filesystem '/dev/root'
setuproot: moving /dev failed: No such file or directory
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

This error likely means that the kernel copied from the old physical system doesn’t have drivers to support the disk hardware in the virtual machine. Because of this, the LVM configuration isn’t loading properly. The easiest way to resolve this is to reinstall the kernel using yum.

1. Boot the VM into the CentOS install CD and at the prompt, type

   linux rescue

2. Enable and configure networking and allow the mounting of the local installation. Make sure to mount is as read/write — we’ll be making changes to it.

3. chroot /mnt/sysconfig
   yum remove kernel
   yum install kernel
   exit
   exit

   Note, it’s OK to remove all versions of the kernel. Just make sure you install one before you reboot.

4. The system will reboot, once it does, let it load into the OS. If everything went OK, your new system will be up and running!