There are many competing mantras about how to best deploy a ruby web application into a ‘production’ state. Some people just say, ‘start up your framework’s WEBrick server on 80 and go for it!’ Others, like myself want more stability and security — for that we turn to Passenger. Passenger provides support for Rackable ruby apps on Apache and Nginx.

I’ve always liked the Apache+Passenger configuration when installing on CentOS — I can schedule regular yum updates to upgrade Apache and gem updates to upgrade Passenger. This way, I know that I’ve got the latest and most secure versions running on my systems. With monitoring and testing, I know if something breaks and can fix it promptly.

After setting up my fresh CentOS 6 machine, I found the Nginx yum repos which will happily keep Nginx updated. Great. I then went about doing the usual rvm+ruby+gem+passenger dance. I ran passenger-install-nginx-module to initiate the passenger+nginx config and ran into this:

Nginx doesn't support loadable modules such as some other web servers do,
so in order to install Nginx with Passenger support, it must be recompiled.

Do you want this installer to download, compile and install Nginx for you?

 1. Yes: download, compile and install Nginx for me. (recommended)
    The easiest way to get started. A stock Nginx 1.0.15 with Passenger
    support, but with no other additional third party modules, will be
    installed for you to a directory of your choice.

 2. No: I want to customize my Nginx installation. (for advanced users)
    Choose this if you want to compile Nginx with more third party modules
    besides Passenger, or if you need to pass additional options to Nginx's
    'configure' script. This installer will  1) ask you for the location of
    the Nginx source code,  2) run the 'configure' script according to your
    instructions, and  3) run 'make install'.

Whichever you choose, if you already have an existing Nginx configuration file,
then it will be preserved.

Enter your choice (1 or 2) or press Ctrl-C to abort: 

‘Doesn’t support loadable modules’… ‘stock Nginx 1.0.15′… ‘compile’. Nope.

So… I can allow passenger to build itself into an old version of nginx (1.0.15) or I can download the Nginx source and recompile it with Passenger support. This is where Nginx lost me.

If I want to run a ruby app through Nginx using Passenger (or any other module that integrates Ruby support) I’ll have to recompile Nginx every time there’s an update to either system. Judging by the release frequency of Nginx, that requires recompiling at least once per month. Or, of course, I could just let my system languish and not upgrade regularly. That’ll only leave me open to vulnerabilities in either system, which is not cool.

So I’ll return to Apache+Passenger which supports dynamic module loading and independently updating components. A secure system is a happy system.

When I would try to login, it would return an error saying I specified invalid credentials. Looking at my maillog I saw the connection failing with the username “user@hostname.tld”. My mailserver is configured to use local accounts which means that authentication is done without having to specify the hostname in the address. Checking my other mail clients confirmed that I only have to specify my username to authenticate. AtMail uses the combination of username and hostname when it attempts to authenticate.

I dug around the install for a bit and found a little hidden setting in [AtMailHome]/libs/Atmail/Config.php. To configure AtMail to only send the username to the mailserver for authentication, change this line:

'mailserver_auth' => '1',

to this

'mailserver_auth' => '0',

.

A lovely undocumented setting.

Here are a list of the tricks I’ve learned so far:

1. Firewall rule numbering: The firewall documentation shows that in order to enforce rule ordering, you have to preface your declaration names with a number (e.g. 001, 002, 003, …, 999, etc). Since my firewall rules selectively allow a few ports to be open and deny all other requests, it is easier to number the declarations based off the port they declare.

   firewall { "00000 accept icmp":
     proto => "icmp",
     action => "accept"
   }
   
   firewall { "00001 accept established, related":
     state  => ['ESTABLISHED', 'RELATED'],
     proto  => 'all',
     action => 'accept',
   }
   
   firewall { "00002 accept localhost":
     source => '127.0.0.1',
     proto  => 'all',
     action => 'accept',
   }
   
   firewall { "00080 http on port 80":
     proto => "tcp",
     dport => "80",
     action => "accept"
   }
   
   firewall { "65536 drop incoming packets":
     action => 'drop'
   }

   This way the rules are loaded in port order. The first rules that setup iptables are loaded in the lowest numbers which are reserved ports and won’t likely be used by me anyway. The last rules are numbered after 65535 — the last port.

2. Auto-save configuration: I’ve also found it useful to configured every declaration to notify puppet that it should save the iptables configuration. This way, when one rule gets changed, it automatically saves the new config.

   firewall { "00080 http on port 80":
     proto => "tcp",
     dport => "80",
     action => "accept",
     notify => Exec["iptables-save"]
   }
   
   exec { "iptables-save":
     command => $operatingsystem ? {
       "debian" => "/sbin/iptables-save > /etc/iptables/rules.v4",
       /(RedHat|CentOS)/ => "/sbin/iptables-save > /etc/sysconfig/iptables",
     },
     refreshonly => true,
     notify => Service["iptables"],
   }

3. Overriding declared rules: Probably the most useful thing I figured out was how to remove firewall rules from the saved ruleset. For example, by default my httpd rule allows requests from any IP on the Internet. There is one server that should only allow requests from specific IP addresses. To configure this, I have to remove the default rule and add in new rules for port 80 in iptables. This is done using inheritence.

   class special-http-rules inherits httpd {
   
     Firewall["00080 http on port 80"] { ensure => absent }
   
     firewall { "00080 http on port 80 from special server (123.45.67.89)":
       proto => "tcp",
       dport => "80",
       source => '123.45.67.89',
       action => "accept",
       notify => Exec["iptables-save"]
     }
   }

   Using the inherits keyword on the parent class, I can call the “00080 http on port 80″ firewall declaration and ensure that it is absent from the rules applied to the node. Any server that I assign the “special-http-rules” class to will only respond to requests from the special server on port 80.

Google just announced their I/O 2012 event which takes place June 27th – 29th. To demonstrate the cool stuff they’ll be presenting they released a small Chrome app which allows you to build a Rube Goldberg machine in your browser. I played with it for a while — it’s pretty cool. Check out the Input/Output machine I created.

1. Using your vSphere client, increase the size of the disk attached to the VM.
2. Boot into the CentOS 6 installation disk and select ‘Rescue Mode’ from the list of options.
3. When prompted to search for LVM partitions on the disk, select ‘Skip’.
4. Type the following commands at the prompt. The following assume you have a standard CentOS LVM configuration.

   fdisk /dev/sda
   p  # Print partition table
   n  # New partition
   p  # Primary partition
   3  # ID = 3
   # When prompted, add 1 to the end block value for the sda2 partition and use it as the start of the sda3 partition.
   # Use the default for the size which should be the rest of the free space on the disk.
   t  # Change partition type
   3  # Change partition 3
   8e # Type = Linux LVM
   p  # Print partition table
   w  # Write partition table
   
   # Create a new LVM physical volume from the new partition
   lvm pvcreate /dev/sda3
   lvm pvdisplay
   
   # Mount the volume group
   lvm vgscan
   lvm vgchange -ay
   # You will see the name of the volume group that is activated.  Usually something like 'vg_hostname'.
   
   # Extend the volume group with the new physical volume.  Be sure to substitute the name of your volume group in the command below.
   lvm vgextend /dev/vg_hostname /dev/sda3
   
   # Extend the logical volume to include 100% of the free space on the volume group.
   lvm lvextend /dev/vg_hostname/lv_root /dev/sda3
   
   # Mount the volume group
   lvm vgscan
   lvm vgchange -ay
   
   # Run a filesystem check on the newly expanded disk
   e2fsck -f /dev/vg_hostname/lv_root
   
   # Resize the filesystem to use the entire disk
   resize2fs /dev/vg_hostname/lv_root

5. After rebooting, you can confirm the final size of your disk using:

   df -h

As always, have a backup of your data.

Note: I spent a long time trying to figure this out. The key is that the ‘banner’ property is the String that the monitor daemon looks for to confirm that it’s communicating with an SSH server. I erroneously put ‘SSH-1920′ which would never match the SSH server’s response banner.

<!-- capsd-configuration.xml -->
<protocol-plugin protocol="SSH-1920" class-name="org.opennms.netmgt.capsd.plugins.SshPlugin" scan="on">
  <property key="banner" value="SSH" />
  <property key="port" value="1920" />
  <property key="timeout" value="3000" />
  <property key="retry" value="1" />
</protocol-plugin>

<!-- poller-configuration.xml -->
<service name="SSH-1920" interval="300000" user-defined="false" status="on">
  <parameter key="retry" value="1"/>
  <parameter key="banner" value="SSH"/>
  <parameter key="port" value="1920"/>
  <parameter key="timeout" value="3000"/>
  <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"/>
  <parameter key="rrd-base-name" value="ssh-1920"/>
  <parameter key="ds-name" value="ssh-1920"/>
</service>

<!-- ... and farther down below ...  -->

<monitor service="SSH-1920" class-name="org.opennms.netmgt.poller.monitors.SshMonitor"/>

Message: dualvar is only available with the XS version of Scalar::Util at /usr/lib/perl5/vendor_perl/5.8.8/IO/Socket/SSL.pm line 19

This error was caused by an upgrade to the Compress::Zlib package. The newer version of the package isn’t compiled with XS support. I found some information about this error from this OTRS mailing list post from 2010 regarding RHEL. Looks like it takes about a year for RHEL packages to make it downstream to CentOS.

To fix the error, I ran the following from the command line:

perl -MCPAN -e "CPAN::Shell->force(qw(install Scalar::Util));"

After letting the cronjobs rerun, the error was gone and tickets successfully processed again.

#!/bin/bash
mailto=email@awesome.tld
subject="The awesome subject line of your email message"
(
  echo "Subject: $subject"
  echo "MIME-Version: 1.0"
  echo "Content-Type: text/html"
  echo "Content-Disposition: inline"
  echo "<html><body><pre>"
  mysql db -uawesome -pawesomer --table < some-sql-to-run.sql
  echo "</pre></body></html>"
) | /usr/sbin/sendmail $mailto

The output looks like this:

To: email@email.tld
From: system@awesome.tld
Subject: The awesome subject line of your email message
+----------+----------+----------+
| Column 1 | Column 2 | Column 3 |
+----------+----------+----------+
| Apples   |        2 |    14.00 |
| Oranges  |        2 |   0.5000 |
+----------+----------+----------+

I just got notifications that all my domains were successfully transferred to my new registrar. I’m now using DynaDot which is a vocal opponent of SOPA. The reason I transferred was partially to promote anti-SOPA messages and also for privacy reasons. Hopefully I’ll be much happier with my new service.

Check out the video below for information about PIPA/SOPA and why you should join the fight to oppose these proposed laws.

Bringing up interface eth0: Device eth0 does not seem to be preset, delaying initialization.      [FAILED]

The reason this error occurs is because networking adapters in cloned VMs are assigned unique MAC addresses, so they don’t conflict with the parent VM. During OS installation, the installer detects the network adapter and udev configures the mapping between the device eth0 and the MAC address. When the MAC address changes udev thinks the device is missing.

To fix this, we need to update udev’s mapping rules to point the eth0 definition to the device with the correct MAC address. Open the file /etc/udev/rules.d/70-persistent-net.rules. You should see something similar to what is below:

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100f (e1000) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:00:16", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100f (e1000) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:00:18", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

As you can see there are two PCI ethernet adapters present. The original one from the parent VM (MAC: 9c:00:16) and the new one from the current VM (MAC: 9c:00:18). To fix the issue you need to change the eth0 device definition to have the correct MAC address (9c:00:18) and remove the eth1 device. Your resulting file will look like so:

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100f (e1000) (custom name provided by external tool)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:9c:00:18", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

You’ll also want to update the /etc/sysconfig/network-scripts/ifcfg-eth0 file to reflect the correct MAC address. Then, after a quick system restart your eth0 adapter will be back up.