I’ve heard a lot of good things about Nginx. Shawn Wall mentioned how much he likes using it and how easy it is to configure. My experience with it has only been tangential at best. I’ve heard good things about it (lightweight, performant and scalable) on Hacker News and /r/programming, so when it came time to deploy a ruby application I’ve been developing, I thought I would give it a whirl.

There are many competing mantras about how to best deploy a ruby web application into a ‘production’ state. Some people just say, ‘start up your framework’s WEBrick server on 80 and go for it!’ Others, like myself want more stability and security — for that we turn to Passenger. Passenger provides support for Rackable ruby apps on Apache and Nginx.

I’ve always liked the Apache+Passenger configuration when installing on CentOS — I can schedule regular yum updates to upgrade Apache and gem updates to upgrade Passenger. This way, I know that I’ve got the latest and most secure versions running on my systems. With monitoring and testing, I know if something breaks and can fix it promptly.

After setting up my fresh CentOS 6 machine, I found the Nginx yum repos which will happily keep Nginx updated. Great. I then went about doing the usual rvm+ruby+gem+passenger dance. I ran passenger-install-nginx-module to initiate the passenger+nginx config and ran into this:

Nginx doesn't support loadable modules such as some other web servers do,
so in order to install Nginx with Passenger support, it must be recompiled.

Do you want this installer to download, compile and install Nginx for you?

 1. Yes: download, compile and install Nginx for me. (recommended)
    The easiest way to get started. A stock Nginx 1.0.15 with Passenger
    support, but with no other additional third party modules, will be
    installed for you to a directory of your choice.

 2. No: I want to customize my Nginx installation. (for advanced users)
    Choose this if you want to compile Nginx with more third party modules
    besides Passenger, or if you need to pass additional options to Nginx's
    'configure' script. This installer will  1) ask you for the location of
    the Nginx source code,  2) run the 'configure' script according to your
    instructions, and  3) run 'make install'.

Whichever you choose, if you already have an existing Nginx configuration file,
then it will be preserved.

Enter your choice (1 or 2) or press Ctrl-C to abort: 

‘Doesn’t support loadable modules’… ‘stock Nginx 1.0.15′… ‘compile’. Nope.

So… I can allow passenger to build itself into an old version of nginx (1.0.15) or I can download the Nginx source and recompile it with Passenger support. This is where Nginx lost me.

If I want to run a ruby app through Nginx using Passenger (or any other module that integrates Ruby support) I’ll have to recompile Nginx every time there’s an update to either system. Judging by the release frequency of Nginx, that requires recompiling at least once per month. Or, of course, I could just let my system languish and not upgrade regularly. That’ll only leave me open to vulnerabilities in either system, which is not cool.

So I’ll return to Apache+Passenger which supports dynamic module loading and independently updating components. A secure system is a happy system.

I’ve been running AtMail Open for my personal email for a while now. I recently upgraded to the latest version 1.05 for the security updates and ran into a bit of a problem after completing the installation. I likely had to fix this when I first installed AtMail way back in the day.

When I would try to login, it would return an error saying I specified invalid credentials. Looking at my maillog I saw the connection failing with the username “user@hostname.tld”. My mailserver is configured to use local accounts which means that authentication is done without having to specify the hostname in the address. Checking my other mail clients confirmed that I only have to specify my username to authenticate. AtMail uses the combination of username and hostname when it attempts to authenticate.

I dug around the install for a bit and found a little hidden setting in [AtMailHome]/libs/Atmail/Config.php. To configure AtMail to only send the username to the mailserver for authentication, change this line:

'mailserver_auth' => '1',

to this

'mailserver_auth' => '0',

.

A lovely undocumented setting.

When using Puppet for systems configuration management, the new Puppet-Firewall module is really useful for managing iptables rules. Sometimes though, there are some tricks that are required to get the firewall to behave properly.

Here are a list of the tricks I’ve learned so far:

1. Firewall rule numbering: The firewall documentation shows that in order to enforce rule ordering, you have to preface your declaration names with a number (e.g. 001, 002, 003, …, 999, etc). Since my firewall rules selectively allow a few ports to be open and deny all other requests, it is easier to number the declarations based off the port they declare.

   firewall { "00000 accept icmp":
     proto => "icmp",
     action => "accept"
   }
   
   firewall { "00001 accept established, related":
     state  => ['ESTABLISHED', 'RELATED'],
     proto  => 'all',
     action => 'accept',
   }
   
   firewall { "00002 accept localhost":
     source => '127.0.0.1',
     proto  => 'all',
     action => 'accept',
   }
   
   firewall { "00080 http on port 80":
     proto => "tcp",
     dport => "80",
     action => "accept"
   }
   
   firewall { "65536 drop incoming packets":
     action => 'drop'
   }

   This way the rules are loaded in port order. The first rules that setup iptables are loaded in the lowest numbers which are reserved ports and won’t likely be used by me anyway. The last rules are numbered after 65535 — the last port.

2. Auto-save configuration: I’ve also found it useful to configured every declaration to notify puppet that it should save the iptables configuration. This way, when one rule gets changed, it automatically saves the new config.

   firewall { "00080 http on port 80":
     proto => "tcp",
     dport => "80",
     action => "accept",
     notify => Exec["iptables-save"]
   }
   
   exec { "iptables-save":
     command => $operatingsystem ? {
       "debian" => "/sbin/iptables-save > /etc/iptables/rules.v4",
       /(RedHat|CentOS)/ => "/sbin/iptables-save > /etc/sysconfig/iptables",
     },
     refreshonly => true,
     notify => Service["iptables"],
   }

3. Overriding declared rules: Probably the most useful thing I figured out was how to remove firewall rules from the saved ruleset. For example, by default my httpd rule allows requests from any IP on the Internet. There is one server that should only allow requests from specific IP addresses. To configure this, I have to remove the default rule and add in new rules for port 80 in iptables. This is done using inheritence.

   class special-http-rules inherits httpd {
   
     Firewall["00080 http on port 80"] { ensure => absent }
   
     firewall { "00080 http on port 80 from special server (123.45.67.89)":
       proto => "tcp",
       dport => "80",
       source => '123.45.67.89',
       action => "accept",
       notify => Exec["iptables-save"]
     }
   }

   Using the inherits keyword on the parent class, I can call the “00080 http on port 80″ firewall declaration and ensure that it is absent from the rules applied to the node. Any server that I assign the “special-http-rules” class to will only respond to requests from the special server on port 80.

Google just announced their I/O 2012 event which takes place June 27th – 29th. To demonstrate the cool stuff they’ll be presenting they released a small Chrome app which allows you to build a Rube Goldberg machine in your browser. I played with it for a while — it’s pretty cool. Check out the Input/Output machine I created.

This is an update to the post I created back in June of 2011 on how to expand an LVM in CentOS 5.6 running on vCenter. CentOS 6 has made some changes to the way the LVMs are setup so I thought I would update this document accordingly.

1. Using your vSphere client, increase the size of the disk attached to the VM.
2. Boot into the CentOS 6 installation disk and select ‘Rescue Mode’ from the list of options.
3. When prompted to search for LVM partitions on the disk, select ‘Skip’.
4. Type the following commands at the prompt. The following assume you have a standard CentOS LVM configuration.

   fdisk /dev/sda
   p  # Print partition table
   n  # New partition
   p  # Primary partition
   3  # ID = 3
   # When prompted, add 1 to the end block value for the sda2 partition and use it as the start of the sda3 partition.
   # Use the default for the size which should be the rest of the free space on the disk.
   t  # Change partition type
   3  # Change partition 3
   8e # Type = Linux LVM
   p  # Print partition table
   w  # Write partition table
   
   # Create a new LVM physical volume from the new partition
   lvm pvcreate /dev/sda3
   lvm pvdisplay
   
   # Mount the volume group
   lvm vgscan
   lvm vgchange -ay
   # You will see the name of the volume group that is activated.  Usually something like 'vg_hostname'.
   
   # Extend the volume group with the new physical volume.  Be sure to substitute the name of your volume group in the command below.
   lvm vgextend /dev/vg_hostname /dev/sda3
   
   # Extend the logical volume to include 100% of the free space on the volume group.
   lvm lvextend /dev/vg_hostname/lv_root /dev/sda3
   
   # Mount the volume group
   lvm vgscan
   lvm vgchange -ay
   
   # Run a filesystem check on the newly expanded disk
   e2fsck -f /dev/vg_hostname/lv_root
   
   # Resize the filesystem to use the entire disk
   resize2fs /dev/vg_hostname/lv_root

5. After rebooting, you can confirm the final size of your disk using:

   df -h

As always, have a backup of your data.

To enhance security, I run SSH on a non-standard port. When using OpenNMS to monitor systems, it only looks for SSH servers on the standard port (22) so I had to update the configuration to monitor systems running SSH on a different port. Here’s the configuration I used:

Note: I spent a long time trying to figure this out. The key is that the ‘banner’ property is the String that the monitor daemon looks for to confirm that it’s communicating with an SSH server. I erroneously put ‘SSH-1920′ which would never match the SSH server’s response banner.

<!-- capsd-configuration.xml -->
<protocol-plugin protocol="SSH-1920" class-name="org.opennms.netmgt.capsd.plugins.SshPlugin" scan="on">
  <property key="banner" value="SSH" />
  <property key="port" value="1920" />
  <property key="timeout" value="3000" />
  <property key="retry" value="1" />
</protocol-plugin>

<!-- poller-configuration.xml -->
<service name="SSH-1920" interval="300000" user-defined="false" status="on">
  <parameter key="retry" value="1"/>
  <parameter key="banner" value="SSH"/>
  <parameter key="port" value="1920"/>
  <parameter key="timeout" value="3000"/>
  <parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"/>
  <parameter key="rrd-base-name" value="ssh-1920"/>
  <parameter key="ds-name" value="ssh-1920"/>
</service>

<!-- ... and farther down below ...  -->

<monitor service="SSH-1920" class-name="org.opennms.netmgt.poller.monitors.SshMonitor"/>