Purging Exported Resources for Inactive Nodes in Puppet

Tue Apr 30 2013 06:31:06 GMT-0400 (EDT)

Puppet's Exported Resources are a wonderful feature of the tool. In my environment, I'm using them to collect Munin configuration files for the Munin master server.

Each munin client gets the class munin-node with the following partial configuration:

class munin_node {
  @@file { "/etc/munin/munin.conf.d/${::fqdn}.node":
    content => "[Example.com;${::fqdn}]
  address ${::ipaddress}
  use_node_name yes",
    tag     => 'munin-node',

Then, the munin master node collects all the munin configuration files.

node 'munin-master' {
  File <<| tag == 'munin-node' |>>

This is a really useful feature of puppet and it makes adding new nodes to Munin incredibly easy. It also makes it difficult to remove nodes from Munin that may have been shutdown permanently.

When I retire a system, I don't want it to show up in Munin anymore, but because Puppet will collect resources for all the nodes its ever known about even if they've stopped reporting, removing the file by hand will cause it to be recreated by puppet. There is an easy fix for this though -- deactivating and cleaning the node on the puppet master.

puppet node deactivate someoldnode.example.com
puppet node clean someoldnode.example.com
rm /etc/munin/munin.conf.d/someoldnode.example.com.node

The first two commands will effectively retire an old node from Puppet. The first will deactivate it (and stop the master from collecting resources for it) and the second will clean its certificate. The last one is obviously to remove the file created by the collected exported resource.

disk_stats Puppet Module Released!

Mon Apr 08 2013 12:55:56 GMT-0400 (EDT)

I've just released my first Puppet module to the forge. It's called disk_stats and you can download it from the Puppet Forge project page or check out the source code in the repo on GitHub.

I've been using Puppet at work for about a year and have developed some pretty advanced modules for use at work. This module I developed for my own personal use with my website. I wanted to be able to see disk usage in the Puppet Dashboard, but the default facts don't expose disk statistics, only swap info.

Feel free to fork the repo on GitHub and contribute back to the project!

Live Resize of Logical Volume in RHEL/CentOS

Mon Apr 01 2013 10:19:45 GMT-0400 (EDT)

The default installation of RHEL 6 and CentOS 6 will create separate LVM volumes for /home. This means much of your disk is allocated to /home when it could be used for something else. Keeping /home on a separate partition is a good idea for multi-user systems, but for machines that act as a service box, it may be wasted disk space. For example, here's a system that has 50G partitioned for /home when it's really not needed.

$ df -h
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/vg_repo-lv_root 148G  113G   29G  80% /
tmpfs                       499M     0  499M   0% /dev/shm
/dev/sda1                   485M  156M  304M  34% /boot
/dev/mapper/vg_repo-lv_home 47G  181M   45G   1%  /home

To remove the logical volume that stores /home and add the free space to the root partition, follow these steps:

Note: Make sure you're logged into the system at the console as a user whose homedir isn't in /home. Logging in as root usually works.

cd /
cp -Ra /home /home.bak  # Make a backup of home
umount /home
lvm lvremove /dev/vg_<hostname>/lv_home  # Remove the logical volume for home
lvm lvresize -l+100%FREE /dev/vg_<hostname>/lv_root  # Resize the root logical volume so it uses 100% of the now free space
resize2fs /dev/vg_<hostname>/lv_root  # Resize the filesystem to use the whole logical volume
mv /home.bak /home  # Restore the backup.

More Notes:

  • This can be done on a live system as long as /home is not in use when you try to unmount it.
  • You can also follow these steps to resize another logical volume if it has another name and isn't in use.
  • This may work with some adjustments on a RHEL/CentOS 5 system, I haven't tried it though.

On the Future of Software...

Fri Mar 22 2013 05:40:17 GMT-0400 (EDT)

Made me chuckle.

I'm not inclined to make grand pronouncements about the future of software, but if anything kills off commercial software, let me tell you, it won't be open source software. They needn't bother. Commercial software will gleefully strangle itself to death on its own licensing terms.

Jeff Atwood, http://www.codinghorror.com/blog/2013/03/why-ruby.html Coding Horror

Add a Wildcard SSL Certificate to NetApp OnCommand System Manager

Mon Feb 18 2013 11:11:00 GMT-0500 (EST)

When running a NetApp, it is preferable to use valid SSL certificates rather than the self-signed certificates generated by the device. To use wildcard certificates on a NetApp you must do the following:

  1. Open the console (remote or local) on your NetApp device.
  2. Mount vol0 from your NetApp so you can edit the system files.
  3. secureadmin disable ssl
  4. Save the contents of your wildcard cert and your intermediate certs into /etc/keymgr/cert/secureadmin.pem
  5. Save the contents of your private key file into /etc/keymgr/key/secureadmin.pem
  6. secureadmin addcert ssl /etc/keymgr/cert/secureadmin.pem
  7. secureadmin enable ssl

You can test that your SSL certificate is working properly by using the openssl binary:

openssl s_client -connect yournetapp.example.tld:443

Don't forget to perform this configuration on all controllers in your NetApp if your system has more than one.

The True Way to Secure Your SSHd

Sun Feb 17 2013 17:15:02 GMT-0500 (EST)

I found an article written by Peter Hansteen on why There's no Protection in High Ports. In it he explains that using a non-standard port for known services (specifically SSHd) doesn't actually add significant additional security to your system.

I've known this for years and I've been opposed to it since before that. Configuring services to run on non-standard ports in the unreserved range (1024+) is just security by obscurity. Since there is only a finite number of ports on any networked system (65536), all of those ports can easily be scanned by an attacker.

nmap -p- target.host.tld

To truly run a secure SSHd service, you must use key-based authentication, as Peter said, and restrict the networks that your service will accept connections from which is called whitelisting. There are two main ways you can configure whitelisting:

TCP Wrappers

This is the easiest way to restrict access any only requires knowing the names of the networks you will be connecting from. The two files, /etc/hosts.allow and /etc/hosts.deny act as white and black lists for services that use TCP Wrappers such as SSHd.

To use them, add lines in your hosts.allow for each network you'll be connecting from. Home, work, mobile. In hosts.deny add 'ALL: ALL' which will stop everything else.

#cat /etc/hosts.allow
ALL:             # Allow connections from the local system
ALL: .mywork.com    # Allow connections from my work
ALL: .alexcline.net        # Allow connections from other hosts in my own network
ALL: .myISP.com    # Allow connections from my home ISP
ALL: .mymobileprovider.com # Allow connections from my mobile provider when I'm tethering on the go

#cat /etc/hosts.deny


IPTables is probably the better solution to whitelisting since it's something every secure system will have configured, but it requires a bit more knowledge of the network topography you'll be connecting from.

Getting IPTables configured correctly would likely be an entire post in itself, so I won't go into detail about how to set it up. Below is an example of what entries you should have in your iptables config file (/etc/sysconfig/iptables in RedHat, CentOS and Fedora).

You'll want to enter the IP addresses of the networks you'll be connecting from, not the domains. You'll also need to specify a subnet range for the addresses. Here's a good subnet calculator I use frequently.

-A INPUT -p tcp -m tcp -s --dport 22 -j ACCEPT  # Accept connections from localhost
-A INPUT -p tcp -m tcp -s --dport 22 -j ACCEPT  # Accept connections from network A
-A INPUT -p tcp -m tcp -s --dport 22 -j ACCEPT  # Accept connections from network B
... snip other rules ...
-A INPUT -j DROP  # Drop everything else.

The combination of using SSH key-based authentication and whitelisting of permitted networks will lockdown your SSHd server to all attackers. Now, just keep your networks secure and your keys private and you're 100% secure.

Why You Should be Your Own SysAdmin

Sat Feb 16 2013 06:53:05 GMT-0500 (EST)

For a very long time people have been making money on the Internet by taking other people's websites and jamming a bunch of them into the same server and branding it. First it was GeoCities with their bundle of servers running your mom's baking website. Then it was every other person with a cPanel license standing up servers for their friends. Now it's companies building Platforms as a Service (PaaS). Each time the goal is the same -- get as many people to sign up and squeeze them into as few servers as possible.

The problem with these services is that both parties in the hosting arrangement (the provider and the customer) have different goals and incentives. The hosting provider wants to reduce their expenses by cramming as many customers into the smallest resource pool possible while the customer wants to have as many resources available while reducing their costs. These two goals are mutually exclusive.

A good example of this has popped up recently regarding Heroku's platform. James Somers over at Rap Genius wrote a wonderful piece about the dirty secrets he discovered about Heroku's platform.

In mid-2010, Heroku redesigned its routing mesh so that new requests would be routed, not to the first available dyno, but randomly, regardless of whether a request was in progress at the destination. James Somers, 2012

Heroku sells dynos which act as application servers -- a single threaded application processing one request at a time. When you have two dynos running, you can run two instances of your application concurrently. Adding more dynos adds additional concurrency as long as new requests are sent to idle dynos.

As James discovered, at scale, this model became too expensive for Heroku to maintain. Keeping the routing queue intelligent meant more cost for Heroku to maintain the infrastructure which isn't billable to one customer. By making the routers dumb (random), they were able to foist the cost onto the customers by forcing them to purchase additional dynos to improve performance. This is where the mutually exclusive business goals came into play.

Heroku didn't have an incentive to disclose their routing changes to their customers -- the changes don't help customers in any way, it only costs them more money. Customers who want to reduce their costs were looking at inaccurate reporting trying to make sense of numbers that really didn't have any indication of the true problem.

Now that Heroku has been outed, they're pleading dumb -- in their follow-up blog post they do a good job of explaining the problem but fall short of actually promising to solve anything for existing customers.

Next Steps

To reiterate, here is what we are doing now:

  • Improving our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks
  • Removing incorrect and confusing metrics reported by Heroku or partner services like New Relic
  • Adding metrics that let customers determine queuing impact on application response times
  • Providing additional tools that developers can use to augment our latency and queuing metrics
  • Working to better support concurrent-request Rails apps on Cedar Jesper, Heroku Inc., 2012

Notice how they aren't promising to go back to the old ways of routing intelligently -- that would require them to spend money on infrastructure. Updating documentation is cheap and doesn't solve anything. Fixing the reporting is only going to show Heroku users that they need to spend more money (possibly exponentially) on dynos to get additional performance increases. The only measurable improvement to the Heroku service is to parts which directly affect the cost of a dyno -- more concurrency in a single application requires larger, more expensive dynos.

This whole situation just highlights the fact that hosting providers are not looking to save their customers money. They're in the business of making money and anything that isn't directly billable to a customer isn't worth investing in.

If you really want to get your moneys worth from your hosting costs learn to be your own sysadmin. Don't let someone else run part of your application, own everything. Take back your webstack. Make sure you know exactly how each part of your environment works so you can identify problems before they arise. If you're relying on someone else to run it for you, you're paying them to give you the cheapest service they can.

A hosting provider's interests lie in pushing as many customers into the same resource pool as possible. This means taking control away from their customers and insisting that they know best. Take back control of your environment by refusing to accept anything less than full disclosure. Spend the time to learn best practices or ask someone knowledgeable to teach you.

Spending the money to become your own SysAdmin will give you peace of mind knowing you have control over your own systems.

Costa Rica 2013 Video

Thu Feb 14 2013 05:39:30 GMT-0500 (EST)

All the awesome activities I participated in while on vacation in Costa Rica in Feb 2013! Waterfall Rappelling, ATVs, Jungle Zipline and Scuba Diving.

All videos shot using an GoPro HD Hero2 at 720p and edited using Adobe Premiere Pro.


2012 In Review - Onward to 2013!

Thu Jan 10 2013 18:06:57 GMT-0500 (EST)

2012 blogging

Greetings readers, Wordpress.com just sent me an awesome compilation of my 2012 stats! This is a really neat feature that I'm glad they developed. I look at my JetPack stats very frequently and it's hard to get visibility into the bigger picture of how the blog is doing. Since I mostly post useful tidbits of tech, and they seem to be the most popular, I'll keep posting stuff like that. I know I frequently look back at how to expand LVM in CentOS 6 running on vCenter and how to send HTML email from the linux command line.

Here's to many more blog posts in 2013! Head over to jetpack.me to see the 2012 report!