dens.one

September 18, 2011 at 9:03pm

Crude Ruby Script to Jumpstart a Chef Server on ec2

Just wanted to share a script I hacked together. Mainly from the the AWS gem docs. This script builds a new ec2 server and jumpstarts a chef server for you. You can pull it from my

Github: https://github.com/densone/chef-server-jumpstart 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 
require ‘rubygems’
require ‘aws’
require ‘net/ssh’
require ‘net/http’
require ‘logger’

def message_lines
  2.times do |x|
    puts ”****************************************”
  end
end

logger = Logger.new(STDOUT)


#load aws.yml — This will allow you to login to aws
aws_conf = YAML.load(File.read(‘aws.yml’))

AWS.config(aws_conf)

instance = key_pair = group = nil

#create a new instance of the aws ec2 module
ec2 = AWS::EC2.new

#lookup the server ami. Is there a better way to do this?  
image = AWS.memoize do
   amazon_linux = ec2.images().
     filter(“root-device-type”, “instance-store”).
     filter(“name”, “*ubuntu-images/ubuntu-natty-11.04-i386-server-20110426*”)
   amazon_linux.to_a.sort_by(&:name).last
 end

logger.debug(“Using AMI: #{image.id} —- #{image.name}”)

#dynamically create a keypair for ec2 and save the private key to a file
key_pair = ec2.key_pairs.create(“chef-kp-#{Time.now.to_i}”)
logger.debug(“Generated keypair #{key_pair.name}, fingerprint: #{key_pair.fingerprint}”)
logger.debug(“Writing #{key_pair.name} to identity.pem”)
identity = File.new(“identity.pem”, “w”)
identity.write(key_pair.private_key)
identity.close
File.chmod(0600,“identity.pem”)

#create a security group for the chef server. Allow ssh and port 4000. 
group = ec2.security_groups.create(“js-gp-#{Time.now.to_i}”)
group.authorize_ingress(:tcp, 22, “0.0.0.0/0”)  

logger.debug(“Using security group: #{group.name}”)

#start the Amazon ec2 instance
instance = image.run_instance(:key_pair => key_pair, :security_groups => group)
logger.info(“Launched instance #{instance.id}, status: #{instance.status}”)
sleep 2 until instance.status != :pending
logger.info(“Launched instance #{instance.id}, status: #{instance.status}”)
exit 1 unless instance.status == :running

#There is an race condition with port 22 being open, but no active SSHD. This sucks, but it’s necessary. 
logger.debug(“There seems to be a firewall rule lag here. Sleeping for 90 seconds…..”)
sleep 90

#login to the machine with ssh and get chefs pre-installation rolling. 
logger.debug(“Logging into IP: #{instance.ip_address}”)

#Because of two interactive dialogs during the chef installation, I generate this script on the newly created server
# log you in, then run the script with an active screen. There may be a better way, but I know if you override these 
# dialogues, many important pieces of the chef configuration will not get generated. 

begin
  Net::SSH.start(instance.ip_address, “ubuntu”, :key_data => [key_pair.private_key]) do |ssh|
    puts ssh.exec!(“echo "deb http://apt.opscode.com/ `lsb_release -cs`-0.10 main" | sudo tee /etc/apt/sources.list.d/opscode.list”)
    puts ssh.exec!(“cat /etc/apt/sources.list.d/opscode.list”)
    puts ssh.exec!(“sudo mkdir -p /etc/apt/trusted.gpg.d”)
    logger.debug(“Configuring Keys for the Chef Apt Repository”)
    puts ssh.exec!(“sudo gpg —keyserver keys.gnupg.net —recv-keys 83EF826A”)
    puts ssh.exec!(“sudo gpg —export packages@opscode.com | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null”)
    logger.debug(“Updating all of the sources on the ubuntu machine”)
    puts ssh.exec!(“sudo apt-get update”)
    logger.debug(“Installing Opscode Keyring”)
    puts ssh.exec!(“sudo DEBIAN_FRONTEND=’noninteractive’ apt-get install opscode-keyring —assume-yes”)
    logger.debug(“Installing XML Libraries for Nokogiri”)
    puts ssh.exec!(“sudo apt-get install libxslt1-dev libxml2-dev —assume-yes”)
    logger.debug(“Installing RubyGems”)
    puts ssh.exec!(“sudo apt-get install rubygems1.8 —assume-yes”)
    logger.debug(“Installing Git”)
    puts ssh.exec!(“sudo apt-get install git —assume-yes”)
    logger.debug(“Installing Merb-Haml”)
    puts ssh.exec!(“sudo gem install merb-haml —no-ri —no-rdoc”)
    puts ssh.exec!(“echo "sudo apt-get install chef chef-server-api chef-expander —assume-yes" > setup_chef”)
    puts ssh.exec!(“echo "sudo gem install knife-ec2 —no-ri —no-rdoc " » setup_chef”)
    puts ssh.exec!(“echo "rm .profile" » setup_chef”)
    puts ssh.exec!(“echo "./setup_chef" > .profile”)
    puts ssh.exec!(“echo "#{key_pair.private_key}" > identity.pem”)
    puts ssh.exec!(“chmod 755 setup_chef create_server”)
    puts ssh.exec!(“chmod 600 identity.pem”)
  end
rescue SystemCallError, Timeout::Error => e
  logger.info(“The ssh port may not be open, trying again”)
  sleep 5
  retry
end 

login = File.new(“login.sh”, “w”)
login.write(“ssh -i identity.pem ubuntu@#{instance.ip_address}”)
login.close
File.chmod(0755,“login.sh”)

message_lines
puts “Automatically logging you into the new chef server to finish installation”
puts “Alternately you can use the command below:”
puts “ssh -i #{Dir.pwd}/identity.pem ubuntu@#{instance.ip_address}”
message_lines
sleep 5















   		 
  
 

        

        

        

        

        

        

        
        

        
        

          

            February 8, 2011 at 3:04am

            
            
            
            
            
          

        
          
rails3 MySQL and taps error on OSX Snow Leopard

          
I recently tried to get the mysql library gem working for rails3 and taps. This would allow me to pull from heroku’s postgres database and push data to my local dev mysql db. 
  
I was getting the most annoying error: 
  
  
 
1 2 3 4 
 		    
Sequel::AdapterNotFound -> LoadError: require ‘mysql’ did not define \ 
Mysql::CLIENT_MULTI_RESULTS! You are probably using the pure ruby mysql.rb \
 driver, which Sequel does not support. You need to install the C based adapter, \ 
and make sure that the mysql.so file is loaded instead of the mysql.rb file.
   		 
  
This issue can also be seen here by a bunch of other taps users: 
  
https://github.com/ricardochimal/taps/issues#issue/47/comment/748899
    
Solution: 
  
  • Uninstall MySql from you mac. Just google it to learn how. 
    •   
  • Download and install the MySQL install of OSX 10.5 (I know you are on 10.6, just do it) 
    http://dev.mysql.com/downloads/mysql/ <— Scroll down for 10.5 
    •   
  • Install your gem with x86_64 arch flags. It’s not going to work if you don’t. 
    sudo env ARCHFLAGS=”-arch x86_64” gem install mysql — —with-mysql-config=/usr/local/mysql/bin/mysql_config     
    •   
  

        

        

        

        

        

        

        
        

        
        

          

            February 7, 2011 at 11:44pm

            
            
            
            
            
          

        
          
Pan Seared Chicken Reduced With Narragansett

          
One of my New Years resolutions this year was to cook a lot more. I am so used to eating out all of the time, that I basically spend a ton of money on shitty food. This past weekend I went to Trader Joe’s and Foodies. Bought a bunch of random food and this is one of my dishes. 
    
Pan Seared Chicken Reduced With Narragansett
    
Ingredients
  
  1. chopped sage
    2.   
  2. chopped garlic
    3.   
  3. sea salt 
    4.   
  4. pepper
    5.   
  5. onion
    6.   
  6. unsalted butter
    7.   
  7. whole wheat pasta
    8.   
  8. chicken breast
    9.   
  9. spinach 
    10.   
  10. feta cheese
    11.   
  11. narragansett beer
    12.   
Instructions 
  
  1. Chop the onions/sage and cook in a pan with the butter.
    2.   
  2. Once the onions are mostly cooked, add the garlic and keep and make sure not to burn it.  
    3.   
  3. In another pan boil, cook, and strain whole wheat pasta. 
    4.   
  4. Grab your chicken breast and try to slice it into more cutlet like pieces. Make sure you use a sharp non-serated knife.  
    5.   
  5. Crank the heat up and sear the chicken in the pan with the spices and onion.
    6.   
  6. Once the the chick is browned on both side, add some narragansett. 
    7.   
  7. Take a fork and scrape all the goodies that burned into the pan. This will enhance the taste of the sauce you are reducing.  
    8.   
  8. Once you have reduced by at least half,  throw the spinach in the pan and take out after about 10 seconds.  
    9.   
  9. Mix the spinach with the pasta.  
    10.   
  10. Add some feta into the the reduction and let it cream up. 
    11.   
  11. Mix the reduction and chicken into the pasta and spinach.  
    12.   

        

        

        

        

        

        

        
        

        
        

          

            10:57pm

            
            
            
            
            
          

        
          
Pan Seared Chicken Reduced With Narragansett

          
One of my New Years resolutions this year was to cook a lot more. I am so used to eating out all of the time, that I basically spend a ton of money on shitty food. This past weekend I went to Trader Joe’s and Foodies. Bought a bunch of random food and this is one of my dishes. 
    
Pan Seared Chicken Reduced With Narragansett
    
Ingredients
  
  1. chopped sage
    2.   
  2. chopped garlic
    3.   
  3. sea salt 
    4.   
  4. pepper
    5.   
  5. unsalted butter
    6.   
  6. whole wheat pasta
    7.   
  7. chicken breast
    8.   
  8. spinach 
    9.   
  9. feta cheese
    10.   
  10. narragansett beer
    11.   

        

        

        

        

        

        

        
        

        
        

          

            January 27, 2011 at 1:42pm

            
            
            
            
            
          

        
          
Using The Grinder 3 to Load Test a Ruby on Rails Application

          
I’ve been using The Grinder3 to run some rails performance tests and ran into an issue where I could not parse some html to submit Ruby on Rails forms. Every form in a Ruby on Rails application has an authenticity_token. This keeps jerks from messing around.
  
The process: 
  
1) Go to the sign_in page.
2) Parse the authenticity token from the response 
3) Post and login to the applicaiton.
  

The Errors
  
1) The Grinder does not support the re module out of the box. Grinder uses Jython, but its slimmed down. I fixed this by downloading the latest Jython installation and dropping it in to $GRINDER_HOME/lib. Make sure to copy jython’s Lib directory also. Yes that was Lib with a capital “L”. 
  
2) You need to compile your regular expression outside of the test thread. At least this was the case for me. See line 7 in the following gist. 
  
3) You need to import the re module into each test thread along with importing re into the agent. See lines 5 and 45 of the test. 
  
Test Away
  
Once you parse the authenticity_token, you will be able to load test your rails apps with the grinder. I am using devise for authentication in this sample. 
  
Also notice I have cookies enabled in the gist. 
  
The Gist
  
  
 
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 
 		    
from net.grinder.script.Grinder import grinder
from net.grinder.script import Test
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
import re

reg = re.compile(‘<input name=”authenticity_token” type=”hidden” value=”([^”]*)” />’)

# Parse global parameters.
control = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()
pluginParameters = grinder.properties.getPropertySubset(“grinder.plugin.parameter.”)

# Parse global parameters.
control = HTTPPluginControl.getConnectionDefaults()
control.followRedirects = pluginParameters.getBoolean(“followRedirects”, 1)
control.useCookies = pluginParameters.getBoolean(“useCookies”, 1)
logHTML = pluginParameters.getBoolean(“logHTML”, 1)

# We declare a default URL for the HTTPRequest.
request = HTTPRequest(url = “http://densone.com”)

def page1():
    import re
    result = request.GET(‘/users/sign_in’)
    grinder.sleep(100)
    #parse the authenticity token from the html.
    token = reg.findall(result.getText())
    print ‘Auth Token:’, token[0] 
    presult = request.POST(‘/users/sign_in’,
      ( NVPair(‘utf8’, ’?’),
        NVPair(‘authenticity_token’, token[0]),
        NVPair(‘student[email]’, ‘yourmom@densone.com’),
        NVPair(‘student[password]’, ‘Betsy’),
        NVPair(‘student[remember_me]’, ‘0’),
        NVPair(‘commit’, ‘Sign In’), ),
      ( NVPair(‘Content-Type’, ‘application/x-www-form-urlencoded’), ))

    request.GET(‘/home’)
    


page1Test = Test(1, “Sign_In”).wrap(page1)

class TestRunner:
    def __call__(self):
        import re
        page1Test()


   		 
  
 
  
Links
  
http://grinder.sourceforge.net/
rubyonrails.org 

        

        

        

        

        

        

        
        

        
        

          

            November 17, 2010 at 11:18pm

            
            
            
            
            
          

        
          
Manage iptables Firewalls on RackSpace Cloud with Chef

          
I ran into an interesting issue with RackSpace Cloud recently. They don’t have a magical firewall layer like ec2. You basically have to manage iptables with a wrapper for each machine on your network. This can be quite the pain in the ass if you have more than a few servers. In my case, I need to block the external interface on about 50 servers, so a Chef cookbook was the right approach.   
  
This cookbook is written specifically for Ubuntu, but can be quickly ported to any Linux Distro.
  
This article is written for a System Engineer or Developer that knows the Chef basics. Please feel free to ask questions below in the comments section for a deeper explanation of this cookbook.
    
Clone My Chef Repo on GitHub:
Every code snippet in this article can be grabbed from my public git repository. 
  
http://github.com/densone/densone-chef
    
Shorewall
I decided to use an iptables wrapper called Shorewall. Shorewall is not daemonized. When running shorwall restart/stop/start, you are just running scripts that update iptables.
  

Simple Installation  
  
Only a single package named “shorewall” needs to be installed. Chef makes this easy with its package resource. 
  
 package "shorewall"  
    
Shorewall Config Files 
Shorewall has a handful of important config files. Proper configuration is essential for iptables to get updated properly. My chef GitHub Repo will work for RackSpace Cloud out of the box.  
  
  • interfaces -  describes the network interfaces to be firewalled.  
    •   
  • policy - defines the high level policy for connections between zones. 
    •   
  • zones - this file allows you to declare your network zones. 
    •   
  • rules - holds all of the exceptions to the firewall policy
    •   
  • shorewall.conf - this is the master shorewall config. For ease of use, this is not edited. 
    •   
  
Let’s Put the Recipe Together 
So far we have talked about shorewall and its various files. Let’s put it all together now in a Chef recipe.
 
  
default.rb
This cookbook only has one recipe, so we can name it default. 
http://gist.github.com/644378
  
The preceding recipe will show you a few things. 
  
  • The package installation of Shorewall
    •   
  • The copy of several config files mentioned earlier 
    •   
  • The generation of an erb template for firewall rules
    •   
  • The enabling of the Shorewall service, and restarting of the service based on file triggers.
    •   
rules.erb
This file iterates through a node attribute defined in the cookbook default.rb 
  
http://gist.github.com/644389
    
Applying The Cookbook To a Role
The cookbook is ready. Now we need to create a role and setup some rules in it. 
  
http://gist.github.com/648330
  
  • Save the role to a file “appserver.rb”
    •   
  • Upload the role: knife role from file $CHEF_REPO/roles/appserver.rb
    •   
  
Apply The Role to an App Server, Sit Back and Let The Machine Work for You
  
To connect this role to an app server run the following command: 
  
knife node run_list add  <app-server-fqdn> “role[appserver]”
    
Light Reading 
  
http://wiki.opscode.com - Information about Chef.
  
http://www.shorewall.net/ - Information about Shorline Firewall. 
    
 

        

        

        

        

        

        

        
        

        
        

          

            May 24, 2009 at 1:57pm

            
            
            
            
            
          

        

        

        
          
Irresponsibility is part of the pleasure of all art; it is the part the schools cannot recognize. -Joyce

          
        

        

        

        

        
        

        

        


        
        

      

      

        

          

            Home  Archive  Mobile  RSS    
          

        

      

      

        

          Themed by langer, powered by Tumblr.