Using Python, Winrm and Powershell to Write Files on Windows from Linux

My primary scripting language is Python and we use it to our manage servers. When it comes to Windows we use WinRM and the Python library pywinrm. One use case is the need to write files from linux to the Windows server, say in the example of writing a config file.

The best way to do this is use Powershell’s StreamWriter. According to this blog post, it’s the fastest of the various methods.

In the sample script below I’ve explained some of the tricks to accomplishing this in the comments.


from winrm import Protocol
import base64
address = "10.10.1.12"
transport = "plaintext"
username = "administrator"
password = "ThePassword"
protocol = "http"
port = 5985

endpoint = "%s://%s:%s/wsman" % (protocol, address, port)

conn = Protocol(endpoint=endpoint, transport=transport,
                username=username, password=password)
shell_id = conn.open_shell()

# the text file we want to send
# this could be populated by reading a file from disk instead
# has some special characters, just to prove they won't be a problem
text_file = """this is a multiline file
that contains special characters such as
"blah"
'#@$*&&($}
that will be written
onto the windows box"""

# first part of the powershell script
# streamwriter is the fastest and most efficient way to write a file
# I have found
# notice the @", this is like a "here document" in linux
# or like triple quotes in python and allows for multiline files
# the file will be placed in the home dir of the user that 
# logs into winrm (administrator in this case)
part_1 = """$stream = [System.IO.StreamWriter] "test.txt"
$s = @"
"""

# second part of the powershell script
# the "@ closes the string
# the replace will change the linux line feed to the
# windows carriage return, line feed
part_2 = """
"@ | %{ $_.Replace("`n","`r`n") }
$stream.WriteLine($s)
$stream.close()"""

# join the beginning of the powershell script with the
# text file and end of the ps script
script = part_1 + text_file + part_2

# base64 encode, utf16 little endian. required for windows
encoded_script = base64.b64encode(script.encode("utf_16_le"))

# send the script to powershell, tell it the script is encoded
command_id = conn.run_command(shell_id, "powershell -encodedcommand %s" %
                            (encoded_script))
stdout, stderr, return_code = conn.get_command_output(shell_id, command_id)
conn.cleanup_command(shell_id, command_id)
print "STDOUT: %s" % (stdout)
print "STDERR: %s" % (stderr)

# print the file
command_id = conn.run_command(shell_id, "type test.txt")
stdout, stderr, return_code = conn.get_command_output(shell_id, command_id)
conn.cleanup_command(shell_id, command_id)
print "STDOUT: %s" % (stdout)
print "STDERR: %s" % (stderr)

# delete the file
command_id = conn.run_command(shell_id, "del test.txt")
stdout, stderr, return_code = conn.get_command_output(shell_id, command_id)
conn.cleanup_command(shell_id, command_id)
print "STDOUT: %s" % (stdout)
print "STDERR: %s" % (stderr)

# always good to clean things up, doesn't hurt

 

Advertisements

Moving a MySQL Database from One EBS Volume to Another on AWS EC2

Recently at Cloud Sidekick we had a booth at Cloud Expo West and it was my job to stand up a demo environment to show off our latest product Maestro. I was using a single server with all of the application components installed. The demo env was installed on an Amazon AWS EC2 m1.medium with the standard 8 gig root device EBS volume. Usually for a demo environment 8 gig is fine, however with all of the monitoring metrics I was collecting on day two of demoing the application we started to run out of disk space. The monitoring polling interval frequency was set very low (several times a minute) so that we could collect more data quickly and make the graphs pretty.

It turned out it was my MySQL database that was growing and consuming the disk so I needed either create a new demo instance with a larger disk and forego all the demo data or move the database off to an external EBS volume. Here’s how to move an existing MySQL database from one EBS volume to another in Amazon EC2.

First a little background: These instructions are for Ubuntu 12.04 Precise with MySQL 5.5. Mileage may vary for other flavors and versions.

Ssh into your EC2 instance and get an idea of your current disk space:

df -h

Now let’s create a new EC2 EBS volume and attach it to our instance. You can create the volume any number of ways (command line, script) but for this example we’ll use the EC2 console.

Login to the AWS Console, select the AWS cloud in which your database instance resides. Navigate to Services, EC2, Instances. Find your MySQL database instance and make note of the availability zone and instance id.

Now navigate to Volumes. Click on the Create Volume button. I leave it up to you to determine the size of the new volume, just remember that disk is relatively cheap. Select the same availability zone as your database server as EBS volumes cannot be attached to servers in different zones.

Note: these images appear small on the post, just click on them to view larger size.

When the volume status shows that it is “available”, select the volume and go ahead and tag it.

Now from the toolbar above the volume list, select More, Attach Volume.

Select the instance with the MySQL database. You should be able to leave the Device as the default unless you have other EBS volumes other than the root device attached.

After refreshing your console a couple times you should see the volume status as attached.

We’re done with the AWS Console, now back to ssh.

Now use fdisk to view all volumes. You will see the new volume (in my case /dev/xvdf) as well as the root volume and another volume which is known as the ephemeral disk. We will use the ephemeral disk later on. For now note the device name that Ubuntu uses (versus /dev/sdf that Amazon reports).

sudo fdisk -l

Format the new volume as an ext4 filesystem.

sudo mkfs -t ext4 /dev/xvdf

The new volume is now attached and formatted. We will mount it later.

Since I was coming close to being out of disk space, I needed some temporary space to move database files around. AWS EC2 comes with free temporary space called ephemeral disk already on instance sizes greater than t1.micro. Some images automatically mount this additional storage when booted, with other images you have to mount the disk yourself. With my Ubuntu 12.04 image, this disk was already mounted under /mnt.

Note about ephemeral disk: this disk space is released when an EC2 instance is terminated or stopped (stays intact on a reboot). Take care what you use it for.

Change your pwd to the ephemeral disk. If your image doesn’t have the ephemeral disk already attached, google “attach ephemeral disk EC2” for plenty of examples.

cd /mnt

Shut down any external processes and connections to the MySQL database. This will be application specific.

Since the MySQL database will be moved, we will want to take a backup of the database. The EC2 root volume is not big enough so we will use the ephemeral disk.

mysqldump -uusername -ppassword databasename | sudo tee database_backup.sql > /dev/null

Compress the backup.

sudo gzip database_backup.sql

Upload the backup to S3 or otherwise copy the file elsewhere. For S3 I used s3put, which is a command line utility that is installed with the Python Boto package and existed already on my Ubuntu image.

s3put -a accesskey -s secretkey -b bucketname database_backup.sql.gz

Stop the MySQL database service.

sudo service mysql stop

Now copy the MySQL data directory from it’s home to the ephemeral disk. Note: you could also tar and gzip the whole directory and store in s3 as well for added safety. Also rsync could be used here as well.

sudo cp -R /var/lib/mysql /mnt/mysql

Verify the files copied as expected.

sudo ls -l /mnt/mysql

Remove the original directory.

sudo rm -rf /var/lib/mysql

Create a new directory in it’s place.

sudo mkdir /var/lib/mysql

Mount the new volume to the new directory.

sudo mount /dev/xvdf /var/lib/mysql

Edit fstab to make sure the new volume is attached at boot. Add another line and use the new device and directory we created.

sudo vi /etc/fstab

Now copy the database files to the mounted volume.

sudo cp -R /mnt/mysql /var/lib/.

Change the permissions back to owner rwx only.

sudo chmod 700 /var/lib/mysql

And change the owner and group back to mysql.

sudo chown -R mysql:mysql /var/lib/mysql

Verify all files are in place, permissions are correct, etc.

sudo ls -l /var/lib/mysql

Start the MySQL service back up.

sudo service mysql start

Test that MySQL is functioning properly by logging into the database. If everything is correct, start your application back up.

That’s it. The database is now on the new, larger EBS volume and the root volume now has space to breath. There are a couple other advantages to putting the database on a separate EBS volume:

  • Snapshots – you can now take EBS volume snapshots of your database on a regular basis without snapshotting the whole OS. This adds security to your database backup strategy (you do have a strategy right?).
  • EBS persistance – If your EC2 server instance is terminated, your database volume will persist. You can always bring up a new server, attach the EBS volume and with some MySQL configuration point to the database.Please feel free to leave comments. Good luck.

Using an xpath query with Python etree to find a value of a node with the same parent path as another node

Recently I needed to search through an xml document using Python for a specific key / value pair and get the text value of a sibling node. For the most part this xml structure was going to be the same every time, but the text value of one of the nodes would change. We’re already using the etree library, here’s how I did it…

Here’s the sample xml structure:

<parameters>
    <parameter>
        <name>abc</name>
        <value>111</value>
    </parameter>
    <parameter>
        <name>efg</name>
        <value>222</value>
    </parameter>
    <parameter>
        <name>hij</name>
        <value>333</value>
    </parameter>
</parameters>

Not an optimal structure but it’s what I’ve got to work with.

Now, let’s say we need to pull out the value node of the parameter with the name “efg”. We will need to search the document for the parent node and pull out the text in the value child node.

Here’s the code …

#!/usr/bin/env python
try:
    import xml.etree.cElementTree as ET
except (AttributeError, ImportError):
    import xml.etree.ElementTree as ET

the_xml = """<parameters>
    <parameter>
        <name>abc</name>
        <value>111</value>
    </parameter>
    <parameter>
        <name>efg</name>
        <value>222</value>
    </parameter>
    <parameter>
        <name>hij</name>
        <value>333</value>
    </parameter>
</parameters>"""

def get_param_value_by_name(xml, name):
    """finds the value of a parameter with a variable name"""

    root = ET.fromstring(xml)
    path = "./parameter/[name='" + name + "']"
    node = root.findall(path)
    if node:
        #print ET.tostring(node[0])
        return_val = node[0].findtext("./value", "")
        #print return_val
    else:
        return_val = None
    del(root)

    return return_val

v = get_param_value_by_name(the_xml, "efg")
print v

A couple things to note…

The try, except block at the top attempts to load the “c” version of etree if it is available on the platform which the script is going to run. This is a good practice no matter what you use etree for, if the c version is available the interpreter will use that and it’s much faster. Otherwise the pure python version of etree will be used.

Next, inside of the function an xpath query string is constructed and searched for using findall. If there is a node tagged “name” with value equal to the name parameter, that node is retrieved. In this example we are only looking for the first instance in the list of nodes returned, you can modify that if it suites your purposes.

findtext is used to get the value of the node tagged “value” and return those results. If either the original search node is not found or there is no value node found, None is returned.

Comparison of Python VMware vSphere client libraries

I recently had the need to manage VMware vSphere from Python code so I went about looking for examples and open source libraries. The vCenter management server has a SOAP web service that exposes most (if not all) of the administrative capabilities that you can perform on vSphere. At first thought, this seemed like a simple endeavor. However once I got into it I found it not as simple as interacting with a RESTful web service.

The first thing I had to do is setup a vSphere test environment. More details on this in another blog post, but basically I created a login at My VMware and downloaded a 60 day trial of the vSphere hypervisor and vCenter.

Next I went searching for Python client libraries for the vSphere SOAP web service.

Here is a summary of what I found:

  • psphere – pure Python library  for the vSphere SDK by Jonathan Kinred
  • pyvisdk – pure Python library  for the vSphere SDK by Eric Plaster
  • pysphere – pure Python library  for the vSphere SDK by Sebastian Tello
  • vijava – vSphere Java api by Steve Jin extended to Python with the Jython interpreter

Here are some details on each:

psphere

source: https://github.com/jkinred/psphere

docs: http://psphere.readthedocs.org/

irc: #psphere on Freenode

mailing list or forum: https://groups.google.com/forum/?fromgroups#!forum/psphere

Psphere is a pure Python library which uses the Suds client Python library to communicate via SOAP. It is actively maintained with activity picking up the summer of 2012. Recently moved from Bitbucket to Github which is nice. Also recently create an irc channel on freenode. Apache 2.0 license.

Advantages

  • Actively maintained
  • Good author response, acknowledges shortcomings and wants to work with the community to resolve them
  • Very easy to work with
  • Fast once initial connection is made
  • Decent user base

Disadvantages

  • “Compiles” WDSL on every connection made
  • CPU and memory footprint is large (see benchmarks)
  • Need more and more examples

pyvisdk

source: https://github.com/xuru/pyvisdk

docs: http://xuru.github.com/pyvisdk/

irc: n/a

mailing list or forum: none I could find

Another pure Python library that uses Suds for the SOAP client. The author recognizes psphere for giving him some idea on how to communicate with vSphere. Also on Github. MIT license. Also actively maintained.

Advantages

  • Pretty easy to code with

Disadvantages

  • “Compiles” WDSL on every connection made
  • Slowest of the pure Python libraries
  • Really high memory and cpu footprint

pysphere  

source: https://code.google.com/p/pysphere/source/checkout

docs: https://code.google.com/p/pysphere/wiki/TableOfContents

irc: n/a

mailing list or forum: https://groups.google.com/forum/?fromgroups=#!forum/pysphere

Pysphere seems to be the oldest of the pure Python libraries, released into the open source world in 2011. It uses a fork of the ZSI Soap library for the Soap client layer. Seems to be actively maintained. Source is on the Google Code site which (is less than optimal for forking). BSD license.

Advantages

  • Actively maintained
  • Pretty easy to work with
  • Fastest of the pure Python libraries
  • Lowest memory and cpu footprint

Disadvantages

  • “Compiles” WDSL on every connection made

vijava 

source: http://sourceforge.net/projects/vijava/

docs: http://vijava.sourceforge.net/

irc: n/a

mailing list or forum: http://sourceforge.net/projects/vijava/forums/forum/826592

Vijava is actually a Java library that utilizes the VMware vSphere Java SDK to communicate with the web service. Vijava was created by Steve Jin while he was at VMware Labs, though the project is not officially supported by VMware.

The interesting thing is that the SOAP WSDL is “pre-loaded” as code so that it doesn’t need to be compiled at runtime. You can write Python scripts which calls vijava using the Jython interpreter. This library seems to have the largest user base but code commits are made much less frequently, usually to coincide with vSphere releases. License BSD.

Advantages

  • Fast, once the Java interpreter is running
  • Good for longer running processes

Disadvantages

  • Java interpreter is slow (IMHO) to start up
  • Not so good for small scripts or single API calls
  • Non-python software to install and configure: Java Runtime Environment, Jython
  • Have to have your Java class path set up “just right”

Examples

Let look at some sample code for each, performing the same exact task. Each script will establish a connection to the vCenter server, retrieve a list of virtual machines and  print some properties.

psphere – 0.5.2

from psphere.client import Client

from psphere.managedobjects import VirtualMachine
client = Client(“192.168.0.104”, “root”, “vmware”)
vms = VirtualMachine.all(client)
for x in vms:
    print x.config.guestFullName, x.name, x.overallStatus, x.config.template

pyvisdk – 0.9.20

from pyvisdk.vim import Vim

server = Vim(server=”192.168.0.104″)
server.login(“root”,”vmware”)
vms = server.getVirtualMachines()
for x in vms:
print x.config.guestFullName, x.name, x.overallStatus, x.config.template

pysphere – 0.1.6

from pysphere import VIServer

server = VIServer()
server.connect(“192.168.0.104”, “root”, “vmware”)
vms = server.get_registered_vms()
for x in vms:
vm = server.get_vm_by_path(x)
print vm.properties.config.guestFullName, vm.properties.name, \
vm.properties.overallStatus, vm.properties.config.template

vijava – 5.0.1

import sys
sys.path.insert(0, ‘/usr/local/lib/vijava/vijava50120120518.jar’)
sys.path.insert(0, ‘/usr/local/lib/vijava/dom4j-1.6.1.jar’)
from java.net import URL
from com.vmware.vim25.mo import *
server = ServiceInstance(URL(“https://192.168.0.104/sdk&#8221;),”root”,”vmware”, True)
rootFolder = server.getRootFolder()
vms = InventoryNavigator(rootFolder).searchManagedEntities(“VirtualMachine”);
for x in vms:
print x.config.guestFullName, x.name, x.overallStatus, x.config.template

Benchmarks
I tested the above scripts on OSX going against the vCenter appliance running inside Virtual Box. The ESX server is on the local network. There are two virtual machines returned in the results.
psphere – 8 seconds, 160M memory used
pyvisdk – 11 seconds, 550M memory
pysphere – 2 seconds, 50M memory
vijava –  5 seconds, 130M memory
Conclusion
Pysphere would seem to win the race. When I get time I will dig deeper into why the results are what they are. In the meantime please share your thoughts.

A Technology Switch: Cato Evolves from .Net to Python

The automation toolset now called Cato has come on a long journey since we started development in the early 2000s. The user interface was first written in VB / ASP to deliver self-service automation to our hosting customers.  We then moved pieces into Flash to address lack of tree control support (ugghh!).  Cato then moved to .Net. All the while long we kept up with the advances in Javascript, Ajax client-side processing, web 2.0, etc.

In 2011 we decided to release Cato as an open source project under the Apache 2.0 license. We faced an internal decision: Do we release the current software with the .Net component or do we take additional time to rewrite that part? Image

One thing was certain: we had to make it so the whole application ran on a single linux instance. This doesn’t scale but it allows people to try the software out without have a multiple machine install. The ideal scenario would be to be completely off .Net, but that could take months.

We decided to proceed with the .Net release to get the software into the open source world and start reaping the benefits of the feedback cycle. We figured out how to get Cato running on linux using Apache / mod_mono, which worked out nicely.

We received some feedback that the .Net codebase was holding back the potential contributor community. A couple of months ago we dove in and started converting the user interface and services to Python.

Why Python? Why not Ruby? Or PHP? Or Java? Or X?

We picked Python because:

  1. it’s a solid language with a large community and lots of extensions
  2. it’s interpreted, no compiling
  3. solid performance and lends well to scale
  4. easy to learn
  5. believe it or not, easily translates to and from C#

I must say the conversion has been easier that I expected. Well, let’s not say easy. We created a conversion script that did some of the work translating C# to Python and then the rest was elbow-grease.

Why did we want to tackle this project?

  •  Widen the developer community of Cato
  • Address a bias against .Net / Mono software
  • Make Cato easier to package, install and deploy
  • Allow for a more agile development process
  • Tighter integration with the backend services components of Cato

If you are of the hardy sort, go ahead and check out our github page and pull the latest. If you are a Pythonista you’ll notice some deviation from Pep 8. Either bear with us or  feel free to dig in and contribute!

If you are a .Net person interested in how we tackled this conversion, please feel free to contact me.

Hello world!

After many months of letting potential blog postings pile up I am forcing myself to start getting them out in the world. Some posting will be Cloud Computing related, some general technology related and some regarding Cloud Sidekick news.

I am excited to get some of these thoughts out of Evernote and into the world.

Hello World!