Skip to main content

Using WSE3 services with Python

Like many companies, we use a number of Web Services at work. These are mainly used for our client side applications to talk back to things like SQL and other services, without going direct to them (and thus having a "security layer" of sorts). Having the ability to use these in some scripts/processes that you design can often be useful, as you have to play by the business logic.

In our case, our web services are .NET Web services using WSE3, to add on WS-Addressing and WS-Security features to them. These tie in easily to C# or other .NET languages, just by adding them into your Application. However, in a SysAdmin/DevOps role, you most likely aren't wanting to use a language like C#. You most likely want something like Python, Ruby or node.js; or you may be wanting to integrate these into something that you want to run on Linux or Solaris.

That was my thought. I wanted something in Python, to be able to run it from a few of our Linux boxes. On having a look around the internet, it seems that there is actually not much documentation on getting WSE3 (with WS-Addressing and WS-Security) web services to work in Python. So this article changes that.

In this article I will provide an example script which will show you how Python can talk to WSE3 scripts. It should hopefully point you in the right direction, that you need to go to achieve your goal.

What you need

Before we start, you will need to have available the following python libraries:

It will also help to have a small bit of knowledge about how SOAP works, but this is not required.

Our final script

We will start out, by showing the full script and then explaining each bit of how it works.

 #!/usr/bin/python

 import logging
 import random
 import string

 from suds import MethodNotFound
 from suds.client import Client
 from suds.wsse import Security, UsernameToken
 from suds.sax.element import Element
 from suds.sax.attribute import Attribute
 from suds.xsd.sxbasic import Import

 WEBSERVICE_URL = 'http://www.example.com/Webservice/Webservice.asmx'
 NS_WSA = ('wsa', 'http://schemas.xmlsoap.org/ws/2004/08/addressing')
 MUST_UNDERSTAND = Attribute('SOAP-ENV:mustUnderstand', 'true')

 def main():
     logging.basicConfig(level=logging.INFO)
     logging.getLogger('suds.client').setLevel(logging.DEBUG)

     client = Client('%s?wsdl' % WEBSERVICE_URL)

     add_security(client, 'DOMAIN\User', 'Password')
     add_addressing(client, WEBSERVICE_URL)
     method = get_method(client, 'method')

     print method()

 def add_security(client, user, passwd):
     sec = Security()
     token = UsernameToken(user, passwd)
     token.setnonce()
     token.setcreated()
     sec.tokens.append(token)
     client.set_options(wsse=sec)

 def add_addressing(client, webservice_url):
     headers = []

     addr = Element('Address', ns=NS_WSA).setText('http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous')

     headers.append(Element('Element').addPrefix(p='SOAP-ENC', u='http://www.w3.org/2003/05/soap-encoding'))
     headers.append(Element('ReplyTo', ns=NS_WSA).insert(addr).append(MUST_UNDERSTAND))
     headers.append(Element('To', ns=NS_WSA).setText(webservice_url).append(MUST_UNDERSTAND))
     headers.append(addr)
     headers.append(Element('MessageID', ns=NS_WSA).setText('urn:uuid:%s' % generate_messageid()))

     client.set_options(soapheaders=headers)

 def get_method(client, method):
     try:
         m = getattr(client.service, method)
         action = client.wsdl.services[0].ports[0].methods[method].soap.action
         action = action.replace('"', '')
     except MethodNotFound:
         return None

     action_header = Element('Action', ns=NS_WSA).setText(action)
     client.options.soapheaders.append(action_header)

     return m

 def generate_messageid():
     fmt = 'xxxxxxxx-xxxxx'
     resp = ''

     for c in fmt:
         if c == '-':
             resp += c
         else:
             resp += string.hexdigits[random.randrange(16)]

     return resp

 if __name__ == '__main__':
     main()

How the script works in detail

Now that we have seen the full script, we will go thru it, and explain what it does.

import logging
import random
import string

from suds import MethodNotFound
from suds.client import Client
from suds.wsse import Security, UsernameToken
from suds.sax.element import Element
from suds.sax.attribute import Attribute
from suds.xsd.sxbasic import Import

First we start by including some libraries we are going to use. We use the random library for generating a random message ID for our SOAP message. The includes from suds.* are for the SOAP library we will be using.

We have also included the logging library, which will enable us to be able to see some debug messages should we wish to.

WEBSERVICE_URL = 'http://www.example.com/Webservice/Webservice.asmx'
NS_WSA = ('wsa', 'http://schemas.xmlsoap.org/ws/2004/08/addressing')
MUST_UNDERSTAND = Attribute('SOAP-ENV:mustUnderstand', 'true')

Next, we define our webservice url (WEBSERVICE_URL). This should be the full path to the endpoint. This should not be the URL to it's WSDL.

We also define some attributes for our XML Elements that we will needed further down in our script. NS_WSA is the namespace we are defining for WS-Addressing. The MUST_UNDERSTAND variable is used to make Elements which the Web Service must understand and process.

def main():
    logging.basicConfig(level=logging.INFO)
    logging.getLogger('suds.client').setLevel(logging.DEBUG)

This is our main() function, that will be called when we start the script. The first two lines, will provide our debug output for running the script. These can be commented out afterwards should you want to.

client = Client('%s?wsdl' % WEBSERVICE_URL)

We next create an instance of the suds.client.Client object, which will be used for connecting and inspecting our web service.

add_security(client, 'DOMAIN\User', 'Password')
add_addressing(client, WEBSERVICE_URL)
method = get_method(client, 'method')

print method()

The last bit of our main() function, will be where we call our other functions to add in support for WS-Addressing and WS-Security. Then the last bit will be finalising the method we are going to call, and calling the method and printing out the results.

You would change/set the DOMAINUser and Password, as well as the method here, so it runs with the right credentials and the correct method of the web service.

def add_security(client, user, passwd):
    sec = Security()
    token = UsernameToken(user, passwd)
    token.setnonce()
    token.setcreated()
    sec.tokens.append(token)
    client.set_options(wsse=sec)

System Message: WARNING/2 (<string>, line 111); backlink

Duplicate explicit target name: "ws-security".

This is our add_security() function which will take care of implementing WS-Security into our request. Luckily for us, WS-Security support is available in Suds already, so we just need to add it to our client object.

The token we add is our username and password (username is prefixed with the Domain when it is sent to the function). We also set a "Nonce" and the created date and time. These together help to prevent replay attacks with the SOAP request.

Finally we add our token to our security object, and add our security object, into our client object.

def add_addressing(client, webservice_url):
    headers = []

    addr = Element('Address', ns=NS_WSA).setText('http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous')

    headers.append(Element('Element').addPrefix(p='SOAP-ENC', u='http://www.w3.org/2003/05/soap-encoding'))
    headers.append(Element('ReplyTo', ns=NS_WSA).insert(addr).append(MUST_UNDERSTAND))
    headers.append(Element('To', ns=NS_WSA).setText(webservice_url).append(MUST_UNDERSTAND))
    headers.append(addr)
    headers.append(Element('MessageID', ns=NS_WSA).setText('urn:uuid:%s' % generate_messageid()))

    client.set_options(soapheaders=headers)

The next function we call in our script, is the add_addressing() function. This function adds _most_ required headers for WS-Addressing. There is one that is added the get_method() function, but we will cover that shortly.

First we start by defining an "Address" Element, which is a static reference to the WS-Addressing specification. Note we also set the namespace to the NS_WSA variable we created befoer. After that, we create a generic Element tag, which is used to define the SOAP Encoding we are using.

Next are the two most important elements we will add to the header. The ReplyTo Element is added, with reference to the Address element, and the mustUnderstand attribute. The ReplyTo element defines the endpoint for the reply to the web service. The To Element is where we define the address of the web service we are calling.

We finish up by adding the Address element into the headers, and defining a unique MessageID for the message we are able to send. We then add these headers into our client object.

def get_method(client, method):
    try:
        m = getattr(client.service, method)
        action = client.wsdl.services[0].ports[0].methods[method].soap.action
        action = action.replace('"', '')
    except MethodNotFound:
        return None

    action_header = Element('Action', ns=NS_WSA).setText(action)
    client.options.soapheaders.append(action_header)

    return m

This function, get_method(), is used to be able to find the action path of the method we want to call, and add this int our SOAP headers.

We first look up to see whether the method we have been given, is a valid method that exists when the WSDL file is introspected. If it is, we store this in m, for returning back later.

Before finishing up however, we look in the WSDL definition for the service, and get the SOAP action attribute for the given method. We remove and bad characters and add this into our SOAP Headers as the Action element. This tells WS-Addressing the function we want to call to do our work.

def generate_messageid():
    fmt = 'xxxxxxxx-xxxxx'
    resp = ''

    for c in fmt:
        if c == '-':
            resp += c
        else:
            resp += string.hexdigits[random.randrange(16)]

    return resp

The generate_messageid() function is used by add_addressing(), to generate a Unique and random message ID. This function simple look over each character in our defined format (fmt), and generates a unique hex character to be returned. Literal dashes (-) in the format string are ignored.

if __name__ == '__main__':
    main()

To finish off, this is stock standard python. If this script is invoked from the command line, the built-in variable __name__ will be set to __main__. We simply say here, that if it is invoked from the command line, run our main() function.

Now, if you have updated some of the variables where needed in the above, you will now be able to run the script, and it should print our the resulting dataset from your web service. If you get any errors, have a look and see if you can work out what may be causing it. Do check that it's something as simple as maybe the wrong username or password tho.

Where to from here?

Now that you have the script, you will most likely want to use what you have learnt here, to form this into a library of functions, so you can use it in your application. And get the values from somewhere else, than what is hardcoded in the script.

You will also want to have a play around with calling method's with parameters and also seeing what happens when you start to need to use more complex data types that the web service method defines. But you should be able to handle this, as Suds includes this introspection in it's API support.

Should you have any questions, feel free to contact me, and i'll see what I can do!

I should also thank the below articles/posts on websites for helping, as without them, I may have had no clue to some bits of this.

Now available via IPv6

Quick Update - This site is now available via IPv6.

You don't have to do anything special, you will just use IPv6 if it is available and you have preferences for IPv6 connections.

If you want to test to see if you can get to the IPv6 site, try visiting http://v6.nullis.net

Steve, Thanks.

It was in High School that my passion for Technology, especially computing, was the most prominent. While I had dabbled with a few PC's and learned BASIC before then, it was High School I became intrigued even more, and was involved in setting up over 300 Macintosh PC's, from LC575's, PowerMac' to iMac's (Bondi Blue ones too!), and some G3's.

Having that experience definitely changed my Life Path, and opened a few more doors than would have been otherwise.

Update: Then again, the first PC I used at Primary School was an Apple IIe. It was the bomb.

And for that, I say: Steve, Thanks.

Steve Jobs, co-founder of Apple, passed today.

Update on Microserver Build

Just in case you are wondering where the first post on the N36L build is, it has been unfortunately a bit delayed! There have been a few developments since the last post, that have been blockers to progress.

To start with, the RAM I initially purchased was incorrect, and now I know the difference the difference with regards to different types of ECC RAM! This is now resolved and I purchased some Corsair XMS3 RAM (2x4G), and will be using this RAM in the build.

Secondly, VMWare recently launched ESXi 5 for general availability. Previously I was going to use ESXi 4.1, but figured that using ESXi 5 would be good, as it had just been released. However, that said, ESXi 5 support in libvirt isn't there yet, so I am going to have to go back to using ESXi 4.1 as per the previous plan.

I'll try and get the build post, with photos up in the next 2 weeks.

Hang in there, it won't be long!

Motorola Mobility has been acquired by Google

Seems google have done the impossible.. well.. what was thought of as the impossible, anyway.

They have purchased Motorola Mobility (Motorola's Mobile Phone Arm), for $12.5billion

The buy is mostly for all the Patents, but should also be scaring the likes of HTC, Samsung and etc at the moment.

My recent purchase - HP Microserver N36L

About a month ago I ordered a [HP Proliant N36L][hpn36l], commonly known as the HP Microserver. I had been looking to replace my aging P3-900mhz Gateway and all-round utility server for quite some time, and when I saw this on special for only $200, I knew this was going to be the perfect replacement.

With the ability for it to have the system boot off an Internal USB device, being able to take up to 8GB of DDR3 RAM, and have 8TB of hard drive space, it will be able to meet the requirements of the current box, and add on a few new abilities in the process.

While I have been waiting for it to arrive, I figured that this would be a good opportunity to do a few guides about the configuration of the machine, both at a Hardware and Software level, and be able to provide a few guides to virtualisation.

Hardware wise, the current plan for the server I have will be to do the following upgrades:

  • Change the standard 1GB of Ram to 8GB of ECC Ram;
  • Remove the 250GB hard drive that comes with it, and replace it with 4 x 2TB hard drives;
  • Into one of the PCI Express ports, add on an additional Network Card (NIC), which will take the machine from 1 Ethernet Interface to 3; and
  • Add in a USB Drive to the internal USB Slot to give us a 16GB System disk, that will provide fast access for the system (and hopefully provide a quick boot).

Software wise, I want to have a go and setting this machine up as a Virtual Machine Host, using Virtual Machine Platforms such as [Xen][xen], [VMWare ESXi][esxi] and [KVM][kvm].

Once we have the machine built up from the base, I will then use the opportunity to go over some other deployment and centralised configuration tools. One example of this will be [Puppet][plabs].

And so, after quite a long wait, the Microserver finally arrived today.

So in the next few days (possibly the weekend), I will kick off with the first article on this series. If you have any suggestions of things you would like to see done with this box, [drop us a line via twitter][contact], and I'll try and give it a go for you.

[![](http://farm5.static.flickr.com/4154/5204509633_d63ce16b8a.jpg "Front of the HP ProLiant N36L by Samat Jain, on Flickr")][hpflickr]

Image from Flickr, by [Samat Jain][hpflickr]. Used under a Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0)

[hpn36l]: http://h10010.www1.hp.com/wwpc/au/en/sm/WF06b/15351-15351-4237916-4237917-4237917-4248009-5040202.html "HP Proliant N36L on hp.com" [hpflickr]: http://www.flickr.com/photos/tamasrepus/5204509633/ [xen]: http://xen.org/ [esxi]: http://www.vmware.com/products/vsphere/esxi-and-esx/index.html [kvm]: http://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine [contact]: http://www.nullis.net/contact/ [plabs]: http://www.puppetlabs.com/

Configure iSCSI with Solaris 10 to a Dell PowerVault

The other day I had to allocate some SAN Space from a [Dell PowerVault MD3000i][md3000i] to a Solaris 10 server, and was amazed that not only do DELL consider Solaris an unsupported platform for connecting to their devices, but the general level of documentation around iSCSI and Dell is pretty low.

So I've decided to document what I did to set it up.

This guide will presume that you have already configured the Virtual Disk, and allocated it to this machine via the DELL SAN Configuration Tool. Additionally, the below steps were for a Solaris Sun Fire V120 running Solaris 10 (120011-14), so your mileage may vary.

### Configure your Network Interface To use your SAN correctly, you should place your SAN connection on a separate network to your normal network traffic. This not only helps keep things secure and separate, but cuts out network congestion you want to avoid so that your SAN response times are fast.

If you aren't using a separate network connection, then jump to the next step.

#### Step 1 - Pre-config Start by determining the interface you are going to configure for the SAN. In this example, mine will be eri1 (the primary interface is eri0). Also determine the IP you will be allocating to this machine for that interface and a hostname for this interface. We will use 192.168.1.11 and server123i.

Next, you will need to configure a couple of files with the settings for this interface.

#### Step 2 - Create /etc/hostname.interface Create a new file called /etc/hostname.eri1 and the only thing to be in this file, is the hostname you determined above (server123i).

# cat /etc/hostname.eri1 server123i

#### Step 3 - Add entry to /etc/hosts Add a new entry to /etc/hosts mapping your new hostname, to it's IP.

# grep server123i /etc/hosts 192.168.1.11 server123i

#### Step 4 - Add entry to /etc/netmasks Add a new entry to /etc/netmasks to store the correct netmask for this IP address range.

# grep -v '^#' /etc/netmasks 10.0.1.0 255.255.255.0 192.168.1.0 255.255.255.0

#### Step 5 - Restart networking service Restart the networking service to have Solaris reload the settings for the interface.

# svcadm restart network/physical

#### Step 6 - Check your interface is now available If all went well, you will now have your interface, eri1 configured with the IP you specified, and in the UP state.

# ifconfig eri1 eri1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

System Message: ERROR/3 (<string>, line 48)

Unexpected indentation.
inet 192.168.1.11 netmask ffffff00 broadcast 192.168.1.255 ether 0:f:4d:a4:c4:9e

### Configure iSCSI on Solaris

This bit isn't too difficult, but there are some extra steps as we have to work around the DELL PowerVault not working 100% as it should, with Solaris.

#### Step 1 - Enable Solaris iSCSI Initiator First we need to check if the iSCSI Initiator service is running. If it isn't, then we need to enable it.

# svcs -a |grep iscsi disabled Jun_22 svc:/network/iscsi_initiator:default disabled Jun_22 svc:/system/iscsitgt:default

System Message: WARNING/2 (<string>, line 58); backlink

Inline substitution_reference start-string without end-string.

# svcadm enable svc:/network/iscsi_initiator

# svcs -a |grep iscsi online 15:10:00 svc:/system/iscsitgt:default online 15:15:10 svc:/network/iscsi_initiator:default

System Message: WARNING/2 (<string>, line 64); backlink

Inline substitution_reference start-string without end-string.

#### Step 2 - Add Discovery Address Next, we need to tell iSCSI Initiator where our SAN is (Discovery Address). In our example, we will use 192.168.11.1 on port 3260.

# iscsiadm add discovery-address 192.168.11.1:3260

#### Step 3 - Enable iSCSI Discovery At this point, if we were to do this with Auto Discovery (SendTargets Discovery) with iSCSI, we would just have to do iscsiadm modify discovery -t enable and our disk and SAN would automatically be found, and the Virtual Disk created as a Device (/dev/rdsk).

However, as this doesn't work, we have to use Statically Assigned Targets. To do this, we first need to get a list of all targets we can see.

# iscsiadm list target -v
Target: iqn.1984-05.com.dell:powervault.md3000i.8a4badb0ba454aa1000000004c1f8e4a

System Message: WARNING/2 (<string>, line 81)

Definition list ends without a blank line; unexpected unindent.
...
IP address (Local): 192.168.11.1:32812 IP address (Peer): 192.168.11.1:3260

Now that you have the target, you can add this address into the iSCSI Static Config. Note: at the end of the target name in the below, you need to add a comma, followed by the IP Address.

iscsiadm add static-config iqn.1984-05.com.dell:powervault.md3000i.8a4badb0ba454aa1000000004c1f8e4a,192.168.11.1

#### Step 4 - Configure DELL PowerVault SAN After we have done this, if you haven't previously, now is a good time to go and configure the Virtual Disk and Server on the DELL SAN Configuration Tool, otherwise the next steps will fail.

Once this is done, move onto the next step.

#### Step 5 - Enable Static Discovery

Now that you have Solaris configured and the SAN configured, you now need to turn on Static Discovery. To do this:

iscsiadm modify discovery -s enable

Once this is done, you should take a look at dmesg and you will notice that you have some messages towards the end about your disks coming online.

iscsi: [ID 240218 kern.notice] NOTICE: iscsi session(40) <target> online scsi: [ID 799468 kern.info] sd2 at iscsi0: name <target>,0, bus address <target>,0 genunix: [ID 936769 kern.info] sd2 is /iscsi/disk@0000<target>,0 scsi: [ID 107833 kern.warning] WARNING: /iscsi/disk@0000<target>,0 (sd2):

System Message: ERROR/3 (<string>, line 108)

Unexpected indentation.
Corrupt label; wrong magic number

System Message: WARNING/2 (<string>, line 109)

Block quote ends without a blank line; unexpected unindent.

genunix: [ID 408114 kern.info] /iscsi/disk@0000<target>,0 (sd2) online scsi: [ID 799468 kern.info] sd3 at iscsi0: name 0000<target>F,31, bus address 0000<target>,31 genunix: [ID 936769 kern.info] sd3 is /iscsi/disk@0000<target>,31 scsi: [ID 107833 kern.warning] WARNING: /iscsi/disk@0000<target>,31 (sd3):

System Message: ERROR/3 (<string>, line 113)

Unexpected indentation.
Corrupt label; wrong magic number

System Message: WARNING/2 (<string>, line 114)

Block quote ends without a blank line; unexpected unindent.

genunix: [ID 408114 kern.info] /iscsi/disk@0000<target>,31 (sd3) online

The above shows two things. It shows that your disk has come online (,0) and that the DELL Universal Xport has come online (,31). The error messages about 'Corrupt label; wrong magic number' are safe to ignore, as our disk hasn't been formatted.

To see more details about the disks that have been mounted, you can get this by using the command iscsiadm list target -S. This will tell you the /dev/rdsk nodes associated with your disk(s), among a few other details.

#### Step 6 - Finishing Up - Partition and Format

Now that you have your disk showing up, you can simply use format and newfs (or however you wish), and use this disk for your need. This disk with work the same as a physical disk attached to the system.

Credits for some of the points in here to the following articles, I have simply tried to join it all into one article.

  1. [iSCSI Troubleshooting][link1]
  2. [comp.unix.solaris thread at Google Groups][link2]
  3. [Connecting to an iSCSI Target with Open-iSCSI Initiator using Solaris][link3]

[md3000i]: http://www.dell.com/content/topics/topic.aspx/global/products/pvaul/topics/en/us/pvaul_md3000i_landing?c=us&l=en [link1]: http://opensolaris.org/jive/thread.jspa?messageID=153729 [link2]: http://groups.google.com/group/comp.unix.solaris/browse_thread/thread/e14b16b24d94c64d/a4b77e511c758b73?lnk=gst [link3]: http://www.idevelopment.info/data/Unix/Solaris/SOLARIS_ConnectingToAniSCSITargetWithOpen-iSCSIInitiatorUsingSolaris.shtml