Lingwo.ws tutorial, part 1: Creating a service

This is the first part in a four part series on Lingwo.ws.  In this part, we will:

• Install the reference implementation.
• Create a simple example service.
• Use "curl" to experiment using the service.

What is Lingwo.ws?

Lingwo.ws is an open specification for a way to design RESTful web services.  I've described earlier why this is desirable, but to summarize:

• Shared expectations: RESTful web services can be just about anything, so long as they follow a few design principles.  So, when someone says a service is RESTful, you will still need to read its documentation and learn how specifically that service works.  If a service conforms to Lingwo.ws, however, this means it conforms to a specific design.
• Shared code: Since all Lingwo.ws compliant services follow the same design, we can create re-usable libraries for both service authors and service consumers.  This means that both parties have a lower barrier to entry (assuming there exists a Lingwo.ws library for their language and environment).

That said, the main design goals of Lingwo.ws are:

• Capable of elegantly modeling atleast 80% of domains. (I recognize that no one standard can most elegantly model every system.  There will always be some uncommon systems that would be better modeled with a custom API.  The goal here is to create something that the majority of people could use if they wanted.)
• Flexible and intuitive.
• Self-descriptive and discoverable.
• Secure.

While the rest of these articles are going to be primarily concerned with the technical details and specifics of the specification, I want to make it clear that the Lingwo.ws design isn't itself something unique, new or overly special -- in fact, much of it draws on existing web-services in the wild.

The true value of Lingwo.ws comes from its potential as a standard, which could (assuming heavy adoption!)  allow people to take advantage of the shared expectations and shared code, mentioned above.

However, the Lingwo.ws specification and its reference implementation are still under heavy development.  You can find the current draft of the specification here.  If you want to get involved in the project, please send an e-mail to the Lingwo Google group.

Installing the reference implementation

The easiest way to start to learn about the design of Lingwo.ws, is by playing with it.  For that reason, these tutorials are going to focus on using the reference implementation, written in Python.

You're on your own installing Python, version 2.4 or greater.  If you're running a modern-ish Linux, it should be available from your distribution.  Otherwise, check out Python's download page.

We are about to install a bunch of Python packages: the Lingwo.ws reference implementation and its dependencies.  I highly recommend using virtualenv and creating a separate Python environment rather than mucking up your system environment.  In fact, I recommend it so much, that we are going to begin by installing virtualenv into your system Python environment (so you need to be "root", but only just for this one install).

$ easy_install virtualenv

This makes the "virtualenv" script available. Run this script to create your new virtual python environment (the below command will put it in the "lingwo_python" directory):

$ virtualenv lingwo_python

To use this virtual environment, you run "lingwo_python/bin/python" instead of "python" and "lingwo_python/bin/easy_install" instead of "easy_install".

Next, we install those dependencies:

$ lingwo_python/bin/easy_install simplejson webob wsgiref

Then we need to get the latest Lingwo.ws code. The code is hosted in a Bazaar repository (a distributed revision control system like Git or Mercurial). You can download Bazaar here.  Once you've got it, check out the code like this:

$ bzr co lp:lingwo.ws

Then install it:

$ cd lingwo.ws/src/python
$ ../../../lingwo_python/bin/python src/python/setup.py develop

For those of you familiar with Python installs, you may notice that rather than "setup.py install", we are using "setup.py develop". This installs a sort-of link, rather than copying the actual files. This allows you to update the Lingwo.ws code (done using "bzr update" in the checkout directory) and not have to install it again. This is important because Lingwo.ws is currently in a heavy state of flux and I wouldn't recommend installing it as it is now.

Running a service

The next logical step in this tutorial would be to go through the creation of a simple example service.  But, we're going to be a little out of the ordinary today, and before looking at the code for the example, we are going to run it and experiment with it.  Don't worry!  In the next section will go through the construction of the script itself.

So, without further ado, download this script and run it from the command-line:

$ lingwo_python/bin/python tutorial_1.py
Starting simple example service on port 5557...

Hurrah!

A little background...

Lingwo.ws models resources as documents and containers.  A document is basically a collection of name-value pairs, where the values must conform to a simple set of types.  The document can be represented in a number of serializations, but the default is JSON.  A container holds a set of documents that all conform to a similar format.

Here is an example document:

{
    "_id": "27",
    "_href": "http://127.0.0.1:5557/people/27",
    "fname": "John",
    "lname": "Smith",
    "age": 27,
    "eye_color": "blue",
    "sex": "male",
    "address": {
        "line1": "123 Fake Street",
        "line2": "Apt. 27A",
        "city": "Milwaukee",
        "state": "Wisconsin",
        "country": "USA"
    }
}

There are couple things to note about:

• All properties that begin with an underscore ("_") have special meaning given by Lingwo.ws.  The rest are declared by the service author.
• All documents have an "_id" property which is the ID of the document inside its container.
• All documents have an "_href" property which is the URL that refers to this document globally.
• Values can be almost anything representable in JSON, including: strings, integers, floats, objects, arrays, etc...

Playing with our service!

We are going to use curl to play with the service. This is available under most (if not all) Linux distributions but may need to be installed (ie. "apt-get install curl" for Debian and Ubuntu users).  It should be installed under MacOS X by default.

You can download Windows binaries here. We are going to be doing this from the command-line, so I'll leave it up to all you Windows Wizards out there to install it in the correct location for that work.

Anyway, open a new command-line window (you can't use the same one you used to run the service -- just let that one be) and we'll get down to business!

Retreiving information

Lingwo.ws is RESTful, so the most basic way to retrieve data is with an HTTP GET request.

Our example service has a container, named "people", located at http://127.0.0.1:5557/people.  You can query a list of all the documents in this container, by performing a GET on the container URL.  Using curl:

$ curl http://127.0.0.1:5557/people
{"documents": [
    {"_id": "27", "_href": "http://127.0.0.1:5557/people/27"},
    {"_id": "28", "_href": "http://127.0.0.1:5557/people/28"},
    {"_id": "29", "_href": "http://127.0.0.1:5557/people/29"}
], "limit": 0, "offset": 0}

(NOTE: I am re-arranging the output to make it easier to read. If you are following along at home, you will see everything crammed onto a single line.)

This is a just a list of IDs and URLs for the documents. To grab a full document, for example, for ID 28, perform a GET on the URL listed in its "_href":

$ curl http://127.0.0.1:5557/people/28
{
    "_id": "28",
    "_href": "http://127.0.0.1:5557/people/28",
    "lname": "Staszewski",
    "fname": "Kaja",
    "age": 63,
    "sex": "female",
    "eye_color": "grey",
    "address": {
        "line1": "Konstantynow 3",
        "city": "Lublin",
        "country": "Poland"
    }
}

Or! You could have requested the service to expand its sub-refs. A sub-ref is a reference to another document or container on the same service, which is actually what is being returned when you perform a GET on a container.

$ curl http://127.0.0.1:5557/people?_limit=2\&_offset=1\&_expand_subrefs=true
{"documents": [
  { "_id": "28",
    "_href": "http://127.0.0.1:5557/people/28",
    "lname": "Staszewski",
    "fname": "Kaja",
    "age": 63,
    "sex": "female",
    "eye_color": "grey",
    "address": {
        "line1": "Konstantynow 3",
        "city": "Lublin",
        "country": "Poland"
    }
  },
  { "_id": "29"
    "_href": "http://127.0.0.1:5557/people/29",
    "lname": "Kowalski",
    "fname": "Jan",
    "age": 23,
    "sex": "male",
    "eye_color": "grey",
    "address": {
        "line1": "Niecala 8",
        "city": "Lublin",
        "country": "Poland"
    }
  }
], "limit": 2, "offset": 1}

(NOTE: The backslash ("\") in front of ampersands ("&"). This is important under a UNIX shell, because without the backslash it will interpret the ampersand as meaning you want to run "curl" in the background.  You should omit the backslash when not using the UNIX shell, ie. under Windows, the web-browser, etc.)

The above example also shows how you can use "_limit" and "_offset" to only select specific elements.  This is useful for making interfaces with multiple pages.

You can also filter the results returned. For example, if you wanted to get a list of all the men, you could do:

$ curl http://127.0.0.1:5557/people?sex=male
{"documents": [
    {"_id": "27", "_href": "http://127.0.0.1:5557/people/27"},
    {"_id": "29", "_href": "http://127.0.0.1:5557/people/29"}
], "limit": 0, "offset": 0}

Or how about all the people with grey eyes?

$ curl http://127.0.0.1:5557/people?eye_color=grey
{"documents": [
    {"_id": "28", "_href": "http://127.0.0.1:5557/people/28"},
    {"_id": "29", "_href": "http://127.0.0.1:5557/people/29"}
], "limit": 0, "offset": 0}

Filtering is rather limited. For more complex queries, there is a separate mechanism which will be covered in a later tutorial. However, here are a few URLs that represent slightly more advanced filters:

• All men with grey eyes: http://127.0.0.1/people?sex=male&eye_color=grey
• All women with either brown or blue eyes: http://127.0.0.1/people?sex=female&eye_color=brown&eye_color=blue

Feel free to experiment!

One last thing: you can also pull just a single property from a document.  Here are a couple examples:

$ curl http://127.0.0.1:5557/people/27/lname
"Smith"
$ curl http://127.0.0.1:5557/people/27/fname
"John"
$ curl http://127.0.0.1:5557/people/27/address
{"city": "Milwaukee", "state": "Wisconsin", "line2": "Apt. 27A", "line1": "123 Fake Street", "country": "USA"}
$ curl http://127.0.0.1:5557/people/27/address/city
"Milwaukee"

Modifying data

In the spirit of REST, to make any changes you use the other HTTP methods: POST, PUT and DELETE.

To modify a document, you have to GET the old document, make your modification and PUT the whole document back to the same URL. For example, to change "John Smith"'s name to "Tom Foolery":

$ curl -X PUT http://127.0.0.1:5557/people/27 -H 'content-type: application/json' -d '{"_href": "http://127.0.0.1:5557/people/27", "age": 27, "sex": "male", "lname": "Foolery", "eye_color": "blue", "fname": "Tom", "address": {"city": "Milwaukee", "state": "Wisconsin", "line2": "Apt. 27A", "line1": "123 Fake Street", "country": "USA"}, "_id": "27"}'

There are a few things to note here:

• -X PUT sets the method to PUT. For compatibility, Lingwo.ws also accepts POST to do updates because some environments have issues generating PUT requests.
• By default curl sends data with a Content-Type of "application/x-www-form-urlencoded". So, we must explicitly change it with -H 'content-type: application/json' so that the service will accept the document.
• -d precedes the data you are sending. You must pass the whole document back with your changes, but you can omit the "_href" and "_id" properties if you like. When doing updates, these properties are ignored.

Creating a new document is done by POST'ing to the container with the data for the new document.

$ curl -X POST http://127.0.0.1:5557/people -H 'content-type: application/json' -d '{"age": 22, "sex": "female", "eye_color": "blue", "fname": "Jennifer", "lname": "Smith" }'
{"_id": 30, "_href": "http://127.0.0.1:5557/people/30"}

As you can see above:

• You can omit the "_id" and "_href" properties (if you provide them, they will be ignored).
• A sub-ref to the newly created document is returned.

To delete a document, simply perform a DELETE:

$ curl -X DELETE http://127.0.0.1:5557/people/27

Simple, eh?

Creating our simple service

Now, I bet you're wondering how our script works!

We start by declaring the format of the "people" documents.  This is done by creating a dict which connects property names to a type object.

from Lingwo.ws.server import *

people_properties = {
    'fname': StringType(required=True),
    'lname': StringType(required=True),
    'age': NumberType(required=True),
    'eye_color': StringType(required=True),
    'sex': StringType(required=True),
    'address': ObjectType(properties={
        'line1': StringType(),
        'line2': StringType(),
        'city': StringType(),
        'state': StringType(),
        'country': StringType()
    })
}

For the most part that should pretty self explanatory. All type classes have a few standard named arguments, but right now you should only worry about "required". Setting required to True, means that the client must provide the property when attempting to create a document of this type.

Here are a few of the available types (the rest, not mentioned here, will be covered in later tutorials):

• StringType
• NumberType
• BooleanType
• ObjectType
• ArrayType

Next, we define some fake data for the "people" container to hold.  For simplicity, we are just going to make an in-memory list of dict's.  A real service would store the data in some persistent backend, like a database.

people = [
    {
        '_id': '27',
        'fname': 'John',
        'lname': 'Smith',
        'age': 27,
        'eye_color': 'blue',
        'sex': 'male',
        'address': {
            'line1': '123 Fake Street',
            'line2': 'Apt. 27A',
            'city': 'Milwaukee',
            'state': 'Wisconsin',
            'country': 'USA'
        }
    },
    # etc...
]

You'll notice that we define the "_id" parameter, which is just an opaque string that only has meaning within its container.  Here its based on an incrementing integer.  As the service author, we don't have to specify the "_href", this will be generated just before the data is passed back to the client.

Next, we tie this all together in a container. We use "SimpleContainer" which is designed for this exact situation (an in-memory list of dict's). But to allow us to add new documents to the container, we must also provide a function to generate new IDs.

people_id = 29
def people_id_generator(container, doc):
    global people_id
    people_id += 1
    return str(people_id)

people_container = SimpleContainer(people_properties, people_id_generator, people)

Bam! Now, we need to create a service that holds this container and attach it to a web-server.

Since the Lingwo.ws reference implementation is built on WSGI, it can be used inside of any WSGI web-server (mod_python, Paste, CherryPY, etc). For simplicity, we are going to use the WSGI reference implementation. However, this isn't suited in any way to production use, because it can only handle one request at a time. If you want to run in production, please use something else (I like Paste, personally).

def main():
    port = 5557

    # The arguments are: version, base URL, and dict of top-level containers
    service = Service('1.0', 'http://127.0.0.1:%i' % port, {
        'people': people_container
    })

    print "Starting simple example service on port %i..." % port
    from wsgiref.simple_server import make_server
    httpd = make_server('', port, service)
    httpd.serve_forever()

if __name__ == '__main__': main()

That's it! I know, not that much interesting going on in there!

The next tutorial will be focused on creating a real container which stores its data in a database using SQLAlchemy. We will begin to get deeper into the reference implementation, use some of the more sophisticated features of Lingwo.ws and will (hopefully!) feature some much more interesting code.