diff --git a/docs/historic/2002/ipc10/twisted-network-framework/errata.html b/docs/historic/2002/ipc10/twisted-network-framework/errata.html deleted file mode 100644 index 8388919e4bc..00000000000 --- a/docs/historic/2002/ipc10/twisted-network-framework/errata.html +++ /dev/null @@ -1,256 +0,0 @@ - - - - - - The World of Software is a World of Constant - Change - - - -

Note: This document is relevant for the - version of Twisted that was current at IPC10. It has since been - superseded by many changes to the Python API. It is remaining - unchanged for historical reasons, but please refer to - documentation for the specific system you are looking for and - not these papers for current information.

- -

The World of Software is a World of Constant Change

- -

Twisted has undergone several major revisions since Moshe - Zadka and I wrote the "The Twisted - Network Framework". Most of these changes have not deviated - from the central vision of the framework, but almost all of the - code listings have been re-visited and enhanced in some - way.

- -

So, while the paper was correct at the time that it was - originally written, a few things have changed which have - invalidated portions of it.

- -

Most significant is the fact that almost all methods which - pass callbacks of some kind have been changed to take no - callback or error-callback arguments, and instead return an - instance of a twisted.python.defer.Deferred. This means - that an asynchronous function can be easily identified visually - because it will be of the form: async_obj.asyncMethod("foo").addCallbacks(succeded, - failed). There is also a utility method addCallback which makes it more - convenient to pass additional arguments to a callback function - and omit special-case error handling.

- -

While it is still backwards compatible, twisted.internet.passport has been re-named - to twisted.cred, and the various - classes in it have been split out into submodules of that - package, and the various remote-object superclasses have been - moved out of twisted.spread.pb and put into - twisted.spread.flavors.

- -

Application.listenOn has been - replaced with the more descripively named Application.listenTCP, Application.listenUDP, and Application.listenSSL.

- -

twisted.web.widgets has progressed - quite far since the paper was written! One description - specifically given in the paper is no longer correct:

- -
- The namespace for evaluating the template expressions is - obtained by scanning the class hierarchy for attributes, and - getting each of those attributes from the current instance. - This means that all methods will be bound methods, so - indicating "self" explicitly is not required. While it is - possible to override the method for creating namespaces, - using this default has the effect of associating all - presentation code for a particular widget in one class, along - with its template. If one is working with a non-programmer - designer, and the template is in an external file, it is - always very clear to the designer what functionality is - available to them in any given scope, because there is a list - of available methods for any given class. -
-

This is still possible to avoid breakages in old code, but -after some experimentation, it became clear that simply passing - self was an easier method for - creating the namespace, both for designers and programmers.

-

In addition, since the advent of Zope3, interoperability - with Zope has become increasingly interesting possibility for - the Twisted development team, since it would be desirable if - Twisted could use their excellent strategy for - content-management, while still maintaining Twisted's - advantages in the arena of multi-protocol servers. Of - particular interest has been Zope Presentation Templates, since - they seem to be a truly robust solution for keeping design - discrete from code, compatible with the event-based method in - which twisted.web.widgets processes web requests. twisted.web.widgets.ZopePresentationTemplate - may be opening soon in a theatre near you!

- -

The following code examples are corrected or modernized - versions of the ones that appear in the paper.

- -
- Listing 9: A remotely accessible object and accompanying call - -
-# Server Side
-class MyObject(pb.Referenceable):
-    def remote_doIt(self):
-        return "did it"
-
-# Client Side
-    ...
-    def myCallback(result):
-        print result # result will be 'did it'
-    def myErrback(stacktrace):
-        print 'oh no, mr. bill!'
-        print stacktrace
-    myRemoteReference.doIt().addCallbacks(myCallback,
-                                          myErrback)
-
-
- -
- Listing 10: An object responding to its calling perspective -
-# Server Side
-class Greeter(pb.Viewable):
-    def view_greet(self, actor):
-        return "Hello %s!\n" % actor.perspectiveName
-
-# Client Side
-    ...
-    remoteGreeter.greet().addCallback(sys.stdout.write)
-    ...
-
-
- - -
- Listing 12: A client for Echoer objects. -
-from twisted.spread import pb
-from twisted.internet import main
-def gotObject(object):
-    print "got object:",object
-    object.echo("hello network".addCallback(gotEcho)
-def gotEcho(echo):
-    print 'server echoed:',echo
-    main.shutDown()
-def gotNoObject(reason):
-    print "no object:",reason
-    main.shutDown()
-pb.getObjectAt("localhost", 8789, gotObject, gotNoObject, 30)
-main.run()
-
-
- -
- Listing 13: A PB server using twisted's "passport" - authentication. -
-from twisted.spread import pb
-from twisted.internet import main
-class SimplePerspective(pb.Perspective):
-    def perspective_echo(self, text):
-        print 'echoing',text
-        return text
-class SimpleService(pb.Service):
-    def getPerspectiveNamed(self, name):
-        return SimplePerspective(name, self)
-if __name__ == '__main__':
-    import pbecho
-    app = main.Application("pbecho")
-    pbecho.SimpleService("pbecho",app).getPerspectiveNamed("guest").makeIdentity("guest")
-    app.listenTCP(pb.portno, pb.BrokerFactory(pb.AuthRoot(app)))
-    app.save("start")
-
-
- -
- Listing 14: Connecting to an Authorized Service -
-from twisted.spread import pb
-from twisted.internet import main
-def success(message):
-    print "Message received:",message
-    main.shutDown()
-def failure(error):
-    print "Failure...",error
-    main.shutDown()
-def connected(perspective):
-    perspective.echo("hello world").addCallbacks(success, failure)
-    print "connected."
-
-pb.connect("localhost", pb.portno, "guest", "guest",
-           "pbecho", "guest", 30).addCallbacks(connected,
-                                               failure)
-main.run()
-
-
- -
- Listing 15: A Twisted GUI application -
-from twisted.internet import main, ingtkernet
-from twisted.spread.ui import gtkutil
-import gtk
-ingtkernet.install()
-class EchoClient:
-    def __init__(self, echoer):
-        l.hide()
-        self.echoer = echoer
-        w = gtk.GtkWindow(gtk.WINDOW_TOPLEVEL)
-        vb = gtk.GtkVBox(); b = gtk.GtkButton("Echo:")
-        self.entry = gtk.GtkEntry(); self.outry = gtk.GtkEntry()
-        w.add(vb)
-        map(vb.add, [b, self.entry, self.outry])
-        b.connect('clicked', self.clicked)
-        w.connect('destroy', gtk.mainquit)
-        w.show_all()
-    def clicked(self, b):
-        txt = self.entry.get_text()
-        self.entry.set_text("")
-        self.echoer.echo(txt).addCallback(self.outry.set_text)
-l = gtkutil.Login(EchoClient, None, initialService="pbecho")
-l.show_all()
-gtk.mainloop()
-
-
- -
- Listing 16: an event-based web widget. -
-from twisted.spread import pb
-from twisted.python import defer
-from twisted.web import widgets
-class EchoDisplay(widgets.Gadget, widgets.Presentation):
-    template = """<H1>Welcome to my widget, displaying %%%%echotext%%%%.</h1>
-    <p>Here it is: %%%%getEchoPerspective()%%%%</p>"""
-    echotext = 'hello web!'
-    def getEchoPerspective(self):
-        return ['<b>',
-            pb.connect("localhost", pb.portno,
-                   "guest", "guest", "pbecho", "guest", 1).
-                addCallbacks(self.makeListOf, self.formatTraceback)
-            ,'</b>']
-    def makeListOf(self, echoer):
-        return [echoer.echo(self.echotext).addCallback(lambda x: [x])]
-if __name__ == "__main__":
-    from twisted.web import server
-    from twisted.internet import main
-    a = main.Application("pbweb")
-    a.listenTCP(8080, server.Site(EchoDisplay()))
-    a.run()
-
-
- - - diff --git a/docs/historic/2002/ipc10/twisted-network-framework/index.html b/docs/historic/2002/ipc10/twisted-network-framework/index.html deleted file mode 100644 index 15368047565..00000000000 --- a/docs/historic/2002/ipc10/twisted-network-framework/index.html +++ /dev/null @@ -1,1568 +0,0 @@ - - - - - - The Twisted Network Framework - - - -

Note: This document is relevant for the - version of Twisted that were current previous to IPC10. Even at the time of - its release, there were errata - issued to make it current. It is remaining unaltered for - historical purposes but it is no longer accurate.

- -

The Twisted Network Framework

- -
Moshe Zadka m@moshez.org
- -
Glyph Lefkowitz glyph@twistedmatrix.com
- -

Abstract

- -

Twisted is a framework for writing asynchronous, - event-driven networked programs in Python -- both clients and - servers. In addition to abstractions for low-level system calls - like select(2) and socket(2), it also - includes a large number of utility functions and classes, which - make writing new servers easy. Twisted includes support for - popular network protocols like HTTP and SMTP, support for GUI - frameworks like GTK+/GNOME and - Tk and many other classes designed to make network - programs easy. Whenever possible, Twisted uses Python's - introspection facilities to save the client programmer as much - work as possible. Even though Twisted is still work in - progress, it is already usable for production systems -- it can - be used to bring up a Web server, a mail server or an IRC - server in a matter of minutes, and require almost no - configuration.

- -

Keywords: internet, network, framework, - event-based, asynchronous

- -

Introduction

- -

Python lends itself to writing frameworks. Python has a - simple class model, which facilitates inheritance. It has - dynamic typing, which means code needs to assume less. Python - also has built-in memory management, which means application - code does not need to track ownership. Thus, when writing a new - application, a programmer often finds himself writing a - framework to make writing this kind of application easier. - Twisted evolved from the need to write high-performance - interoperable servers in Python, and making them easy to use - (and difficult to use incorrectly).

- -

There are three ways to write network programs:

- -
    -
  1. Handle each connection in a separate process
    2. - -
  2. Handle each connection in a separate thread
    3. - -
  3. Use non-blocking system calls to handle all connections - in one thread.
    4. -
- -

When dealing with many connections in one thread, the - scheduling is the responsibility of the application, not the - operating system, and is usually implemented by calling a - registered function when each connection is ready to for - reading or writing -- commonly known as event-driven, or - callback-based, programming.

- -

Since multi-threaded programming is often tricky, even with - high level abstractions, and since forking Python processes has - many disadvantages, like Python's reference counting not - playing well with copy-on-write and problems with shared state, - it was felt the best option was an event-driven framework. A - benefit of such approach is that by letting other event-driven - frameworks take over the main loop, server and client code are - essentially the same - making peer-to-peer a reality. While - Twisted includes its own event loop, Twisted can already - interoperate with GTK+'s and Tk's - mainloops, as well as provide an emulation of event-based I/O - for Jython (specific support for the Swing toolkit is planned). - Client code is never aware of the loop it is running under, as - long as it is using Twisted's interface for registering for - interesting events.

- -

Some examples of programs which were written using the - Twisted framework are twisted.web (a web server), - twisted.mail (a mail server, supporting both SMTP - and POP3, as well as relaying), twisted.words (a - chat application supporting integration between a variety of IM - protocols, like IRC, AOL Instant Messenger's TOC and - Perspective Broker, a remote-object protocol native to - Twisted), im (an instant messenger which connects - to twisted.words) and faucet (a GUI client for the - twisted.reality interactive-fiction framework). - Twisted can be useful for any network or GUI application - written in Python.

- -

However, event-driven programming still contains some tricky - aspects. As each callback must be finished as soon as possible, - it is not possible to keep persistent state in function-local - variables. In addition, some programming techniques, such as - recursion, are impossible to use. Event-driven programming has - a reputation of being hard to use due to the frequent need to - write state machines. Twisted was built with the assumption - that with the right library, event-driven programming is easier - then multi-threaded programming. Twisted aims to be that - library.

- -

Twisted includes both high-level and low-level support for - protocols. Most protocol implementation by twisted are in a - package which tries to implement "mechanisms, not policy". On - top of those implementations, Twisted includes usable - implementations of those protocols: for example, connecting the - abstract HTTP protocol handler to a concrete resource-tree, or - connecting the abstract mail protocol handler to deliver mail - to maildirs according to domains. Twisted tries to come with as - much functionality as possible out of the box, while not - constraining a programmer to a choice between using a - possibly-inappropriate class and rewriting the non-interesting - parts himself.

- -

Twisted also includes Perspective Broker, a simple - remote-object framework, which allows Twisted servers to be - divided into separate processes as the end deployer (rather - then the original programmer) finds most convenient. This - allows, for example, Twisted web servers to pass requests for - specific URLs with co-operating servers so permissions are - granted according to the need of the specific application, - instead of being forced into giving all the applications all - permissions. The co-operation is truly symmetrical, although - typical deployments (such as the one which the Twisted web site - itself uses) use a master/slave relationship.

- -

Twisted is not alone in the niche of a Python network - framework. One of the better known frameworks is Medusa. Medusa - is used, among other things, as Zope's native server serving - HTTP, FTP and other protocols. However, Medusa is no longer - under active development, and the Twisted development team had - a number of goals which would necessitate a rewrite of large - portions of Medusa. Twisted seperates protocols from the - underlying transport layer. This seperation has the advantages - of resuability (for example, using the same clients and servers - over SSL) and testability (because it is easy to test the - protocol with a much lighter test harness) among others. - Twisted also has a very flexible main-loop which can - interoperate with third-party main-loops, making it usable in - GUI programs too.

- -

Complementing Python

- -

Python comes out of the box with "batteries included". - However, it seems that many Python projects rewrite some basic - parts: logging to files, parsing options and high level - interfaces to reflection. When the Twisted project found itself - rewriting those, it moved them into a separate subpackage, - which does not depend on the rest of the twisted framework. - Hopefully, people will use twisted.python more and - solve interesting problems instead. Indeed, it is one of - Twisted's goals to serve as a repository for useful Python - code.

- -

One useful module is twisted.python.reflect, - which has methods like prefixedMethods, which - returns all methods with a specific prefix. Even though some - modules in Python itself implement such functionality (notably, - urllib2), they do not expose it as a function - usable by outside code. Another useful module is - twisted.python.hook, which can add pre-hooks and - post-hooks to methods in classes.

- -
-
-# Add all method names beginning with opt_ to the given
-# dictionary. This cannot be done with dir(), since
-# it does not search in superclasses
-dct = {}
-reflect.addMethodNamesToDict(self.__class__, dct, "opt_")
-
-# Sum up all lists, in the given class and superclasses,
-# which have a given name. This gives us "different class
-# semantics": attributes do not override, but rather append
-flags = []
-reflect.accumulateClassList(self.__class__, 'optFlags', flags)
-
-# Add lock-acquire and lock-release to all methods which
-# are not multi-thread safe
-for methodName in klass.synchronized:
-    hook.addPre(klass, methodName, _synchPre)
-    hook.addPost(klass, methodName, _synchPost)
-
-
- -
Listing 1: Using twisted.python.reflect and - twisted.python.hook
-
- -

The twisted.python subpackage also contains a - high-level interface to getopt which supplies as much power as - plain getopt while avoiding long - if/elif chains and making many common - cases easier to use. It uses the reflection interfaces in - twisted.python.reflect to find which options the - class is interested in, and constructs the argument to - getopt. Since in the common case options' values - are just saved in instance attributes, it is very easy to - indicate interest in such options. However, for the cases - custom code needs to be run for an option (for example, - counting how many -v options were given to - indicate verbosity level), it will call a method which is named - correctly.

- -
-
-class ServerOptions(usage.Options):
-    # Those are (short and long) options which
-    # have no argument. The corresponding attribute
-    # will be true iff this option was given
-    optFlags = [['nodaemon','n'],
-                ['profile','p'],
-                ['threaded','t'],
-                ['quiet','q'],
-                ['no_save','o']]
-    # This are options which require an argument
-    # The default is used if no such option was given
-    # Note: since options can only have string arguments,
-    # putting a non-string here is a reliable way to detect
-    # whether the option was given
-    optStrings = [['logfile','l',None],
-                  ['file','f','twistd.tap'],
-                  ['python','y',''],
-                  ['pidfile','','twistd.pid'],
-                  ['rundir','d','.']]
-
-    # For methods which can be called multiple times
-    # or have other unusual semantics, a method will be called
-    # Twisted assumes that the option needs an argument if and only if
-    # the method is defined to accept an argument.
-    def opt_plugin(self, pkgname):
-        pkg = __import__(pkgname)
-        self.python = os.path.join(os.path.dirname(
-                         os.path.abspath(pkg.__file__)), 'config.tac')
-
-    # Most long options based on methods are aliased to short
-    # options. If there is only one letter, Twisted knows it is a short
-    # option, so it is "-g", not "--g"
-    opt_g = opt_plugin
-
-try:
-    config = ServerOptions()
-    config.parseOptions()
-except usage.error, ue:
-    print "%s: %s" % (sys.argv[0], ue)
-    sys.exit(1)
-
- -
Listing 2: twistd's Usage Code
-
- -

Unlike getopt, Twisted has a useful abstraction - for the non-option arguments: they are passed as arguments to - the parsedArgs method. This means too many - arguments, or too few, will cause a usage error, which will be - flagged. If an unknown number of arguments is desired, - explicitly using a tuple catch-all argument will work.

- -

Configuration

- -

The formats of configuration files have shown two visible - trends over the years. On the one hand, more and more - programmability has been added, until sometimes they become a - new language. The extreme end of this trend is using a regular - programming language, such as Python, as the configuration - language. On the other hand, some configuration files became - more and more machine editable, until they become a miniature - database formates. The extreme end of that trend is using a - generic database tool.

- -

Both trends stem from the same rationale -- the need to use - a powerful general purpose tool instead of hacking domain - specific languages. Domain specific languages are usually - ad-hoc and not well designed, having neither the power of - general purpose languages nor the predictable machine editable - format of generic databases.

- -

Twisted combines these two trends. It can read the - configuration either from a Python file, or from a pickled - file. To some degree, it integrates the approaches by - auto-pickling state on shutdown, so the configuration files can - migrate from Python into pickles. Currently, there is no way to - go back from pickles to equivalent Python source, although it - is planned for the future. As a proof of concept, the RPG - framework Twisted Reality already has facilities for creating - Python source which evaluates into a given Python object.

- -
-
-from twisted.internet import main
-from twisted.web import proxy, server
-site = server.Site(proxy.ReverseProxyResource('www.yahoo.com', 80, '/'))
-application = main.Application('web-proxy')
-application.listenOn(8080, site)
-
- -
Listing 3: The configuration file for a reverse web - proxy
-
- -

Twisted's main program, twistd, can receive - either a pickled twisted.internet.main.Application - or a Python file which defines a variable called - application. The application can be saved at any - time by calling its save method, which can take an - optional argument to save to a different file name. It would be - fairly easy, for example, to have a Twisted server which saves - the application every few seconds to a file whose name depends - on the time. Usually, however, one settles for the default - behavior which saves to a shutdown file. Then, if - the shutdown configuration proves suitable, the regular pickle - is replaced by the shutdown file. Hence, on the fly - configuration changes, regardless of complexity, can always - persist.

- -

There are several client/server protocols which let a - suitably privileged user to access to application variable and - change it on the fly. The first, and least common denominator, - is telnet. The administrator can telnet into twisted, and issue - Python statements to her heart's content. For example, one can - add ports to listen on to the application, reconfigure the web - servers and various other ways by simple accessing - __main__.application. Some proof of concepts for a - simple suite of command-line utilities to control a Twisted - application were written, including commands which allow an - administrator to shut down the server or save the current state - to a tap file. These are especially useful on Microsoft - Windows(tm) platforms, where the normal UNIX way of - communicating shutdown requests via signals are less - reliable.

- -

If reconfiguration on the fly is not necessary, Python - itself can be used as the configuration editor. Loading the - application is as simple as unpickling it, and saving it is - done by calling its save method. It is quite easy - to add more services or change existing ones from the Python - interactive mode.

- -

A more sophisticated way to reconfigure the application on - the fly is via the manhole service. Manhole is a client/server - protocol based on top of Perspective Broker, Twisted's - translucent remote-object protocol which will be covered later. - Manhole has a graphical client called gtkmanhole - which can access the server and change its state. Since Twisted - is modular, it is possible to write more services for user - friendly configuration. For example, through-the-web - configuration is planned for several services, notably - mail.

- -

For cases where a third party wants to distribute both the - code for a server and a ready to run configuration file, there - is the plugin configuration. Philosophically similar to the - --python option to twistd, it - simplifies the distribution process. A plugin is an archive - which is ready to be unpacked into the Python module path. In - order to keep a clean tree, twistd extends the - module path with some Twisted-specific paths, like the - directory TwistedPlugins in the user's home - directory. When a plugin is unpacked, it should be a Python - package which includes, alongside __init__.py a - file named config.tac. This file should define a - variable named application, in a similar way to - files loaded with --python. The plugin way of - distributing configurations is meant to reduce the temptation - to put large amount of codes inside the configuration file - itself.

- -

Putting class and function definition inside the - configuration files would make the persistent servers which are - auto-generated on shutdown useless, since they would not have - access to the classes and functions defined inside the - configuration file. Thus, the plugin method is intended so - classes and functions can still be in regular, importable, - Python modules, but still allow third parties distribute - powerful configurations. Plugins are used by some of the - Twisted Reality virtual worlds.

- -

Ports, Protocol and Protocol Factories

- -

Port is the Twisted class which represents a - socket listening on a port. Currently, twisted supports both - internet and unix-domain sockets, and there are SSL classes - with identical interface. A Port is only - responsible for handling the transfer layer. It calls - accept on the socket, checks that it actually - wants to deal with the connection and asks its factory for a - protocol. The factory is usually a subclass of - twisted.protocols.protocol.Factory, and its most - important method is buildProtocol. This should - return something that adheres to the protocol interface, and is - usually a subclass of - twisted.protocols.protocol.Protocol.

- -
-
-from twisted.protocols import protocol
-from twisted.internet import main, tcp
-
-class Echo(protocol.Protocol):
-    def dataReceived(self, data):
-        self.transport.write(data)
-
-factory = protocol.Factory()
-factory.protocol = Echo
-port = tcp.Port(8000, factory)
-app = main.Application("echo")
-app.addPort(port)
-app.run()
-
- -
Listing 4: A Simple Twisted Application
-
- -

The factory is responsible for two tasks: creating new - protocols, and keeping global configuration and state. Since - the factory builds the new protocols, it usually makes sure the - protocols have a reference to it. This allows protocols to - access, and change, the configuration. Keeping state - information in the factory is the primary reason for keeping an - abstraction layer between ports and protocols. Examples of - configuration information is the root directory of a web server - or the user database of a telnet server. Note that it is - possible to use the same factory in two different Ports. This - can be used to run the same server bound to several different - addresses but not to all of them, or to run the same server on - a TCP socket and a UNIX domain sockets.

- -

A protocol begins and ends its life with - connectionMade and connectionLost; - both are called with no arguments. connectionMade - is called when a connection is first established. By then, the - protocol has a transport attribute. The - transport attribute is a Transport - - it supports write and loseConnection. - Both these methods never block: write actually - buffers data which will be written only when the transport is - signalled ready to for writing, and loseConnection - marks the transport for closing as soon as there is no buffered - data. Note that transports do not have a - read method: data arrives when it arrives, and the - protocol must be ready for its dataReceived - method, or its connectionLost method, to be - called. The transport also supports a getPeer - method, which returns parameters about the other side of the - transport. For TCP sockets, this includes the remote IP and - port.

- -
-
-# A tcp port-forwarder
-# A StupidProtocol sends all data it gets to its peer.
-# A StupidProtocolServer connects to the host/port,
-# and initializes the client connection to be its peer
-# and itself to be the client's peer
-from twisted.protocols import protocol
-
-class StupidProtocol(protocol.Protocol):
-    def connectionLost(self): self.peer.loseConnection();del self.peer
-    def dataReceived(self, data): self.peer.write(data)
-
-class StupidProtocolServer(StupidProtocol):
-    def connectionMade(self):
-        clientProtocol = StupidProtocol()
-        clientProtocol.peer = self.transport
-        self.peer = tcp.Client(self.factory.host, self.factory.port, 
-                               clientProtocol)
-
-# Create a factory which creates StupidProtocolServers, and
-# has the configuration information they assume
-def makeStupidFactory(host, port):
-    factory = protocol.Factory()
-    factory.host, factory.port = host, port
-    factory.protocol = StupidProtocolServer
-    return factory
-
- -
Listing 5: TCP forwarder code
-
- -

The Event Loop

- -

While Twisted has the ability to let other event loops take - over for integration with GUI toolkits, it usually uses its own - event loop. The event loop code uses global variables to - maintain interested readers and writers, and uses Python's - select() function, which can accept any object - which has a fileno() method, not only raw file - descriptors. Objects can use the event loop interface to - indicate interest in either reading to or writing from a given - file descriptor. In addition, for those cases where time-based - events are needed (for example, queue flushing or periodic POP3 - downloads), Twisted has a mechanism for repeating events at - known delays. While far from being real-time, this is enough - for most programs' needs.

- -

Going Higher Level

- -

Unfortunately, handling arbitrary data chunks is a hard way - to code a server. This is why twisted has many classes sitting - in submodules of the twisted.protocols package which give - higher level interface to the data. For line oriented - protocols, LineReceiver translates the low-level - dataReceived events into lineReceived - events. However, the first naive implementation of - LineReceiver proved to be too simple. Protocols - like HTTP/1.1 or Freenet have packets which begin with header - lines that include length information, and then byte streams. - LineReceiver was rewritten to have a simple - interface for switching at the protocol layer between - line-oriented parts and byte-stream parts.

- -

Another format which is gathering popularity is Dan J. - Bernstein's netstring format. This format keeps ASCII text as - ASCII, but allows arbitrary bytes (including nulls and - newlines) to be passed freely. However, netstrings were never - designed to be used in event-based protocols where over-reading - is unavoidable. Twisted makes sure no user will have to deal - with the subtle problems handling netstrings in event-driven - programs by providing NetstringReceiver.

- -

For even higher levels, there are the protocol-specific - protocol classes. These translate low-level chunks into - high-level events such as "HTTP request received" (for web - servers), "approve destination address" (for mail servers) or - "get user information" (for finger servers). Many RFCs have - been thus implemented for Twisted (at latest count, more then - 12 RFCs have been implemented). One of Twisted's goals is to be - a repository of event-driven implementations for various - protocols in Python.

- -
-
-class DomainSMTP(SMTP):
-
-    def validateTo(self, helo, destination):
-        try:
-            user, domain = string.split(destination, '@', 1)
-        except ValueError:
-            return 0
-        if domain not in self.factory.domains: 
-            return 0
-        if not self.factory.domains[domain].exists(user, domain, self): 
-            return 0
-        return 1
-
-    def handleMessage(self, helo, origin, recipients, message):
-        # No need to check for existence -- only recipients which
-        # we approved at the validateTo stage are passed here
-        for recipient in recipients:
-            user, domain = string.split(recipient, '@', 1)
-            self.factory.domains[domain].saveMessage(origin, user, message,
-                                                     domain)
-
- -
Listing 6: Implementation of virtual domains using the - SMTP protocol class
-
- -

Copious documentation on writing new protocol abstraction - exists, since this is the largest amount of code written -- - much like most operating system code is device drivers. Since - many different protocols have already been implemented, there - are also plenty of examples to draw on. Usually implementing - the client-side of a protocol is particularly challenging, - since protocol designers tend to assume much more state kept on - the client side of a connection then on the server side.

- -

The twisted.tap Package and - mktap

- -

Since one of Twisted's configuration formats are pickles, - which are tricky to edit by hand, Twisted evolved a framework - for creating such pickles. This framework is contained in the - twisted.tap package and the mktap - script. New servers, or new ways to configure existing servers, - can easily participate in the twisted.tap framework by creating - a twisted.tap submodule.

- -

All twisted.tap submodules must conform to a - rigid interface. The interface defines functions to accept the - command line parameters, and functions to take the processed - command line parameters and add servers to - twisted.main.internet.Application. Existing - twisted.tap submodules use - twisted.python.usage, so the command line format - is consistent between different modules.

- -

The mktap utility gets some generic options, - and then the name of the server to build. It imports a - same-named twisted.tap submodule, and lets it - process the rest of the options and parameters. This makes sure - that the process configuring the main.Application - is agnostic for where it is used. This allowed - mktap to grow the --append option, - which appends to an existing pickle rather then creating a new - one. This option is frequently used to post-add a telnet server - to an application, for net-based on the fly configuration - later.

- -

When running mktap under UNIX, it saves the - user id and group id inside the tap. Then, when feeding this - tap into twistd, it changes to this user/group id - after binding the ports. Such a feature is necessary in any - production-grade server, since ports below 1024 require root - privileges to use on UNIX -- but applications should not run as - root. In case changing to the specified user causes difficulty - in the build environment, it is also possible to give those - arguments to mktap explicitly.

- -
-
-from twisted.internet import tcp, stupidproxy
-from twisted.python import usage
-
-usage_message = """
-usage: mktap stupid [OPTIONS]
-
-Options are as follows:
-        --port <#>, -p:         set the port number to <#>.
-        --host <host>, -h:      set the host to <host>
-        --dest_port <#>, -d:    set the destination port to <#>
-"""
-
-class Options(usage.Options):
-    optStrings = [["port", "p", 6666],
-                  ["host", "h", "localhost"],
-                  ["dest_port", "d", 6665]]
-
-def getPorts(app, config):
-    s = stupidproxy.makeStupidFactory(config.host, int(config.dest_port))
-    return [(int(config.port), s)]
-
- -
Listing 7: twisted.tap.stupid
-
- -

The twisted.tap framework is one of the reasons - servers can be set up with little knowledge and time. Simply - running mktap with arguments can bring up a web - server, a mail server or an integrated chat server -- with - hardly any need for maintainance. As a working - proof-on-concept, the tap2deb utility exists to - wrap up tap files in Debian packages, which include scripts for - running and stopping the server and interact with - init(8) to make sure servers are automatically run - on start-up. Such programs can also be written to interface - with the Red Hat Package Manager or the FreeBSD package - management systems.

- -
-
-% mktap --uid 33 --gid 33 web --static /var/www --port 80
-% tap2deb -t web.tap -m 'Moshe Zadka <moshez@debian.org>'
-% su
-password:
-# dpkg -i .build/twisted-web_1.0_all.deb
-
- -
Listing 8: Bringing up a web server on a Debian - system
-
- -

Multi-thread Support

- -

Sometimes, threads are unavoidable or hard to avoid. Many - legacy programs which use threads want to use Twisted, and some - vendor APIs have no non-blocking version -- for example, most - database systems' API. Twisted can work with threads, although - it supports only one thread in which the main select loop is - running. It can use other threads to simulate non-blocking API - over a blocking API -- it spawns a thread to call the blocking - API, and when it returns, the thread calls a callback in the - main thread. Threads can call callbacks in the main thread - safely by adding those callbacks to a list of pending events. - When the main thread is between select calls, it searches - through the list of pending events, and executes them. This is - used in the twisted.enterprise package to supply - an event driven interfaces to databases, which uses Python's DB - API.

- -

Twisted tries to optimize for the common case -- no threads. - If there is need for threads, a special call must be made to - inform the twisted.python.threadable module that - threads will be used. This module is implemented differently - depending on whether threads will be used or not. The decision - must be made before importing any modules which use threadable, - and so is usually done in the main application. For example, - twistd has a command line option to initialize - threads.

- -

Twisted also supplies a module which supports a threadpool, - so the common task of implementing non-blocking APIs above - blocking APIs will be both easy and efficient. Threads are kept - in a pool, and dispatch requests are done by threads which are - not working. The pool supports a maximum amount of threads, and - will throw exceptions when there are more requests than - allowable threads.

- -

One of the difficulties about multi-threaded systems is - using locks to avoid race conditions. Twisted uses a mechanism - similar to Java's synchronized methods. A class can declare a - list of methods which cannot safely be called at the same time - from two different threads. A function in threadable then uses - twisted.python.hook to transparently add - lock/unlock around these methods. This allows Twisted classes - to be written without thought about threading, except for one - localized declaration which does not entail any performance - penalty for the single-threaded case.

- -

Twisted Mail Server

- -

Mail servers have a history of security flaws. Sendmail is - by now the poster boy of security holes, but no mail servers, - bar maybe qmail, are free of them. Like Dan Bernstein of qmail - fame said, mail cannot be simply turned off -- even the - simplest organization needs a mail server. Since Twisted is - written in a high-level language, many problems which plague - other mail servers, notably buffer overflows, simply do not - exist. Other holes are avoidable with correct design. Twisted - Mail is a project trying to see if it is possible to write a - high quality high performance mail server entirely in - Python.

- -

Twisted Mail is built on the SMTP server and client protocol - classes. While these present a level of abstraction from the - specific SMTP line semantics, they do not contain any message - storage code. The SMTP server class does know how to divide - responsibility between domains. When a message arrives, it - analyzes the recipient's address, tries matching it with one of - the registered domain, and then passes validation of the - address and saving the message to the correct domain, or - refuses to handle the message if it cannot handle the domain. - It is possible to specify a catch-all domain, which will - usually be responsible for relaying mails outwards.

- -

While correct relaying is planned for the future, at the - moment we have only so-called "smarthost" relaying. All e-mail - not recognized by a local domain is relayed to a single outside - upstream server, which is supposed to relay the mail further. - This is the configuration for most home machines, which are - Twisted Mail's current target audience.

- -

Since the people involved in Twisted's development were - reluctant to run code that runs as a super user, or with any - special privileges, it had to be considered how delivery of - mail to users is possible. The solution decided upon was to - have Twisted deliver to its own directory, which should have - very strict permissions, and have users pull the mail using - some remote mail access protocol like POP3. This means only a - user would write to his own mail box, so no security holes in - Twisted would be able to adversely affect a user.

- -

Future plans are to use a Perspective Broker-based service - to hand mail to users to a personal server using a UNIX domain - socket, as well as to add some more conventional delivery - methods, as scary as they may be.

- -

Because the default configuration of Twisted Mail is to be - an integrated POP3/SMTP servers, it is ideally suited for the - so-called POP toaster configuration, where there are a - multitude of virtual users and domains, all using the same IP - address and computer to send and receive mails. It is fairly - easy to configure Twisted as a POP toaster. There are a number - of deployment choices: one can append a telnet server to the - tap for remote configuration, or simple scripts can add and - remove users from the user database. The user database is saved - as a directory, where file names are keys and file contents are - values, so concurrency is not usually a problem.

- -
-
-% mktap mail -d foobar.com=$HOME/Maildir/ -u postmaster=secret -b \
-             -p 110 -s 25
-% twistd -f mail.tap
-
-
- -
Bringing up a simple mail-server
-
- -

Twisted's native mail storage format is Maildir, a format - that requires no locking and is safe and atomic. Twisted - supports a number of standardized extensions to Maildir, - commonly known as Maildir++. Most importantly, it supports - deletion as simply moving to a subfolder named - Trash, so mail is recoverable if accessed through - a protocol which allows multiple folders, like IMAP. However, - Twisted itself currently does not support any such protocol - yet.

- -

Introducing Perspective Broker

- -

All the World's a Game

- -

Twisted was originally designed to support multi-player - games; a simulated "real world" environment. Experience with - game systems of that type is enlightening as to the nature of - computing on the whole. Almost all services on a computer are - modeled after some simulated real-world activity. For example, - e-"mail", or "document publishing" on the web. Even - "object-oriented" programming is based around the notion that - data structures in a computer simulate some analogous - real-world objects.

- -

All such networked simulations have a few things in common. - They each represent a service provided by software, and there - is usually some object where "global" state is kept. Such a - service must provide an authentication mechanism. Often, there - is a representation of the authenticated user within the - context of the simulation, and there are also objects aside - from the user and the simulation itself that can be - accessed.

- -

For most existing protocols, Twisted provides these - abstractions through twisted.internet.passport. - This is so named because the most important common - functionality it provides is authentication. A simulation - "world" as described above -- such as an e-mail system, - document publishing archive, or online video game -- is - represented by subclass of Service, the - authentication mechanism by an Authorizer (which - is a set of Identities), and the user of the - simulation by a Perspective. Other objects in the - simulation may be represented by arbitrary python objects, - depending upon the implementation of the given protocol.

- -

New problem domains, however, often require new protocols, - and re-implementing these abstractions each time can be - tedious, especially when it's not necessary. Many efforts have - been made in recent years to create generic "remote object" or - "remote procedure call" protocols, but in developing Twisted, - these protocols were found to require too much overhead in - development, be too inefficient at runtime, or both.

- -

Perspective Broker is a new remote-object protocol designed - to be lightweight and impose minimal constraints upon the - development process and use Python's dynamic nature to good - effect, but still relatively efficient in terms of bandwidth - and CPU utilization. twisted.spread.pb serves as a - reference implementation of the protocol, but implementation of - Perspective Broker in other languages is already underway. - spread is the twisted subpackage - dealing with remote calls and objects, and has nothing to do - with the spread toolkit.

- -

Perspective Broker extends - twisted.internet.passport's abstractions to be - concrete objects rather than design patterns. Rather than - having a Protocol implementation translate between - sequences of bytes and specifically named methods (as in the - other Twisted Protocols), Perspective Broker - defines a direct mapping between network messages and - quasi-arbitrary method calls.

- -

Translucent, not Transparent

- -

In a server application where a large number of clients may - be interacting at once, it is not feasible to have an - arbitrarily large number of OS threads blocking and waiting for - remote method calls to return. Additionally, the ability for - any client to call any method of an object would present a - significant security risk. Therefore, rather than attempting to - provide a transparent interface to remote objects, - twisted.spread.pb is "translucent", meaning that - while remote method calls have different semantics than local - ones, the similarities in semantics are mirrored by - similarities in the syntax. Remote method calls impose as - little overhead as possible in terms of volume of code, but "as - little as possible" is unfortunately not "nothing".

- -

twisted.spread.pb defines a method naming - standard for each type of remotely accessible object. For - example, if a client requests a method call with an expression - such as myPerspective.doThisAction(), the remote - version of myPerspective would be sent the message - perspective_doThisAction. Depending on the manner - in which an object is accessed, other method prefixes may be - observe_, view_, or - remote_. Any method present on a remotely - accessible object, and named appropriately, is considered to be - published -- since this is accomplished with - getattr, the definition of "present" is not just - limited to methods defined on the class, but instances may have - arbitrary callable objects associated with them as long as the - name is correct -- similarly to normal python objects.

- -

Remote method calls are made on remote reference objects - (instances of pb.RemoteReference) by calling a - method with an appropriate name. However, that call will not - block -- if you need the result from a remote method call, you - pass in one of the two special keyword arguments to that method - -- pbcallback or pberrback. - pbcallback is a callable object which will be - called when the result is available, and pberrback - is a callable object which will be called if there was an - exception thrown either in transmission of the call or on the - remote side.

- -

In the case that neither pberrback or - pbcallback is provided, - twisted.spread.pb will optimize network usage by - not sending confirmations of messages.

- -
-
-# Server Side
-class MyObject(pb.Referenceable):
-    def remote_doIt(self):
-        return "did it"
-
-# Client Side
-    ...
-    def myCallback(result):
-        print result # result will be 'did it'
-    def myErrback(stacktrace):
-        print 'oh no, mr. bill!'
-        print stacktrace
-    myRemoteReference.doIt(pbcallback=myCallback,
-                           pberrback=myErrback)
-
- -
Listing 9: A remotely accessible object and accompanying - call
-
- -

Different Behavior for Different Perspectives

- -

Considering the problem of remote object access in terms of - a simulation demonstrates a requirement for the knowledge of an - actor with certain actions or requests. Often, when processing - message, it is useful to know who sent it, since different - results may be required depending on the permissions or state - of the caller.

- -

A simple example is a game where certain an object is - invisible, but players with the "Heightened Perception" - enchantment can see it. When answering the question "What - objects are here?" it is important for the room to know who is - asking, to determine which objects they can see. Parallels to - the differences between "administrators" and "users" on an - average multi-user system are obvious.

- -

Perspective Broker is named for the fact that it does not - broker only objects, but views of objects. As a user of the - twisted.spread.pb module, it is quite easy to - determine the caller of a method. All you have to do is - subclass Viewable.

- -
-
-# Server Side
-class Greeter(pb.Viewable):
-    def view_greet(self, actor):
-        return "Hello %s!\n" % actor.perspectiveName
-
-# Client Side
-    ...
-    remoteGreeter.greet(pbcallback=sys.stdout.write)
-    ...
-
- -
Listing 10: An object responding to its calling - perspective
-
- Before any arguments sent by the client, the actor - (specifically, the Perspective instance through which this - object was retrieved) will be passed as the first argument to - any view_xxx methods. - -

Mechanisms for Sharing State

- -

In a simulation of any decent complexity, client and server - will wish to share structured data. Perspective Broker provides - a mechanism for both transferring (copying) and sharing - (caching) that state.

- -

Whenever an object is passed as an argument to or returned - from a remote method call, that object is serialized using - twisted.spread.jelly; a serializer similar in some - ways to Python's native pickle. Originally, - pickle itself was going to be used, but there were - several security issues with the pickle code as it - stands. It is on these issues of security that - pickle and twisted.spread.jelly part - ways.

- -

While twisted.spread.jelly handles a few basic - types such as strings, lists, dictionaries and numbers - automatically, all user-defined types must be registered both - for serialization and unserialization. This registration - process is necessary on the sending side in order to determine - if a particular object is shared, and whether it is shared as - state or behavior. On the receiving end, it's necessary to - prevent arbitrary code from being run when an object is - unserialized -- a significant security hole in - pickle for networked applications.

- -

On the sending side, the registration is accomplished by - making the object you want to serialize a subclass of one of - the "flavors" of object that are handled by Perspective Broker. - A class may be Referenceable, - Viewable, Copyable or - Cacheable. These four classes correspond to - different ways that the object will be seen remotely. - Serialization flavors are mutually exclusive -- these 4 classes - may not be mixed in with each other.

- - - -

Publishing Objects with PB

- -

The previous samples of code have shown how an individual - object will interact over a previously-established PB - connection. In order to get to that connection, you need to do - some set-up work on both the client and server side; PB - attempts to minimize this effort.

- -

There are two different approaches for setting up a PB - server, depending on your application's needs. In the simplest - case, where your application does not deal with the - abstractions above -- services, identities, and perspectives -- - you can simply publish an object on a particular port.

- -
-
-from twisted.spread import pb
-from twisted.internet import main
-class Echoer(pb.Root):
-    def remote_echo(self, st):
-        print 'echoing:', st
-        return st
-if __name__ == '__main__':
-    app = main.Application("pbsimple")
-    app.listenOn(8789, pb.BrokerFactory(Echoer()))
-    app.run()
-
- -
Listing 11: Creating a simple PB server
-
- -

Listing 11 shows how to publish a simple object which - responds to a single message, "echo", and returns whatever - argument is sent to it. There is very little to explain: the - "Echoer" class is a pb.Root, which is a small subclass of - Referenceable designed to be used for objects published by a - BrokerFactory, so Echoer follows the same rule for remote - access that Referenceable does. Connecting to this service is - almost equally simple.

- -
-
-from twisted.spread import pb
-from twisted.internet import main
-def gotObject(object):
-    print "got object:",object
-    object.echo("hello network", pbcallback=gotEcho)
-def gotEcho(echo):
-    print 'server echoed:',echo
-    main.shutDown()
-def gotNoObject(reason):
-    print "no object:",reason
-    main.shutDown()
-pb.getObjectAt("localhost", 8789, gotObject, gotNoObject, 30)
-main.run()
-
- -
Listing 12: A client for Echoer objects.
-
- -

The utility function pb.getObjectAt retrieves - the root object from a hostname/port-number pair and makes a - callback (in this case, gotObject) if it can - connect and retrieve the object reference successfully, and an - error callback (gotNoObject) if it cannot connect - or the connection times out.

- -

gotObject receives the remote reference, and - sends the echo message to it. This call is - visually noticeable as a remote method invocation by the - distinctive pbcallback keyword argument. When the - result from that call is received, gotEcho will be - called, notifying us that in fact, the server echoed our input - ("hello network").

- -

While this setup might be useful for certain simple types of - applications where there is no notion of a "user", the - additional complexity necessary for authentication and service - segregation is worth it. In particular, re-use of server code - for things like chat (twisted.words) is a lot easier with a - unified notion of users and authentication.

- -
-
-from twisted.spread import pb
-from twisted.internet import main
-class SimplePerspective(pb.Perspective):
-    def perspective_echo(self, text):
-        print 'echoing',text
-        return text
-class SimpleService(pb.Service):
-    def getPerspectiveNamed(self, name):
-        return SimplePerspective(name, self)
-if __name__ == '__main__':
-    import pbecho
-    app = main.Application("pbecho")
-    pbecho.SimpleService("pbecho",app).getPerspectiveNamed("guest")\
-        .makeIdentity("guest")
-    app.listenOn(pb.portno, pb.BrokerFactory(pb.AuthRoot(app)))
-    app.save("start")
-
- -
Listing 13: A PB server using twisted's "passport" - authentication.
-
- -

In terms of the "functionality" it offers, this server is - identical. It provides a method which will echo some simple - object sent to it. However, this server provides it in a manner - which will allow it to cooperate with multiple other - authenticated services running on the same connection, because - it uses the central Authorizer for the application.

- -

On the line that creates the SimpleService, - several things happen.

- -
    -
  1. A SimpleService is created and persistently added to the - Application instance.
    2. - -
  2. A SimplePerspective is created, via the overridden - getPerspectiveNamed method.
    3. - -
  3. That SimplePerspective has an - Identity generated for it, and persistently - added to the Application's - Authorizer. The created identity will have the - same name as the perspective ("guest"), and the password - supplied (also, "guest"). It will also have a reference to - the service "pbecho" and a perspective named "guest", by - name. The Perspective.makeIdentity utility - method prevents having to deal with the intricacies of the - passport Authorizer system when one doesn't - require strongly separate Identitys and - Perspectives.
    4. -
-
-
- - -

Also, this server does not run itself, but instead persists - to a file which can be run with twistd, offering all the usual - amenities of daemonization, logging, etc. Once the server is - run, connecting to it is similar to the previous example.

- -
-
-from twisted.spread import pb
-from twisted.internet import main
-def success(message):
-    print "Message received:",message
-    main.shutDown()
-def failure(error):
-    print "Failure...",error
-    main.shutDown()
-def connected(perspective):
-    perspective.echo("hello world",
-                     pbcallback=success,
-                     pberrback=failure)
-    print "connected."
-pb.connect(connected, failure,   "localhost", pb.portno,
-           "guest", "guest",     "pbecho", "guest", 30)
-main.run()
-
- -
Listing 14: Connecting to an Authorized Service
-
-
-
- - -

This introduces a new utility -- pb.connect. - This function takes a long list of arguments and manages the - handshaking and challenge/response aspects of connecting to a - PB service perspective, eventually calling back to indicate - either success or failure. In this particular example, we are - connecting to localhost on the default PB port (8787), - authenticating to the identity "guest" with the password - "guest", requesting the perspective "guest" from the service - "pbecho". If this can't be done within 30 seconds, the - connection will abort.

- -

In these examples, I've attempted to show how Twisted makes - event-based scripting easier; this facilitates the ability to - run short scripts as part of a long-running process. However, - event-based programming is not natural to procedural scripts; - it is more generally accepted that GUI programs will be - event-driven whereas scripts will be blocking. An alternative - client to our SimpleService using GTK illustrates - the seamless meshing of Twisted and GTK.

- -
-
-from twisted.internet import main, ingtkernet
-from twisted.spread.ui import gtkutil
-import gtk
-ingtkernet.install()
-class EchoClient:
-    def __init__(self, echoer):
-        l.hide()
-        self.echoer = echoer
-        w = gtk.GtkWindow(gtk.WINDOW_TOPLEVEL)
-        vb = gtk.GtkVBox(); b = gtk.GtkButton("Echo:")
-        self.entry = gtk.GtkEntry(); self.outry = gtk.GtkEntry()
-        w.add(vb)
-        map(vb.add, [b, self.entry, self.outry])
-        b.connect('clicked', self.clicked)
-        w.connect('destroy', gtk.mainquit)
-        w.show_all()
-    def clicked(self, b):
-        txt = self.entry.get_text()
-        self.entry.set_text("")
-        self.echoer.echo(txt, pbcallback=self.outry.set_text)
-l = gtkutil.Login(EchoClient, None, initialService="pbecho")
-l.show_all()
-gtk.mainloop()
-
- -
Listing 15: A Twisted GUI application
-
- -

Event-Driven Web Object Publishing with Web.Widgets

- -

Although PB will be interesting to those people who wish to - write custom clients for their networked applications, many - prefer or require a web-based front end. Twisted's built-in web - server has been designed to accommodate this desire, and the - presentation framework that one would use to write such an - application is twisted.web.widgets. Web.Widgets - has been designed to work in an event-based manner, without - adding overhead to the designer or the developer's - work-flow.

- -

Surprisingly, asynchronous web interfaces fit very well into - the normal uses of purpose-built web toolkits such as PHP. Any - experienced PHP, Zope, or WebWare developer will tell you that - separation of presentation, content, and logic is very - important. In practice, this results in a "header" block of - code which sets up various functions which are called - throughout the page, some of which load blocks of content to - display. While PHP does not enforce this, it is certainly - idiomatic. Zope enforces it to a limited degree, although it - still allows control structures and other programmatic elements - in the body of the content.

- -

In Web.Widgets, strict enforcement of this principle - coincides very neatly with a "hands-free" event-based - integration, where much of the work of declaring callbacks is - implicit. A "Presentation" has a very simple structure for - evaluating Python expressions and giving them a context to - operate in. The "header" block which is common to many - templating systems becomes a class, which represents an - enumeration of events that the template may generate, each of - which may be responded to either immediately or latently.

- -

For the sake of simplicity, as well as maintaining - compatibility for potential document formats other than HTML, - Presentation widgets do not attempt to parse their template as - HTML tags. The structure of the template is "HTML Text - %%%%python_expression()%%%% more HTML Text". Every set - of 4 percent signs (%%%%) switches back and forth between - evaluation and printing.

- -

No control structures are allowed in the template. This was - originally thought to be a potentially major inconvenience, but - with use of the Web.Widgets code to develop a few small sites, - it has seemed trivial to encapsulate any table-formatting code - within a method; especially since those methods can take string - arguments if there's a need to customize the table's - appearance.

- -

The namespace for evaluating the template expressions is - obtained by scanning the class hierarchy for attributes, and - getting each of those attributes from the current instance. - This means that all methods will be bound methods, so - indicating "self" explicitly is not required. While it is - possible to override the method for creating namespaces, using - this default has the effect of associating all presentation - code for a particular widget in one class, along with its - template. If one is working with a non-programmer designer, and - the template is in an external file, it is always very clear to - the designer what functionality is available to them in any - given scope, because there is a list of available methods for - any given class.

- -

A convenient event to register for would be a response from - the PB service that we just implemented. We can use the - Deferred class in order to indicate to the widgets - framework that certain work has to be done later. This is a - Twisted convention which one can currently use in PB as well as - webwidgets; any framework which needs the ability to defer a - return value until later should use this facility. Elements of - the page will be rendered from top to bottom as data becomes - available, so the page will not be blocked on rendering until - all deferred elements have been completed.

- -
-
-from twisted.spread import pb
-from twisted.python import defer
-from twisted.web import widgets
-class EchoDisplay(widgets.Presentation):
-    template = """<H1>Welcome to my widget, displaying %%%%echotext%%%%.</h1>
-    <p>Here it is: %%%%getEchoPerspective()%%%%</p>"""
-    echotext = 'hello web!'
-    def getEchoPerspective(self):
-        d = defer.Deferred()
-        pb.connect(d.callback, d.errback, "localhost", pb.portno,
-                   "guest", "guest",      "pbecho", "guest", 1)
-        d.addCallbacks(self.makeListOf, self.formatTraceback)
-        return ['<b>',d,'</b>']
-    def makeListOf(self, echoer):
-        d = defer.Deferred()
-        echoer.echo(self.echotext, pbcallback=d.callback, pberrback=d.errback)
-        d.addCallbacks(widgets.listify, self.formatTraceback)
-        return [d]
-if __name__ == "__main__":
-    from twisted.web import server
-    from twisted.internet import main
-    a = main.Application("pbweb")
-    gdgt = widgets.Gadget()
-    gdgt.widgets['index'] = EchoDisplay()
-    a.listenOn(8080, server.Site(gdgt))
-    a.run()
-
- -
Listing 16: an event-based web widget.
-
- -

Each time a Deferred is returned as part of the page, the - page will pause rendering until the deferred's - callback method is invoked. When that callback is - made, it is inserted at the point in the page where rendering - left off.

- -

If necessary, there are options within web.widgets to allow - a widget to postpone or cease rendering of the entire page -- - for example, it is possible to write a FileDownload widget, - which will override the rendering of the entire page and - replace it with a file download.

- -

The final goal of web.widgets is to provide a framework - which encourages the development of usable library code. Too - much web-based code is thrown away due to its particular - environment requirements or stylistic preconceptions it carries - with it. The goal is to combine the fast-and-loose iterative - development cycle of PHP with the ease of installation and use - of Zope's "Product" plugins.

- -

Things That Twisted Does Not Do

- -

It is unfortunately well beyond the scope of this paper to - cover all the functionality that Twisted provides, but it - serves as a good overview. It may seem as though twisted does - anything and everything, but there are certain features we - never plan to implement because they are simply outside the - scope of the project.

- -

Despite the multiple ways to publish and access objects, - Twisted does not have or support an interface definition - language. Some developers on the Twisted project have - experience with remote object interfaces that require explicit - specification of all datatypes during the design of an object's - interface. We feel that such interfaces are in the spirit of - statically-typed languages, and are therefore suited to the - domain of problems where statically-typed languages excel. - Twisted has no plans to implement a protocol schema or static - type-checking mechanism, as the efficiency gained by such an - approach would be quickly lost again by requiring the type - conversion between Python's dynamic types and the protocol's - static ones. Since one of the key advantages of Python is its - extremely flexible dynamic type system, we felt that a - dynamically typed approach to protocol design would share some - of those advantages.

- -

Twisted does not assume that all data is stored in a - relational database, or even an efficient object database. - Currently, Twisted's configuration state is all stored in - memory at run-time, and the persistent parts of it are pickled - at one go. There are no plans to move the configuration objects - into a "real" database, as we feel it is easier to keep a naive - form of persistence for the default case and let - application-specific persistence mechanisms handle persistence. - Consequently, there is no object-relational mapping in Twisted; - twisted.enterprise is an interface to the - relational paradigm, not an object-oriented layer over it.

- -

There are other things that Twisted will not do as well, but - these have been frequently discussed as possibilities for it. - The general rule of thumb is that if something will increase - the required installation overhead, then Twisted will probably - not do it. Optional additions that enhance integration with - external systems are always welcome: for example, database - drivers for Twisted or a CORBA IDL for PB objects.

- -

Future Directions

- -

Twisted is still a work in progress. The number of protocols - in the world is infinite for all practical purposes, and it - would be nice to have a central repository of event-based - protocol implementations. Better integration with frameworks - and operating systems is also a goal. Examples for integration - opportunities are automatic creation of installer for "tap" - files (for Red Hat Packager-based distributions, FreeBSD's - package management system or Microsoft Windows(tm) installers), - and integration with other event-dispatch mechanisms, such as - win32's native message dispatch.

- -

A still-nascent feature of Twisted, which this paper only - touches briefly upon, is twisted.enterprise: it is - planned that Twisted will have first-class database support - some time in the near future. In particular, integration - between twisted.web and twisted.enterprise to allow developers - to have SQL conveniences that they are used to from other - frameworks.

- -

Another direction that we hope Twisted will progress in is - standardization and porting of PB as a messaging protocol. Some - progress has already been made in that direction, with XEmacs - integration nearly ready for release as of this writing.

- -

Tighter integration of protocols is also a future goal, such - an FTP server that can serve the same resources as a web - server, or a web server that allows users to change their POP3 - password. While Twisted is already a very tightly integrated - framework, there is always room for more integration. Of - course, all this should be done in a flexible way, so the - end-user will choose which components to use -- and have those - components work well together.

- -

Conclusions

- -

As shown, Twisted provides a lot of functionality to the - Python network programmer, while trying to be in his way as - little as possible. Twisted gives good tools for both someone - trying to implement a new protocol, or someone trying to use an - existing protocol. Twisted allows developers to prototype and - develop object communication models with PB, without designing - a byte-level protocol. Twisted tries to have an easy way to - record useful deployment options, via the - twisted.tap and plugin mechanisms, while making it - easy to generate new forms of deployment. And last but not - least, even Twisted is written in a high-level language and - uses its dynamic facilities to give an easy API, it has - performance which is good enough for most situations -- for - example, the web server can easily saturate a T1 line serving - dynamic requests on low-end machines.

- -

While still an active project, Twisted can already used for - production programs. Twisted can be downloaded from the main - Twisted site (http://www.twistedmatrix.com) where there is also - documentation for using and programming Twisted.

- -

Acknowledgements

- -

We wish to thank Sean Riley, Allen Short, Chris Armstrong, - Paul Swartz, Jürgen Hermann, Benjamin Bruheim, Travis B. - Hartwell, and Itamar Shtull-Trauring for being a part of the - Twisted development team with us.

- -

Thanks also to Jason Asbahr, Tommi Virtanen, Gavin Cooper, - Erno Kuusela, Nick Moffit, Jeremy Fincher, Jerry Hebert, Keith - Zaback, Matthew Walker, and Dan Moniz, for providing insight, - commentary, bandwidth, crazy ideas, and bug-fixes (in no - particular order) to the Twisted team.

- -

References

- -
    -
  1. The Twisted site, http://www.twistedmatrix.com
    2. - -
  2. Douglas Schmidt, Michael Stal, Hans Rohnert and Frank - Buschmann, Pattern-Oriented Software Architecture, Volume 2, - Patterns for Concurrent and Networked Objects, John Wiley - & Sons
    3. - -
  3. Abhishek Chandra, David Mosberger, Scalability of Linux - Event-Dispatch Mechanisms, USENIX 2001, - http://lass.cs.umass.edu/~abhishek/papers/usenix01/paper.ps
    4. - -
  4. Protocol specifications, http://www.rfc-editor.com
    5. - -
  5. The Twisted Philosophical FAQ, - http://www.twistedmatrix.com/page.epy/twistedphil.html
    6. - -
  6. Twisted Advocacy, - http://www.twistedmatrix.com/page.epy/whytwisted.html
    7. - -
  7. Medusa, http://www.nightmare.com/medusa/index.html
    8. - -
  8. Using Spreadable Web Servers, - http://www.twistedmatrix.com/users/jh.twistd/python/moin.cgi/TwistedWeb
    9. - -
  9. Twisted Spread implementations for other languages, - http://www.twistedmatrix.com/users/washort/
    10. - -
  10. PHP: Hypertext Preprocessor, http://www.php.net/
    11. - -
  11. The Z Object Publishing Environment, - http://www.zope.org/, http://zope.com/
    12. -
- - - diff --git a/docs/historic/2003/europython/doanddont.html b/docs/historic/2003/europython/doanddont.html deleted file mode 100644 index d4adcaca291..00000000000 --- a/docs/historic/2003/europython/doanddont.html +++ /dev/null @@ -1,508 +0,0 @@ -Idioms and Anti-Idioms in Python - -

Idioms and Anti-Idioms in Python

- -

Idioms and Anti-Idioms, AKA Do and Don't

-
-Prue (Something Wicca This Way Comes, season 1) -- No, we are not supposed to use our powers -

Python

-
-Prue (Something Wicca This Way Comes, season 1) -- Uh, it doesn't work out there either. -

Exceptions

-
-Leo (Paige From the Past, season 4) -- You have to [...] Paige. No exceptions. -

Exceptions -- Catching Too Much

-
-Piper (Charmed Again, season 4) -- Okay, well this is way too much for me to handle. -

Exceptions -- Catching Too Much -- Example

-
-try:
-   f = opne("file")
-except BaseException:
-   sys.exit("no such file")
-
- -
-Piper (Charmed Again, season 4) -- Way too much. -

Exceptions -- Catching Too Soon

-
-Phoebe (Knight to Remember, season 4) -- Maybe it's just too soon. -

Exceptions -- Catching Too Soon -- Example

-
-def readlinesfromfile(file):
-    try:
-        return open(file).readlines()
-    except IOError:
-        pass # do what?
-
- -
-Paige (Knight to Remember, season 4) -- I'm already a little late. -

Catching multiple exceptions

- -
-try:
-   fp = open("file")
-except IOError, OSError:
-   print "could not open file"
-
-
-Paige (Knight to Remember, season 4) -- Because I've got too many responsibilities -

Catching multiple exceptions (cont'd)

-
-Piper (Knight to Remember, season 4) -- Alright! Calm down! -

Catching multiple exceptions (cont'd 2)

- -
-try:
-   fp = open("file")
-except (IOError, OSError):
-   print "could not open file"
-
-
-Phoebe (Knight to Remember, season 4) -- Besides that, maybe we can help -

Catching NameError

- -
-try:
-   import foo
-except ImportError:
-   pass
-
-try:
-   foo.Function()
-except NameError:
-   pass # some replacement
-
- -
-Piper (Morality Bites, season 4) -- That's OK, I forgot your name too. -

Catching NameError (cont'd)

- -
-try:
-   import foo
-except ImportError:
-   foo = None
-
-if foo is not None:
-   foo.Function()
-else:
-   pass # some replacement
-
- -
-Anne (Morality Bites, season 4) -- Oh, right, sorry. -

Importing Modules -- A Review

-
-Phoebe (Animal Pragmatism, season 2) -- Rome was not built in a day, -

Importing __main__

-
-Piper (Animal Pragmatism, season 2) -- And why mess with a good thing? -

Importing a File Twice

- -
-# file: hello.py
-import hello
-class Foo: pass
-
- -
-Phoebe (Which Prue Is It, Anyway?, season 1) -- Okay, which one of you is the real Prue? -

Importing *

- -
-from os import *
-
-fp = open("file") # works
-fp.readline() # fails with a weird error...?
-
- -
-Pink Prue (Which Prue Is It, Anyway?, season 1) -- So, um, what did I do now? -

Importing * Inside Functions

-
-Pink Prue (Which Prue Is It, Anyway?, season 1) -- What ever it is, I have an alibi. -

Importing Names

-
-Real Prue (Which Prue Is It, Anyway?, season 1) -- Because I still have to work here when all of this is over. -

Reloading

-
-Real Prue (Which Prue Is It, Anyway?, season 1) -- Don't worry I'm never casting that spell again. -

exec, execfile and eval

-
-Reporter (Morality Bites, season 2) -- More news on the execution of Phoebe Halliwell coming up. -

exec, execfile and eval -- Modify namespaces

-
-Nathaniel (Morality Bites, season 2) -- Executions are a bitch to plan. -

exec, execfile and eval -- Inside functions

-
-Nathaniel (Morality Bites, season 2) -- Phoebe, what is this? An attempt to stay your execution? -

Conclusion: recommended usage

-
-Phoebe (Morality Bites, season 2) -- Just because you don't understand something, doesn't make it evil. -

exec, execfile and eval -- Restricted Execution (Don't)

-
-Leo (Morality Bites, season 2) -- Nobody's gonna rescue you. -

Syntax

-
-Prue (Morality Bites, season 2) -- You know, we can still make the good things happen. -

Syntax -- Tabs and Spaces

-
-Prue (The Painted World, season 2) -- We've seen so many bizarre things. -

Syntax -- Backslash Continuations

- -
-# Extra newline
-r = 1 \
-
-+2
-
-# Missing backslash in long series
-r = 1 \
-+2 \
-+3 \
-+4 
-+5 \
-+6
-
- - -

Syntax -- Backslash Continuations (cont'd)

- - - -
-# Extra newline
-r = (1
-
-+2)
-
-# Long series
-r = (1
-+2
-+3
-+4
-+5
-+6)
-
- -
-Prue (The Painted World, season 2) -- Uh, what just happened here? -

Hand Hacking Batteries

-
-Prue (Animal Pragmatism, season 2) -- Well, we didn't find anything in the Book Of Shadows. -

Further Reading

-
-Phoebe (The Painted World, season 2) -- I think you'll find me pretty knowledgeable about all areas -

Questions?

-Piper (The Painted World, season 2) -- You're like ask rainman.com - -

Bonus Slides

-Phoebe (The Painted World, season 2) -- Oh, and P.S. there will be no personal gain. - -

Packages and __init__.py

-
-Piper (Animal Pragmatism, season 2) -- It's a package. One I would like to share with you. -

Type Checking

-
-Phoebe (Black as Cole, season 2) -- I never thought of myself as the marrying type -

Type Checking -- Example

- -
-class Foo:
-    def __init__(self, i):
-        if type(i) is types.StringType:
-            self.content = open(i).readlines()
-        elif type(i) is types.ListType:
-            self.content = i
-
- -
-Phoebe (Muse to My Ears, season 4) -- You're an artistic, creative type. -

Type Checking -- Example -- Fixed

-
-
-class Foo:
-    pass
-
-class FooFromFile(Foo):
-
-    def __init__(self, filename):
-         self.content = open(filename).readlines()
-
-class FooFromList(Foo):
-
-    def __init__(self, list):
-         self.content = list
-
- -
-Phoebe (Muse to My Ears, season 4) -- You see how well this worked out? -

Private __Attributes

-
-Tessa (Animal Pragmatism, season 2) -- Maybe it's our fault because we tried to make them into something they're not. - -

Using Mutable Default Arguments

- -
-def foo(l=[]):
-    l.append(5);return l
-
- - - -
-def foo(l=None):
-    if l is None: l=[]
-    l.append(5);return l
-
-
-Snake guy (Animal Pragmatism, season 2) -- -You two are acting like nothing's changed. - - diff --git a/docs/historic/2003/europython/index.html b/docs/historic/2003/europython/index.html deleted file mode 100644 index 051fb6d5efc..00000000000 --- a/docs/historic/2003/europython/index.html +++ /dev/null @@ -1,35 +0,0 @@ -Moshe's Talks - -

Moshe's talks

- -

Slides

- - - -

HTML

- - - -

PDF

- - - - diff --git a/docs/historic/2003/europython/lore.html b/docs/historic/2003/europython/lore.html deleted file mode 100644 index edb33cc0550..00000000000 --- a/docs/historic/2003/europython/lore.html +++ /dev/null @@ -1,502 +0,0 @@ -Lore - -

Lore

-

Lore - A Document Generation System

-
-Rory (Concert Interruptus, season 1) -- Yeah, well I've always thought easy is completely overrated. -

Source Format

-
-Alex (I Solemnly Swear, season 3) -- That would've been far too logical. - -

Output Formats

-
-Madelaine (The Lorelais' First Day at Chilton, season 1) -- You don't know she's going out for the paper. -

Minimal Lore Document

-
-<html>
-<head><title>Title</title></head>
-<body><h1>Title</h1></body>
-</html>
- -
-Luke (There's the Rub, season 2) -- You said minimal -

Minimal Lore Document Explained

-
-Tom (There's the Rub, season 2) -- Hey, this is minimal -

External Listings

-
-Kirk (Red Light on the Wedding Night, season 2) -- I include it as an example of the excellence I aspire to. -

Using Lore to Generate HTML

-
-Paris (Run Away, Little Boy, season 2) -- I went on the web and found this site -

Generating LaTeX

-
-Rory (Christopher Returns, season 1) -- He had already printed like a million -

Using Lint

-
-Max (The Deer-Hunters, season 1) -- I know a D seems pretty dismal -

Further Reading

-
-Paris (The Bracebridge Dinner, season 2) -- Rereading the Iliad a third time is not not doing anything -

Questions?

-Lorelai (Forgiveness and Stuff, season 1) -- A person needs details. -

Bonus Slides

-Miss James (The Lorelais' First Day at Chilton, season 1) -- If you do it in Latin you get extra credit. -

Lore Alternatives - LaTeX

-
-Michel (Love, Daisies and Troubadors, season 1) -- It increases my ennui -

Lore Alternatives - HTML

-
-Lorelai (Love, Daisies and Troubadors, season 1) -- It was broken [...] I'm not crazy -

Lore Alternatives - Docbook

-
-Rory (Hammers and Veils, season 2) -- What do you want me to do it? -

Lore Alternatives - Texinfo

-
-Man (Hammers and Veils, season 2) -- There's a ton of hurt that almost happened here. -

Lore Alternatives - reST

-
-Emily (Hammers and Veils, season 2) -- And this is what we need to discuss right now? -

Lore Alternatives - LyX

-
-Rory (Hammers and Veils, season 2) -- Well, it's just dressed up a little. -

Some Standard Tags -- XHTML Primer

-
-Rory (Kiss and Tell, season 1) -- See, even a little information in your hands is dangerous. -

More HTML

-
-Max (The Lorelais' First Day at Chilton, season 1) -- Tolstoy's favourite author, for instance, was... -

More HTML -- cross references

-
-Christopher (Christopher Returns, season 1) -- It's just a weird reference. -

Special Markup

-
-Taylor (Take The Deviled Eggs, season 3) -- Out attention spans are gnat-like tonight -

API References

-
-Lorelai (The Road Trip To Harvard, season 2) -- We're just kinda hanging out between classes -

API References Explained

-
-Luke (Love and War and Snow, season 1) -- How do you know? Do you have written documentation? -

Inline Listings

-
-<pre class="python">
-def foo():
-    return forbnicate(4)
-</pre>
-
-Taylor (Take The Deviled Eggs, season 3) -- That's not even English. -

Inline Listings -- short

-
-Rory (Double Date, season 1) -- It's like this weird code thing with her. -

Generating HTML -- writing templates

-
-Paris (I Can't Get Started, season 2) -- How's this sound for a template? -

Generating HTML -- using commandline

-
-Richard (The Third Lorelai, season 1) -- Your wish is my command. -

Generating HTML -- using commandline -- examples

-
-Jackson (A Deep-Fried Korean Thanksgiving, season 3) -- Deep-fried cake! -

Generating HTML -- using commandline -- examples (cont'd)

-
-Jackson (A Deep-Fried Korean Thanksgiving, season 3) -- Deep-fried shoe! -

Generating HTML -- notes about stylesheets

-
-Miss Patty (Cinnamon's Wake, season 1) -- If you had a better hair style I might consider dating -

Generating LaTeX -- examples

-
-Luke (Hammers and Veils, season 2) -- Just an example -

Using Lint -- notes

-
-Paris (The Deer-Hunters, season 1) -- That would be cause for concern. -

Understanding Lint Warnings

-
-Jess (Teach Me Tonight, season 2) -- I appreciate the warning. -

Using Lore For Slides

-
-Emily (Road Trip to Harvard, season 2) -- Why in the world do you insist on taking slides? -

Extending Lore

-
-Rory (The Lorelais' First Day at Chilton, season 1) -- Well, add a couple of plaid skirts -

Extending Lore -- example

-
-Lorelai (Presenting Lorelai Gilmore, season 2) -- No, no, if you wanna do it, I'll help. It's just weird. -

Extending Lore -- example (cont'd)

-
-# blinker/html.py
-from twisted.lore import tree
-from twisted.web import microdom, domhelpers
-
-def doBlink(document):
-    for node in domhelpers.findElementsWithAttribute(document, 'class',
-                                                    'blink'):
-        newNode = microdom.Element('blink')
-        newNode.children = node.children
-        node.parentNode.replaceChild(newNode, node)
-
-def doFile(fn, docsdir, ext, url, templ, linkrel=''):
-    doc = tree.parseFileAndReport(fn)
-    doBlink(doc)
-    cn = templ.cloneNode(1)
-    tree.munge(doc, cn, linkrel, docsdir, fn, ext, url)
-    cn.writexml(open(os.path.splitext(fn)[0]+ext, 'wb'))
-
- -
-Christopher (Presenting Lorelai Gilmore, season 2) -- I can't believe you're letting her do it. -

Extending Lore -- example (cont'd 2)

-
-# blinker/latex.py
-class BlinkerLatexSpitter(latex.LatexSpitter):
-
-    def visitNode_span_blink(self, node):
-        self.writer('{\sc ')
-        self.visitNodeDefault(node)
-        self.writer('}')
-
- -
-Lorelai (Presenting Lorelai Gilmore, season 2) -- I'm sorry, I meant what scenario on my planet -

Extending Lore -- example (cont'd 3)

-
-# blinker/factory.py
-from blinker import html, latex
-from twisted.lore import default
-
-class ProcessingFunctionFactory(default.ProcessingFunctionFactory):
-
-    doFile = [doFile]
-
-    latexSpitters = {None: latex.BlinkLatexSpitter}
-
-    def getLintChecker(self):
-        checker = lint.getDefaultChecker()
-        checker.allowedClasses = checker.allowedClasses.copy()
-        oldSpan = checker.allowedClasses['span']
-        checker.allowedClasses['span'] = (lambda x:oldSpan(x) or
-                                                     x=='blink')
-        return checker
-
-factory = ProcessingFunctionFactory()
-
-
-# blinker/plugins.tml
-register("Blink-Lore",
-         "blinker.factory",
-         description="Lore format with blink",
-         type="lore",
-         tapname="blinklore")
-
-

...and that's it!

- -
-Rory (Presenting Lorelai Gilmore, season 2) -- Sorry, we haven't tamed my wild ways yet. -

Man page support

-
-Lorelai (Concert Interruptus, season 1) -- would you like to write out some sort of instruction manual to go with the dishes? -

Man page support

-
-Lorelai (Concert Interruptus, season 1) -- would you like to write out some sort of instruction manual to go with the dishes? - - diff --git a/docs/historic/2003/europython/slides-template.tpl b/docs/historic/2003/europython/slides-template.tpl deleted file mode 100644 index fd33fc3d18f..00000000000 --- a/docs/historic/2003/europython/slides-template.tpl +++ /dev/null @@ -1,19 +0,0 @@ - - - - - - - - - - - [ | - ] -

-
- -
- - diff --git a/docs/historic/2003/europython/tw-deploy.html b/docs/historic/2003/europython/tw-deploy.html deleted file mode 100644 index 2b4d7ec1f76..00000000000 --- a/docs/historic/2003/europython/tw-deploy.html +++ /dev/null @@ -1,1106 +0,0 @@ -A Twisted Web Tutorial - -

A Twisted Web Tutorial

- -

Twisted Web -- The Tutorial

-
-Sweet (Once More With Feeling, season 6) -- Showtime -

Twisted Web

-
-Giles (I Robot -- You Jane, season 1) -- There's a demon in the internet -

Short Example: Putting a Server Up

- -
-% mktap --uid=33 --gid=33 web --path=/var/www/htdocs --port=80
-% sudo twistd -f web.tap
-
- -
-Buffy (Once More, With Feeling, season 6) -- I've got a theory. It doesn't matter. -

Setup and Configuration Utilities

-
-Xander (The Harvest, season 1) -- crosses, garlic, stake through the heart -

Digression: What are TAPs

-
-Master (The Wish, season 3) -- Behold the technical wonder -

mktap

-
-Buffy (Bad Eggs, season 2) -- I'm gonna need a *big* weapon -

mktap web: Common Useful Options

-
-Buffy (Bad Eggs, season 2) -- That's probably not gonna be the winning argument, is it? - -

twistd

-
-Giles (Teacher's Pet, season 1) -- That's all he said? Fork Guy? -

What's a Resource?

-
-Sean (Go Fish, season 3) -- You're soakin' in it, bud. -

Resource Examples

-
-Xander (As You Were, season 6) -- We have friends, family and demons -

Web Development

-
-Xander (Family, season 5) -- That was a tangled web -

Custom Processor

- -
-from twisted.web import static, twcgi
-
-class PerlScript(twcgi.FilteredScript):
-    filter = '/usr/bin/perl' # Points to the perl parser
-
- -
-Tara (Family, season 4) -- There was the front of a camel -

Resource Scripting

-
-Tara (Once More, With Feeling, season 6) -- You make me complete -

.rpy example

-
-from twisted.web import resource as resourcelib
-
-class MyGreatResource(resourcelib.Resource):
-    def render(self, request):
-        return "<html>foo</html>"
-
-resource = MyGreatResource()
-
-
-Willow (Welcome to the Hellmouth, season 1) -- It's probably easy for you. -

Alternative Configuration Formats

-
-Ben (The Gift, season 5) -- I wish there was another way -

Alternative Configuration Formats -- Python

-
-Buffy (The I In Team, season 4) -- But I've learned that it pays to be flexible in life. - -

Python Configuration Example

- -
-from twisted.internet import app
-from twisted.web import static, server
-
-application = app.Application('web')
-application.listenTCP(80,
-              server.Site(static.File("/var/www/htdocs")))
-
-
-Willow (The Pack, season 1) -- It's simple, really. - -

Bannerfish -- A Case Study in Deployment

-
-Xander (Halloween, season 2) -- Let's move out. -

Bannerfish -- Standalone tap

-
-Ethan (Halloween, season 2) -- Don't wish to blow my own trumpet, but -- -

Bannerfish -- Standalone tap (behind reverse proxy)

- -
-resource = proxy.ReverseProxyResource('localhost', 81, '/')
-
-
-Buffy (Halloween, season 2) -- You're sweet. A terrible liar, but sweet. -

Bannerfish -- Standalone Python

-
-from twisted.internet import app
-from twisted.cred import authorizer
-from twisted.web import server
-from bannerfish import service
-
-application = app.Application("bannerfish")
-auth = authorizer.DefaultAuthorizer(app)
-svc = service.BannerService('/var/bannerfish',
-                            "bannerfish", application, auth)
-site = server.Site(svc.buildResource(None, None))
-application.listenTCP(80, site)
-
-
-Spike (Halloween, season 2) -- Shaking. Terrified. Alone. Lost little lamb. -

Bannerfish -- /etc/twisted-web/local.d Drop In

-
-from twisted.cred import authorizer
-from bannerfish import service
-
-auth = authorizer.DefaultAuthorizer(app)
-svc = service.BannerService('/var/bannerfish',
-                       "bannerfish", application, auth)
-resource = svc.buildResource(None, None)
-default.addChild("bannerfish", resource)
-
-
-Cordelia (Halloween, season 2) -- Well, I guess you better get them back to their parents. -

Bannerfish -- Resource Script

-
-from twisted.cred import authorizer
-from twisted.internet import app
-from bannerfish import service
-
-application = registry.getComponent(app.Application)
-auth = authorizer.DefaultAuthorizer(application)
-svc = service.BannerService('/var/bannerfish',
-                           "bannerfish", application, auth)
-resource = svc.buildResource(None, None)
-
- -
-Xander (Innocence, season 2) -- They like to see the big guns. -

Bannerfish -- Distributed (Slave)

- -
-from twisted.internet import application
-from twisted.cred import authorizer
-from twisted.web import server
-from bannerfish import service
-application = app.Application("bannerfish")
-auth = authorizer.DefaultAuthorizer(application)
-svc = service.BannerService('/var/bannerfish',
-                         "bannerfish", application, auth)
-site = server.Site(svc.buildResource(None, None))
-fact = pb.BrokerFactory(site)
-site = server.Site(root)
-application.listenUNIX('/var/run/bannerfish', fact)
-
-

Bannerfish -- Distributed (Master, Resource Script)

- -
-from twisted.web import distrib
-
-resource = distrib.ResourceSubscription('unix',
-                        '/var/run/bannerfish')
-
-
-Oz (Innocence, season 2) -- So, do you guys steal weapons from the Army a lot? -

Bannerfish -- Other options

-
-Buffy (Tabula Rasa, season 6) -- I'm like a superhero or something -

Bannerfish -- Conclusions

-
-Giles (Killed By Death, season 2) -- Simple enough, but, but -

Further Reading

-
-Giles (I Was Made to Love You, Season 5) -- There's an enormous amount of research we should do before -- no I'm lying -

Questions?

-Vampire Willow (Dopplegangland, season 3): Questions? Comments? -

Bonus Slides

-Xander (The Dark Age, season 2) -- A bonus day of class plus Cordelia. - -

Python Configuration -- Hints

-
-Buffy (Phases, season 2) -- Have you dropped any hints? -

Python Configuration -- Persistence

-
-Spike (Once More, With Feeling) -- Let me rest in peace -

Python Configuration -- Processors

-
-
-from twisted.internet import app
-from twisted.web import static, server
-from twisted.web import twcgi
-
-root = static.File("/var/www")
-root.processors = {".cgi": twcgi.CGIScript}
-application = app.Application('web')
-application.listenTCP(80, server.Site(root))
-
-
-Manny (Doublemeat Palace, season 6) -- It's a meat process -

Python Configuration -- Indices

-
-
-root = static.File("/var/www")
-root.indices = ['index.rpy', 'index.html']
-
-
-Willow (Buffy vs. Dracula, season 5) -- Labelling your amulets and indexing your diaries -

Python Configuration -- Virtual Hosts

-
-
-from twisted.web import vhost
-default = static.File("/var/www")
-foo = static.File("/var/foo")
-root = vhost.NamedVirtualHost(default)
-root.addHost('foo.com', foo)
-
-
-Fritz (I Robot, You Jane, season 1) -- The only reality is virtual. -

Python Configuration -- uber example

-
-
-from twisted.internet import app
-from twisted.web import static, server, vhost, script
-
-default = static.File("/var/www")
-default.processors = {".rpy", script.ResourceScript}
-root = vhost.NamedVirtualHost(default
-foo = static.File("/var/foo")
-foo.indices = ['index.xhtml', 'index.html']
-root.addHost('foo.com', foo)
-site = server.Site(root)
-application = app.Application('web')
-application.listenTCP(80, site, interface='127.0.0.1')
-
-
-Buffy (Potential, season 7) -- It was putting a lot of stock in that uber-vamp -

Python Configuration -- Splitting With Reverse Proxy

- -
-from twisted.web import proxy
-
-root.putChild('foo',
-     proxy.ReverseProxyResource('localhost',
-                                81, '/foo/'))
-
-
-Buffy (Once More, With Feeling, season 6 -- So I will walk through the fire -

mktap examples

-
-Anya (I Was Made to Love You, season 4) -- You can also see the website I designed for the magic shop -

mktap examples (cont'd)

-
-Buffy (Once More, With Feeling, season 6) -- All the twists and bends -

mktap examples (alternate formats)

-
-Tara (Seeing Red, season 6) -- It isn't written in any ancient language we could identify. -

mktap examples (setting uid)

-
-Buffy (Who Are You?, season 4) -- I would be Buffy - -

twistd examples

-
-Xander (Teacher's Pet, season 1) -- How come *that* never came up? -

Shutting down twistd

-
-Buffy (Prophecy Girl, season 1) -- I don't wanna die. -

Shutdown TAPs

-
-Headstone (The Gift, season 5) -- She saved the world. A lot. -

twistd and security

-
-Buffy (Dopplegangland, season 3) -- I think it's good to be reliable - -

Resource Call Examples

- -
-site.getChild('foo', request
-   ).getChild('bar', request
-   ).getChild('baz', request
-   ).render(request)
-
-
-Willow/Tara (Afterlife, Part 2, season 6) -- Child of words, hear thy makers -

Resource Call Examples (cont'd)

- -
-site.getChild('foo', request
-   ).getChild('bar', request
-   ).getChild('baz', request
-   ).getChild('', request
-   ).render(request)
-
-
-Buffy (Gone, season 6) -- Stop trying to see me. -

Distributed Servers -- Theory

-
-Anya (Once More, With Feeling, season 6) -- I've got a theory, it could be bunnies -

Distributed Servers -- Manually

-
-from twisted.internet import app, protocol
-from twisted.web import server, distrib, static
-from twisted.spread import pb
-
-application = app.Application("silly-web")
-# The "master" server
-site = server.Site(distrib.ResourceSubscription('unix', '.rp'))
-application.listenTCP(19988, site)
-# The "slave" server
-fact = pb.BrokerFactory(distrib.ResourcePublisher(
-           server.Site(static.File('static'))))
-application.listenUNIX('./.rp', fact)
-
-
-Buffy (Some Assembly Required, season 2) -- Men dig up the corpses and the women have the babies. -

Distributed Servers -- Manual (cont'd)

- -
-from twisted.internet import app, protocol
-from twisted.web import server, distrib, static, vhost
-from twisted.spread import pb
-
-application = app.Application("ping-web")
-
-default = static.File("/var/www/foo")
-root = vhost.NamedVirtualHost(default)
-root.addVhost("foo.com", default)
-bar = distrib.ResourceSubscription('unix', '.bar')
-root.addVhost("bar.com", bar)
-
-fact = pb.BrokerFactory(static.Site(default))
-site = server.Site(root)
-application.listenTCP(19988, site)
-application.listenUNIX('./.foo', fact)
-
-
-Buffy (Welcome to the Hellmouth, season 1) -- Now, we can do this the hard way, or... -

Distributed Servers -- Manual (cont'd 2)

- -
-from twisted.internet import app, protocol
-from twisted.web import server, distrib, static, vhost
-from twisted.spread import pb
-
-application = app.Application("pong-web")
-
-foo = distrib.ResourceSubscription('unix', '.foo')
-root = vhost.NamedVirtualHost(foo)
-root.addVhost("foo.com", foo)
-bar = static.File("/var/www/bar")
-root.addVhost("bar.com", bar)
-
-fact = pb.BrokerFactory(static.Site(bar))
-site = server.Site(root)
-application.listenTCP(19989, site)
-application.listenUNIX('./.bar', fact)
-
-
-Buffy (Welcome to the Hellmouth, season 1) -- ...well, actually there's just the hard way. -

Distributed Servers -- User Directory

-
-Master (The Wish, season 3) -- Mass production! -

Distributed Servers -- User Directory Server

- -
-from twisted.internet import app
-from twisted.web import static, server, distrib
-
-root = static.File("/var/www")
-root.putChild("users", distrib.UserDirectory())
-site = server.Site(root)
-application = app.Application('web')
-application.listenTCP(80, site)
-
-
-Richard (Reptile Boy, season 2) -- In his name. -

Distributed Servers -- Personal Servers

- -
-from twisted.internet import app
-from twisted.web import static, server, distrib
-from twisted.spread import pb
-
-root = static.File("/home/moshez/twistd")
-site = server.Site(root)
-
-fact = pb.BrokerFactory(distrib.ResourcePublisher(site))
-application.listenUNIX('/home/moshez/.twisted-web-pb', fact)
-
-
-Giles (Bargaining, season 6) -- It's my personal collection -

Debian Configuration

-
-Buffy (Bad Girls, season 3) -- We can help each other. -

Debian Configuration -- Usage

-
-Faith (Home Coming, season 3) -- we'll use 'em -

Debian Configuration -- Drop In Examples

-
-from twisted.web import static
-import os
-
-vhostDir = '/var/www/vhost/'
-
-for file in os.listdir(vhostDir):
-    root.addHost(file, static.File(os.path.join(vhostDir, file)))
-
-
-Buffy (The Freshman, season 4) -- I just thought I'd drop in -

Debian Configuration -- Drop In Examples (cont'd)

-
-from twisted.web import script, static
-
-default.processors['.rpy'] = script.ResourceScript
-default.ignoreExt('rpy')
-
-
-Riley (As You Were, season 6) -- Sorry to just drop in on you -

Debian Configuration -- Drop In Examples (cont'd 2)

-
-from twisted.web import vhost
-
-default.putChild('vhost', vhost.VHostMonsterResource())
-
-
-Sam (As You Were, season 6) -- a hairy night drop into hostile territory -

twistedmatrix.com Configuration

- -
-...
-indexNames = ['index', 'index.html', 'index.xhtml', 'index.rpy','index.cgi']
-...
-root.putChild('mailman', twcgi.CGIDirectory('/usr/lib/cgi-bin'))
-root.putChild('users', distrib.UserDirectory())
-root.putChild('cgi-bin', twcgi.CGIDirectory('/usr/lib/cgi-bin'))
-root.putChild('doc', static.File('/usr/share/doc'))
-...
-uid = pwd.getpwnam('www-data')[2]
-gid = grp.getgrnam('www-data')[2]
-...
-top = rewrite.RewriterResource(root, rewrite.tildeToUsers)
-...
-application = app.Application("web", uid=uid, gid=gid)
-
-
-Xander (The Witch, season 1) -- May all lesser cretins bow before me. -

Apache vs. Twisted Web

-
-Willow (Buffy vs. Dracular, season 5) -- I think we've just put our finger on why we're the sidekicks -

Apache/Twisted Web Integration

-
-Xander (What's My Line, season 2) -- Angel's our friend! Except I don't like him. -

Zope vs. Twisted Web

-
-Willow (Dopplegangland, season 3) -- Competition is natural and healthy -

Zope/Twisted Web Integration

-
-Snyder (Dopplegangland, season 3) -- It's a perfect match. -

Zope/Twisted Web Integration (cont'd)

-
-Wesley (Dopplegangland, season 3) -- Still a little sloppy, though -

Applications Appropriate for Twisted Web

-
-Sweet (Once More, With Feeling) -- Why don't you come and play? -

Behind Reverse Proxy

-
-Jenny (I Robot -- You Jane, season 1) -- The divine exists in cyberspace -

Rewrite Rules

-
-Spike (What's My Line, season 2) -- Read it again. -

Rewrite Rules -- Example

-
-
-root = static.File("/var/www")
-root.putChild("users", distrib.UserDirectory())
-root = rewrite.RewriterResource(root, rewrite.tildeToUsers)
-
- -
-Spike (What's My Line, season 2) -- I think it's just enough kill. -

websetroot

-
-Manny (DoubleMeat Palace, season 6) -- We have a lot of turnover here -

Sample websetroot command lines

-
-Manny (DoubleMeat Palace, season 6) -- You can toss it -

init.d

-
-Bob (Zeppo, season 3) -- He hasn't been initiated. -

Special Bonus - How to Configure <user>.example.com

-
-import pwd, os
-from twisted.web import resource, error, distrib
-from twisted.protocols import http
-
-class UserNameVirtualHost(resource.Resource):
-
-    def __init__(self, default, tail):
-        resource.Resource.__init__(self)
-        self.default = default
-        self.tail = tail
-        self.users = {}
-
-    def _getResourceForRequest(self, request):
-        host=request.getHeader('host')
-        if host.endswith(tail):
-            username = host[:-len(tail)]
-        else:
-            username = default
-        if username in self.users:
-            return self.users[username]
-        try:
-            (pw_name, pw_passwd, pw_uid, pw_gid, pw_gecos, pw_dir,
-             pw_shell) = pwd.getpwnam(username)
-        except KeyError:
-            return error.ErrorPage(http.NOT_FOUND,
-                                   "No Such User",
-                          "The user %s was not found on this system." %
-                                   repr(username))
-        twistdsock = os.path.join(pw_dir, ".twistd-web-pb")
-        rs = distrib.ResourceSubscription('unix',twistdsock)
-        self.users[username] = rs
-        return rs
-
-    def render(self, request):
-        resrc = self._getResourceForRequest(request)
-        return resrc.render(request)
-
-    def getChild(self, path, request):
-        resrc = self._getResourceForRequest(request)
-        request.path=request.path[:-1]
-        request.postpath=request.uri.split('/')[1:]
-        print request, request.path, request.postpath
-        return resrc.getChildForRequest(request)
-
-
-Morgan (The Puppet Show, season 1) -- Weird? What d'you mean? -

Special Bonus - How to Configure <user>.example.com (cont'd)

- -
-from twisted.internet import app
-from twisted.web import server
-import uservhost
-
-root = UserNameVirtualHost("www", "example.com")
-site = server.Site(root)
-application = app.Application('web')
-application.listenTCP(80, site)
-
-
-Snyder (The Puppet Show, season 1) -- You need to integrate into this school, people. -

Using the Twisted Registry

-
-Angel (Helpless, season 3) -- I wanted to keep it safe -

Using the Twisted Registry -- example

-
-
-from twisted.web import distrib
-
-resource = registry.getComponent(distrib.UserDirectory)
-if not resource:
-    resource = distrib.UserDirectory()
-    registry.setComponent(distrib.UserDirectory, resource)
-
-
-Paul (The Freshman, season 4) -- Do you know where they're distributing the [...] applications? -

Using the Twisted Registry -- problems

-
-Anya (Once More, With Feeling, season 6) -- The only trouble is [pause] I'll never tell. -

Alternative Configuration Formats -- XML

-
-Buffy (Once More, With Feeling, season 6) -- To fit in in this glittering world. -

Alternative Configuration Formats -- XML -- example

-
-<?xml version="1.0"?>
-
-<instance class="twisted.internet.app.Application" reference="1">
-  <dictionary>
-...
-    <string role="key" value="tcpPorts" />
-    <list>
-      <tuple>
-        <int value="80" />
-        <instance class="twisted.web.server.Site">
-          <dictionary>
-...
-            <string role="key" value="resource" />
-            <instance class="twisted.web.static.File">
-              <dictionary>
-...
-                <string role="key" value="path" />
-                <string value="/var/www" />
-...
-              </dictionary>
-            </instance>
-...
-          </dictionary>
-        </instance>
-...
-      </tuple>
-    </list>
-...
-  </dictionary>
-</instance>
-
-
-Natalie (Teacher's Pet, season 1) -- There's nothing ugly about these creatures -

Alternative Configuration Formats -- Source

-
-Willow/Giles/Xander (Primeval, season 4) -- You could never hope to grasp the source -

Alternative Configuration Formats -- Source -- Example

-
-app=Ref(1,
-  Instance('twisted.internet.app.Application',{
-...
-      'tcpPorts':[
-        (
-          80,
-          Instance('twisted.web.server.Site',
-...
-            resource=Instance('twisted.web.static.File',{
-...
-                'path':'/var/www',
-...
-          ),
-        ],
-...
-      }))
-
-
-Tara (Family, season 5) -- You learn her source, and, uh we'll introduce her to her insect reflection -

Twisted Web - Beginnings

-
-
-<glyphAtWork> the http server was so we could say "Web!" if we ever did
-              a freshmeat announcement
-<glyphAtWork> this makes people excited
-
- -
-Dawn (Get It Done, season 7) -- I think it's an origin myth. -

Woven Overview

-
-Razor (Bargaining, season 6) -- A pretty toy - diff --git a/docs/historic/2003/europython/twisted.html b/docs/historic/2003/europython/twisted.html deleted file mode 100644 index 2576bca4073..00000000000 --- a/docs/historic/2003/europython/twisted.html +++ /dev/null @@ -1,608 +0,0 @@ -Twisted Tutorial - - -

Twisted Tutorial

- -

Twisted -- The Tutorial

-
-Prue (Something Wicca This Way Comes, season 1) -- Piper, the girl has no vision, no sense of the future. -

Twisted -- Networking For Python

-
-Leo (Bite Me, season 4) -- As far as I know they're apart of a whole different network now. -

Finger

-
-Natalie (Blinded By the Whitelighter) -- I'll assume a demon attacked your finger -

Finger - Protocol code

-
-from twisted.protocols import basic
-
-class FingerClient(basic.LineReceiver):
-
-    # This will be called when the connection is made
-    def connectionMade(self): self.sendLine(self.factory.user)
-
-    # This will be called when the server sends us a line.
-    # IMPORTANT: line *without "\n" at end.
-    # Yes, this means empty line does not mean EOF
-    def lineReceived(self, line): print line
-
-    # This will be called when the connection is terminated
-    def connectionLost(self, _): print "-"*40
-
-
-Phoebe (Blind Sided, season 1) -- Standard dating protocol. -

Finger - client factory

- -
-from twisted.internet import protocol
-
-class FingerFactory(protocol.ClientFactory):
-    protocol = FingerProtocol
-
-    def __init__(self, user): self.user = user
-
-    def clientConnectionFailed(self, _, reason):
-        print "error", reason.value
-
- -
-Jack (Ms. Hellfire, season 2) -- Well, they'd better be a rich client -

Finger - tying it all together

- -
-from twisted.internet import reactor
-import sys
-
-user, host = sys.argv[1].split('@')
-port = 79
-reactor.connectTCP(host, port, FingerFactory(port))
-reactor.run()
-
-
-Prue/Phoebe/Piper (Something Wicca This Way Comes, season 1) -- The power of three will set us free -

Finger - a bug

-
-Leo (Trial By Magic, season 4) -- Demons you can handle but not rats? -

Digression - Deferreds

-
-Piper (Morality Bites, season 2) -- Talk about it later. -

Finger - reimplementing correctly

-
-from twisted.protocols import basic
-from twisted.internet import protocol, defer
-import sys
-
-class FingerClient(basic.LineReceiver):
-
-    def connectionMade(self):
-        self.transport.write(self.factory.user+"\n")
-
-    def lineReceived(self, line):
-        self.factory.gotLine(line)
-
- - -

Finger - reimplementing correctly (cont'd)

-
-class FingerFactory(protocol.ClientFactory):
-    protocol = FingerProtocol
-
-    def __init__(self, user):
-        self.user, self.d = user, defer.Deferred()
-
-    def gotLine(self, line): print line
-
-    def clientConnectionLost(self, _, why): self.d.callback(None)
-
-    def clientConnectionFailed(self, _, why): self.d.errback(why)
-
- -

Finger - reimplementing correctly (cont'd 2)

-
-if __name__ == '__main__':
-    from twisted.internet import reactor
-    from twisted.python import util
-    user, host = sys.argv[1].split('@')
-    f = FingerFactory(user)
-    port = 79
-    reactor.connectTCP(host, port, FingerFactory(port))
-    f.d.addCallback(lambda _: reactor.stop())
-    f.d.addErrback(lambda _: (util.println("could not connect"),
-                              reactor.stop()))
-    reactor.run()
-
-
-Phoebe (Charmed and Dangerous, season 4) -- That's what we were missing. -

Servers

-
-Genie (Be Careful What You Witch For, season 2) -- All I know is that you rubbed and now I serve. -

Finger - protocol

-
-class FingerServer(basic.LineReceiver):
-
-    def lineReceived(self, line):
-        self.transport.write(self.factory.getUser(line))
-        self.transport.loseConnection()
-
-
-Secretary (The Painted World, season 2) -- Well, you won't have any trouble with this if you figured that out. -

Finger - factory

-
-class FingerServerFactory(protocol.Factory):
-
-    protocol = FingerServer
-
-    def __init__(self):
-        self.users = {}
-        self.message = "No such user\n"
-
-    def getUser(self, name):
-        return self.users.get(name, self.message)
-
-    def setUser(self, user, status):
-        self.users[user] = status
-
-
-Prue (The Demon Who Came In From the Cole, season 3) -- Okay, so who are they? -

Finger - glue

-
-factory = FingerServerFactory()
-factory.setUser("moshez", "Online - Sitting at computer\n")
-factory.setUser("spiv", "Offline - Surfing the waves\n")
-
-reactor.listenTCP(79, factory)
-
-
-Prue (All Halliwell's Eve, season 3) -- Put it all together, it may just work. -

Finger Server - problem

-
-Piper (All Halliwell's Eve, season 3) -- We've got big problems, a little time and a little magic. -

Finger server -- new protocol

-
-class FingerServer(basic.LineReceiver):
-
-    def lineReceived(self, line):
-        d = self.factory.getUser(line)
-        d.addCallback(self.writeResponse)
-        d.addErrback(self.writeError)
-
-    def writeResponse(self, response):
-        self.transport.write(response)
-        self.transport.loseConnection()
-
-    def writeError(self, error):
-        self.transport.write("Server error -- try later\n")
-        self.transport.loseConnection()
-
-
-Piper (Ex Libris, season 2) -- We'll worry about it later. -

Finger - factory

-
-class FingerServerFactory(protocol.Factory):
-
-    protocol = FingerServer
-
-    def __init__(self):
-        self.users = {}
-        self.message = "No such user\n"
-
-    def getUser(self, name):
-        return defer.succeed(self.users.get(name, self.message))
-
-    def setUser(self, user, status):
-        self.users[user] = status
-
-
-Piper/Zen Master (Enter the Demon, season 4) -- It is a different realm down there with new rules. -

Finger - web factory

-
-from twisted.web import client
-
-class FingerWebFactory(protocol.Factory):
-    protocol = FingerServer
-
-    def getUser(self, name):
-        url = "http://example.com/~%s/online" % name
-        d = client.getPage(url)
-        d.addErrback(lambda _: "No such user\n")
-        return d
-
-
-Applicant #3 (The Painted World, season 2) -- in this day and age, who can't write in the HTML numeric languages, right? -

Application

- -

Application (Example)

-
-# File: finger.tpy
-from twisted.internet import app
-import fingerserver
-
-factory = fingerserver.FingerServerFactory()
-factory.setUser("moshez", "Online - Sitting at computer\n")
-factory.setUser("spiv", "Offline - Surfing the waves\n")
-application = app.Application("finger")
-application.listenTCP(79, factory)
-
- -
-Paige (Hell Hath No Fury, season 4) -- I am taking full responsibility for being late with the application. -

twistd

-
-Phoebe (Sleuthing With the Enemy, season 3) -- Was it some sick twisted demonic thrill? -

twistd examples

-
-Professor Whittlessy (Is There a Woogy In the House?, season 1) -- I use your house as an example -

Writing Plugins

- -

Writing Plugins (Example)

-
-# File finger/tap.py
-from twisted.python import usage
-
-class Options(usage.Options):
-    synopsis = "Usage: mktap finger [options]"
-    optParameters = [["port", "p", 6666,"Set the port number."]]
-    longdesc = 'Finger Server'
-    users = ()
-
-    def opt_user(self, user):
-        if not '=' in user: status = "Online"
-        else: user, status = user.split('=', 1)
-        self.users += ((user, status+"\n"),)
-
- - -

Writing Plugins (Example cont'd)

-
-def updateApplication(app, config):
-    f = FingerFactory()
-    for (user, status) in config.users:
-        f.setUser(user, status)
-    app.listenTCP(int(config.opts['port']), s)
-
-
-Paige (Bite Me, season 4) -- They won't join us willingly. -

Writing Plugins (Example cont'd 2)

-
-# File finger/plugins.tml
-register("Finger",
-         "finger.tap",
-         description="Finger Server",
-         type='tap',
-         tapname="finger")
-
-
-Queen (Bite Me, season 4) -- That's what families are for. -

Using mktap

-
-Piper (Charmed and Dangerous, season 4) -- We'll use potions instead. -

Delayed execution

-
-Cole (Enter the Demon, season 4) -- I know, but not right now. -

callLater(0,) -- An idiom

- -
-def calculateFact(cur, acc=1, d=None):
-    d = d or defer.Deferred()
-    if cur<=1: d.callback(acc)
-    else: reactor.callLater(0, calculateFact, acc*cur, cur-1, d)
-
-calculateFact(10
-).addCallback(lambda n: (util.println(n), reactor.stop()))
-reactor.run()
-
-
-Piper (Lost and Bound, season 4) -- Someone, I won't say who, has the insane notion -

UNIX Domain Sockets

-
-Kate (Once Upon a Time, season 3) -- Fairies don't talk the same way people do. -

SSL Servers

- -
-from OpenSSL import SSL
-
-class ServerContextFactory:
-
-    def getContext(self):
-        ctx = SSL.Context(SSL.SSLv23_METHOD)
-        ctx.use_certificate_file('server.pem')
-        ctx.use_privatekey_file('server.pem')
-        return ctx
-
-reactor.listenSSL(111, FingerWebFactory(), ServerContextFactory())
-
- -

SSL Clients

- - -
-Natalie (Blinded By the Whitelighter, season 3) -- I mean, in private if you wouldn't mind -

Running Processes

- -
-class Advertizer(protocol.ProcessProtocol):
-    def outReceived(self, data): print "out", `data`
-
-    def errReceived(self, data): print "error", `data`
-
-    def processEnded(self, reason): print "ended", reason
-
-reactor.spawnProcess(Advertizer(),
-                     "echo", ["echo", "hello", "world"])
-
-
-Prue (Coyote Piper, season 3) -- You have to know that you can talk to me -

Further Reading

-
-Phoebe (Animal Pragmatism, season 2) -- Ooh, the girls in school are reading this. -

Questions?

-Piper (Something Wicca This Way Comes, season 1) -- Tell me that's not our old spirit board? -

Bonus Slides

-Prue (Sleuthing With the Enemy, season 3) -- All right, you start talking or we start the bonus round. -

Perspective Broker

-
-Paige (Charmed Again, season 4) -- I guess I just kind of feel - connected somehow. -

PB Remote Control Finger (Server)

-
-from twisted.spread import pb
-
-class FingerSetter(pb.Root):
-
-    def __init__(self, ff): self.ff = ff
-
-    def remote_setUser(self, name, status):
-        self.ff.setUser(name, status+"\n")
-
-ff = FingerServerFactory()
-setter = FingerSetter(ff)
-reactor.listenUNIX("/var/run/finger.control",
-                  pb.BrokerFactory(setter))
-
-
-Piper (Be Careful What You Witch For, season 2) -- Okay, you think you can control the power this time? -

PB Remote Control Finger (Client)

-
-from twisted.spread import pb
-from twisted.internet import reactor
-import sys
-
-def failed(reason):
-    print "failed:", reason.value;reactor.stop()
-
-pb.getObjectAt("unix", "/var/run/finger.control", 30
-).addCallback(lambda o: o.callRemote("setUser", *sys.argv[1:3],
-).addCallbacks(lambda _: reactor.stop(), failed)
-
-reactor.run()
-
-
-Leo (Be Careful What You Witch For, season 2) -- How about you just keep your arms down until you learn how to work the controls. -

Perspective Broker (Trick)

-
-Piper (Lost and Bound, season 4) -- They're not good or bad by themselves, it's how we use them -

Perspective Broker (Authentication)

-
-Piper (She's a Man, Baby, a Man!, season 2) -- Okey-Dokey. I get the point. - -

Perspective Broker - About Large Data Streams

- - - -
-Piper (Womb Raider, season 4) -- -Oral tradition tales of a giant whose body served as a portal to other -dimensions. - -

Producers and Consumers

-
-Phoebe (Black as Cole, season 4) -- Apparently he feeds on the remains of other demons' victims. -

Threads (callInThread)

-
-Piper (The Painted World, season 2) -- There will be consequences. There always are. -

Threads (callFromThread)

-
-Phoebe (Witch Trial, season 2) -- Maybe it's still in the house. Just on different plane. - -

Using ApplicationService

-
-Phoebe (Marry Go Round, season 4) -- Yeah, that's just in case you need psychic services. - -

Playing With Persistence

-
-Cole (Marry Go Round, season 4) -- That Lazerus demon is a time bomb waiting to explode - diff --git a/docs/historic/2003/europython/webclients.html b/docs/historic/2003/europython/webclients.html deleted file mode 100644 index 8a26f71e59a..00000000000 --- a/docs/historic/2003/europython/webclients.html +++ /dev/null @@ -1,482 +0,0 @@ -Writing Web Clients - -

Writing Web Clients

- -

Web Clients -- The Tutorial

-
-Anya (Family, season 5) -- Thank you for coming. We value your patronage. -

What Are Web Clients?

-
-Giles (Family, season 5) -- Could we please be a little less effusive, Anya? -

What Are Web Clients Useful For?

-
-Harmony (Family, season 5) -- Aww. You're my little lamb. -

Review of Modules

-
-Buffy (Family, season 5) -- Your definition of narrow is impressively wide. -

Modules -- htmllib

-
-Xander (Family, season 5) -- The answer is somewhere here. -

Modules -- htmllib -- idiomatic usage

-
-# For lists
-import htmllib, formatter
-
-h = htmllib.HTMLParser(formatter.NullFormatter())
-h.feed(htmlString)
-print h.anchorlist
-
- -
-Xander (Family, season 5) -- I'm helping, I'm reading, I'm quiet. -

Modules -- htmllib -- idiotmatic usage (cont'd)

-
-import htmllib, formatter
-
-class IMGFinder(htmllib.HTMLParser):
-
-    def __init__(self, *args, **kw):
-        htmllib.HTMLParser.__init__(self, *args, **kw)
-        self.ims = []
-
-    def handle_image(self, src, *args): self.ims.append(src)
-
-h = IMGFinder(formatter.NullFormatter())
-h.feed(htmlString)
-print h.ims
-
- -
-Donny (Family, season 5) -- Look what I found! -

Modules -- htmllib -- base

-
-Dawn (Family, season 5) -- This is the source of my gladness. -

Modules -- htmllib -- base (example)

-
-Riley (Family, season 5) -- Every time I think I'm getting close to you... -

Modules -- urllib/urllib2

-
-Glory (Family, season 5) -- I am great and I am beautiful. -

Modules -- urllib/urllib2 (cont'd)

-
-Joyce (Ted, season 2) -- He redid my entire system. -

Modules -- urllib/urllib2 (examples)

-
-Xander (Ted, season 2) -- Yum-my! -

Digression -- HTTP Overview

-
-Tara (Family, season 5) -- ...in terms of the karmic cycle. -

Example HTTP Sessions

- -
-GET /foo/bar.html HTTP/1.0
-Host: www.example.org
-<blank line>
-
- - - -
-HTTP/1.0 200 OK
-Content-Type: text/html
-
-<html><body>lalalala</body></html>
-
- -
-Giles (Family, season 5) -- And you are talking about what on earth? -

Modules -- httplib

-
-Mr. MacLay (Family, season 5) -- We know how to control her...problem. -

Modules -- httplib -- example

- -
->>> import httplib
->>> h=httplib.HTTP("moshez.org")
->>> h.putrequest('GET', '/')
->>> h.putheader('Host', 'moshez.org')
->>> h.endheaders()
->>> h.getreply()
-(200, 'OK', <mimetools.Message instance at 0x81220dc>)
->>> h.getfile().read(10)
-"<HTML>\n<HE"
-
-
-Anya (Family, season 5) -- ...and it was fun! -

Modules -- urlparse

-
-Buffy (Family, season 5) -- You know what, you guys, just leave it here. -

Downloading Dilbert

-
-import urllib2, re
-
-URL = 'http://www.dilbert.com/'
-f = urllib2.urlopen(URL)
-s = f.read()
-href = re.compile('<a href="(/comics/.*?/dilbert.*?gif)">')
-m = href.search(value)
-f = urllib2.urlretrieve(urlparse.urljoin(URL, m.group(1)),
-                        "dilbert.gif")
-
-
-Tara (Family, season 5) -- That was funny if you [...] are a complete dork. -

Downloading Dark Angel Transcripts

- -
-import urllib2, htmllib, formatter, posixpath
-URL="http://www.darkangelfan.com/episode/"
-LINK_RE = re.compile('/trans_[0-9]+\.shtml$')
-s = urllib2.urlopen(URL).read()
-h = htmllib.HTMLParser(formatter.NullFormatter())
-h.feed(s)
-links = [urlparse.urljoin(URL, link)
-              for link in h.anchorlist if LINK_RE.search(link)]
-### -- really download --
-for link in links:
-    urllib2.urlretrieve(link, posixpath.basename(link))
-
- -
-Intern (Family, season 5) -- Yeah. That makes like five this month. -

Downloading Dark Angel Transcripts (select)

- -
-class Downloader:
-
-    def __init__(self, fin, fout):
-        self.fin, self.fout, self.fileno = fin, fout, fin.fileno
-
-    def read(self):
-        buf = self.fin.read(4096)
-        if not buf:
-            for f in [self.fout, self.fin]: f.close()
-            return 1
-        self.fout.write(buf)
-
-
-Joyce (Ted, season 2) -- I've been looking for the right moment. -

Downloading Dark Angel Transcripts (select, cont'd)

- -
-downloaders = [Downloader(urllib2.urlopen(link),
-                 open(posixpath.basename(link), 'wb'))
-                                      for link in links]
-while downloaders:
-    toRead = select.select(None, [downloaders], [], [])
-    for downloader in toRead:
-         if downloader.read():
-             downloaders.remove(downloader)
-
-
-Buffy (Family, season 5) -- Tara's damn birthday is just one too many things for me to worry about. -

Downloading Dark Angel Transcripts (threads)

- -
-import threading
-
-for link in links:
-    Thread(target=urllib2.urlretrieve,
-           args=(link,posixpath.basename(link)))
-
-
-Buffy (Ted, season 2) -- Sounds like fun. -

Digression - twisted.web.client

-
-Buffy (Ted, season 2) -- You're supposed to use your powers for good! -

Downloading Dark Angel Transcripts (web.client)

-
-from twisted.web import client
-from twisted.internet import import reactor, defer
-
-defer.DeferredList(
-[client.downloadPage(link, posixpath.basename(link))
-            for link in links]).addBoth(lambda _: reactor.stop())
-reactor.run()
-
-
-Ted (Ted, season 2) -- You don't have to worry about anything. -

HTTP Authentication

-
-Buffy (Ted, season 2) -- Ummm... Who are these people? -

HTTP Authentication - manually

-
-user = 'moshez'
-password = 's3kr1t'
-import httplib
-h=httplib.HTTP("localhost")
-h.putrequest('GET', '/protected/stuff.html')
-h.putheader('Authorization',
-            base64.encodestring(user+":"+password).strip())
-h.endheaders()
-h.getreply()
-print h.getfile().read()
-
-
-Tara (Family, season 5) -- And, uh, these are my-my friends. -

HTTP Authentication - urllib2

-
-Xander (Ted, season 2) -- I am really jinxing the hell out of us. -

Further Reading

-
-Willow (Ted, season 2) -- 'Book-cracker Buffy', it's kind of her nickname. -

Questions?

-Buffy (Family, season 5) -- I let you come, now sit down and look studious. -

Bonus Slides

-Tara (Family, season 5) -- You always make me feel special. - -

Cookies

-
-Ted (Ted, season 2) -- Who's up for dessert? I made chocolate-chip cookies! -

urllib2 cookies

-
-Joyce (Ted, season 2) -- Mm! Buffy, you've got to try one of these! -

Logging Into Advogato

-
-
-import urllib2
-
-u = urllib2.urlopen("http://advogato.org/acct/loginsub.html",
-                    urllib2.urlencode({'u': 'moshez',
-                                       'pass': 'not my real pass'})
-cookie = u.info()['set-cookie']
-cookie = cookie[:cookie.find(';')]
-r = Request('http://advogato.org/diary/post.html',
-            urllib2.urlencode(
-            {'entry': open('entry').read(), 'post': 'Post'}),
-            {'Cookie': cookie})
-urllib2.urlopen(r).read()
-
- -
-Anya (Family, season 5) -- I have a place in the world now. -

On Being Nice - Robots

-
-Willow (Ted, season 2) -- There were design features in that robot that pre-date... -

Using robotparser

-
-import robotparser
-rp = robotparser.RobotFileParser()
-rp.set_url('http://www.example.com/robots.txt')
-rp.read()
-if not rp.can_fetch('', 'http://www.example.com/'):
-    sys.exit(1)
-
- -
-Buffy (Ted, season 2) -- Tell me you didn't keep any parts. -

webchecker

-
-Willow (Ted, season 2) -- What do you mean, check him out? -

websucker

-
-Buffy (Ted, season 2) -- Find out his secrets, hack into his life. - - diff --git a/docs/historic/2003/haifux/haifux.html b/docs/historic/2003/haifux/haifux.html deleted file mode 100644 index 44206147b14..00000000000 --- a/docs/historic/2003/haifux/haifux.html +++ /dev/null @@ -1,2235 +0,0 @@ -Evolution of Finger -

Evolution of Finger

- -

Refuse Connections

- -
-from twisted.internet import reactor
-reactor.run()
-
- -

Here, we just run the reactor. Nothing at all will happen, -until we interrupt the program. It will not consume (almost) -no CPU resources. Not very useful, perhaps -- but this -is the skeleton inside which the Twisted program -will grow.

- -

Do Nothing

- -
-from twisted.internet import protocol, reactor
-class FingerProtocol(protocol.Protocol):
-    pass
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-reactor.listenTCP(1079, FingerFactory())
-reactor.run()
-
- -

Here, we start listening on port 1079 [which is supposed to be -a reminder that eventually, we want to run on port 79, the port -the finger server is supposed to run on. We define a protocol which -does not respond to any events. Thus, connections to 1079 will -be accepted, but the input ignored.

- -

Drop Connections

- -
-from twisted.internet import protocol, reactor
-class FingerProtocol(protocol.Protocol):
-    def connectionMade(self):
-        self.transport.loseConnection()
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-reactor.listenTCP(1079, FingerFactory())
-reactor.run()
-
- -

Here we add to the protocol the ability to respond to the -event of beginning a connection -- by terminating it. -Perhaps not an interesting behaviour, but it is already -not that far from behaving according to the letter of the -protocol. After all, there is no requirement to send any -data to the remote connection in the standard, is there. -The only technical problem is that we terminate the connection -too soon. A client which is slow enough will see his send() -of the username result in an error.

- -

Read Username, Drop Connections

- -
-from twisted.internet import protocol, reactor
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.transport.loseConnection()
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-reactor.listenTCP(1079, FingerFactory())
-reactor.run()
-
- -

Here we make FingerProtocol inherit from -LineReceiver, so that we get data-based events -on a line-by-line basis. We respond to the event of receiving -the line with shutting down the connection. Congratulations, -this is the first standard-compliant version of the code. -However, usually people actually expect some data about -users to be transmitted.

- - -

Read Username, Output Error, Drop Connections

- -
-from twisted.internet import protocol, reactor
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.transport.write("No such user\r\n")
-        self.transport.loseConnection()
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-reactor.listenTCP(1079, FingerFactory())
-reactor.run()
-
- -

Finally, a useful version. Granted, the usefulness is somewhat -limited by the fact that this version only prints out a no such -user message. It could be used for devestating effect in honeypots, -of course :)

- -

Output From Empty Factory

- -
-# Read username, output from empty factory, drop connections
-from twisted.internet import protocol, reactor
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.transport.write(self.factory.getUser(user)+"\r\n")
-        self.transport.loseConnection()
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-    def getUser(self, user): return "No such user"
-reactor.listenTCP(1079, FingerFactory())
-reactor.run()
-
- -

The same behaviour, but finally we see what usefuleness the -factory has: as something that does not get constructed for -every connection, it can be in charge of the user database. -In particular, we won't have to change the protocol if -the user database backend changes.

- -

Output from Non-empty Factory

- -
-# Read username, output from non-empty factory, drop connections
-from twisted.internet import protocol, reactor
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.transport.write(self.factory.getUser(user)+"\r\n")
-        self.transport.loseConnection()
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-    def __init__(self, **kwargs): self.users = kwargs
-    def getUser(self, user):
-        return self.users.get(user, "No such user")
-reactor.listenTCP(1079, FingerFactory(moshez='Happy and well'))
-reactor.run()
-
- -

Finally, a really useful finger database. While it does not -supply information about logged in users, it could be used to -distribute things like office locations and internal office -numbers. As hinted above, the factory is in charge of keeping -the user database: note that the protocol instance has not -changed. This is starting to look good: we really won't have -to keep tweaking our protocol.

- -

Use Deferreds

- -
-# Read username, output from non-empty factory, drop connections
-# Use deferreds, to minimize synchronicity assumptions
-from twisted.internet import protocol, reactor, defer
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-    def __init__(self, **kwargs): self.users = kwargs
-    def getUser(self, user):
-        return defer.succeed(self.users.get(user, "No such user"))
-reactor.listenTCP(1079, FingerFactory(moshez='Happy and well'))
-reactor.run()
-
- -

But, here we tweak it just for the hell of it. Yes, while the -previous version worked, it did assume the result of getUser is -always immediately available. But what if instead of an in memory -database, we would have to fetch result from a remote Oracle? -Or from the web? Or, or...

- -

Run 'finger' Locally

- -
-# Read username, output from factory interfacing to OS, drop connections
-from twisted.internet import protocol, reactor, defer, utils
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-    def getUser(self, user):
-        return utils.getProcessOutput("finger", [user])
-reactor.listenTCP(1079, FingerFactory())
-reactor.run()
-
- -

...from running a local command? Yes, this version (safely!) runs -finger locally with whatever arguments it is given, and returns the -standard output. This will do exactly what the standard version -of the finger server does -- without the need for any remote buffer -overflows, as the networking is done safely.

- -

Read Status from the Web

- -
-# Read username, output from factory interfacing to web, drop connections
-from twisted.internet import protocol, reactor, defer, utils
-from twisted.protocols import basic
-from twisted.web import client
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-    def __init__(self, prefix): self.prefix=prefix
-    def getUser(self, user):
-        return client.getPage(self.prefix+user)
-reactor.listenTCP(1079, FingerFactory(prefix='http://livejournal.com/~'))
-reactor.run()
-
- -

The web. That invention which has infiltrated homes around the -world finally gets through to our invention. Here we use the built-in -Twisted web client, which also returns a deferred. Finally, we manage -to have examples of three different database backends, which do -not change the protocol class. In fact, we will not have to change -the protocol again until the end of this talk: we have achieved, -here, one truly usable class.

- - -

Use Application

- -
-# Read username, output from non-empty factory, drop connections
-# Use deferreds, to minimize synchronicity assumptions
-# Write application. Save in 'finger.tpy'
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-    def __init__(self, **kwargs): self.users = kwargs
-    def getUser(self, user):
-        return defer.succeed(self.users.get(user, "No such user"))
-application = app.Application('finger', uid=1, gid=1)
-application.listenTCP(79, FingerFactory(moshez='Happy and well'))
-
- -

Up until now, we faked. We kept using port 1079, because really, -who wants to run a finger server with root privileges? Well, the -common solution is "privilege shedding": after binding to the network, -become a different, less privileged user. We could have done it ourselves, -but Twisted has a builtin way to do it. Create a snippet as above, -defining an application object. That object will have uid and gid -attributes. When running it (later we will see how) it will bind -to ports, shed privileges and then run.

- -

twistd

- -
-root% twistd -ny finger.tpy # just like before
-root% twistd -y finger.tpy # daemonize, keep pid in twistd.pid
-root% twistd -y finger.tpy --pidfile=finger.pid
-root% twistd -y finger.tpy --rundir=/
-root% twistd -y finger.tpy --chroot=/var
-root% twistd -y finger.tpy -l /var/log/finger.log
-root% twistd -y finger.tpy --syslog # just log to syslog
-root% twistd -y finger.tpy --syslog --prefix=twistedfinger # use given prefix
-
- -

This is how to run "Twisted Applications" -- files which define an -'application'. twistd (TWISTed Daemonizer) does everything a daemon -can be expected to -- shuts down stdin/stdout/stderr, disconnects -from the terminal and can even change runtime directory, or even -the root filesystems. In short, it does everything so the Twisted -application developer can concentrate on writing his networking code.

- -

Setting Message By Local Users

- -
-# But let's try and fix setting away messages, shall we?
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerFactory(protocol.ServerFactory):
-    protocol = FingerProtocol
-    def __init__(self, **kwargs): self.users = kwargs
-    def getUser(self, user):
-        return defer.succeed(self.users.get(user, "No such user"))
-class FingerSetterProtocol(basic.LineReceiver):
-      def connectionMade(self): self.lines = []
-      def lineReceived(self, line): self.lines.append(line)
-      def connectionLost(self): self.factory.setUser(*self.lines)
-class FingerSetterFactory(protocol.ServerFactory):
-      def __init__(self, ff): self.setUser = self.ff.users.__setitem__
-ff = FingerFactory(moshez='Happy and well')
-fsf = FingerSetterFactory(ff)
-application = app.Application('finger', uid=1, gid=1)
-application.listenTCP(79, ff)
-application.listenTCP(1079, fsf, interface='127.0.0.1')
-
- -

Now that port 1079 is free, maybe we can run on it a different -server, one which will let people set their messages. It does -no access control, so anyone who can login to the machine can -set any message. We assume this is the desired behaviour in -our case. Testing it can be done by simply: -

- -
-% nc localhost 1079
-moshez
-Giving a talk now, sorry!
-^D
-
- -

Use Services to Make Dependencies Sane

- -
-# Fix asymmetry
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerSetterProtocol(basic.LineReceiver):
-      def connectionMade(self): self.lines = []
-      def lineReceived(self, line): self.lines.append(line)
-      def connectionLost(self): self.factory.setUser(*self.lines)
-class FingerService(app.ApplicationService):
-      def __init__(self, *args, **kwargs):
-          app.ApplicationService.__init__(self, *args)
-          self.users = kwargs
-      def getUser(self, user):
-          return defer.succeed(self.users.get(u, "No such user"))
-      def getFingerFactory(self):
-          f = protocol.ServerFactory()
-          f.protocol, f.getUser = FingerProtocol, self.getUser
-          return f
-      def getFingerSetterFactory(self):
-          f = protocol.ServerFactory()
-          f.protocol, f.setUser = FingerSetterProtocol, self.users.__setitem__
-          return f
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService(application, 'finger', moshez='Happy and well')
-application.listenTCP(79, f.getFingerFactory())
-application.listenTCP(1079, f.getFingerSetterFactory(), interface='127.0.0.1')
-
- -

The previous version had the setter poke at the innards of the -finger factory. It's usually not a good idea: this version makes -both factories symmetric by making them both look at a single -object. Services are useful for when an object is needed which is -not related to a specific network server. Here, we moved all responsibility -for manufacturing factories into the service. Note that we stopped -subclassing: the service simply puts useful methods and attributes -inside the factories. We are getting better at protocol design: -none of our protocol classes had to be changed, and neither will -have to change until the end of the talk.

- -

Read Status File

- -
-# Read from file
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerSetterProtocol(basic.LineReceiver):
-      def connectionMade(self): self.lines = []
-      def lineReceived(self, line): self.lines.append(line)
-      def connectionLost(self): self.factory.setUser(*self.lines)
-class FingerService(app.ApplicationService):
-      def __init__(self, file, *args, **kwargs):
-          app.ApplicationService.__init__(self, *args, **kwargs)
-          self.file = file
-      def startService(self):
-          app.ApplicationService.startService(self)
-          self._read()
-      def _read(self):
-          self.users = {}
-          for line in file(self.file):
-              user, status = line.split(':', 1)
-              self.users[user] = status
-          self.call = reactor.callLater(30, self._read)
-      def stopService(self):
-          app.ApplicationService.stopService(self)
-          self.call.cancel()
-      def getUser(self, user):
-          return defer.succeed(self.users.get(u, "No such user"))
-      def getFingerFactory(self):
-          f = protocol.ServerFactory()
-          f.protocol, f.getUser = FingerProtocol, self.getUser
-          return f
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, f.getFingerFactory())
-
- -

This version shows how, instead of just letting users set their -messages, we can read those from a centrally managed file. We cache -results, and every 30 seconds we refresh it. Services are useful -for such scheduled tasks.

- -

Announce on Web, Too

- -
-# Read from file, announce on the web!
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic
-from twisted.web import resource, server, static
-import cgi
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerSetterProtocol(basic.LineReceiver):
-      def connectionMade(self): self.lines = []
-      def lineReceived(self, line): self.lines.append(line)
-      def connectionLost(self): self.factory.setUser(*self.lines)
-class FingerService(app.ApplicationService):
-      def __init__(self, file, *args, **kwargs):
-          app.ApplicationService.__init__(self, *args, **kwargs)
-          self.file = file
-      def startService(self):
-          app.ApplicationService.startService(self)
-          self._read()
-      def _read(self):
-          self.users = {}
-          for line in file(self.file):
-              user, status = line.split(':', 1)
-              self.users[user] = status
-          self.call = reactor.callLater(30, self._read)
-      def stopService(self):
-          app.ApplicationService.stopService(self)
-          self.call.cancel()
-      def getUser(self, user):
-          return defer.succeed(self.users.get(u, "No such user"))
-      def getFingerFactory(self):
-          f = protocol.ServerFactory()
-          f.protocol, f.getUser = FingerProtocol, self.getUser
-          return f
-      def getResource(self):
-          r = resource.Resource()
-          r.getChild = (lambda path, request:
-           static.Data('text/html',
-            '<h1>%s</h1><p>%s</p>' %
-              tuple(map(cgi.escape,
-                        [path,self.users.get(path, "No such user")]))))
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, f.getFingerFactory())
-application.listenTCP(80, server.Site(f.getResource()))
-
- -

The same kind of service can also produce things useful for -other protocols. For example, in twisted.web, the factory -itself (the site) is almost never subclassed -- instead, -it is given a resource, which represents the tree of resources -available via URLs. That hierarchy is navigated by site, -and overriding it dynamically is possible with getChild.

- -

Announce on IRC, Too

- -
-# Read from file, announce on the web, irc
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic, irc
-from twisted.web import resource, server, static
-import cgi
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerSetterProtocol(basic.LineReceiver):
-      def connectionMade(self): self.lines = []
-      def lineReceived(self, line): self.lines.append(line)
-      def connectionLost(self): self.factory.setUser(*self.lines)
-class IRCReplyBot(irc.IRCClient):
-    def connectionMade(self):
-        self.nickname = self.factory.nickname
-        irc.IRCClient.connectionMade(self)
-    def privmsg(self, user, channel, msg):
-        if user.lower() == channel.lower():
-            self.factory.getUser(msg
-            ).addErrback(lambda _: "Internal error in server"
-            ).addCallback(lambda m: self.msg(user, m))
-class FingerService(app.ApplicationService):
-      def __init__(self, file, *args, **kwargs):
-          app.ApplicationService.__init__(self, *args, **kwargs)
-          self.file = file
-      def startService(self):
-          app.ApplicationService.startService(self)
-          self._read()
-      def _read(self):
-          self.users = {}
-          for line in file(self.file):
-              user, status = line.split(':', 1)
-              self.users[user] = status
-          self.call = reactor.callLater(30, self._read)
-      def stopService(self):
-          app.ApplicationService.stopService(self)
-          self.call.cancel()
-      def getUser(self, user):
-          return defer.succeed(self.users.get(u, "No such user"))
-      def getFingerFactory(self):
-          f = protocol.ServerFactory()
-          f.protocol, f.getUser = FingerProtocol, self.getUser
-          return f
-      def getResource(self):
-          r = resource.Resource()
-          r.getChild = (lambda path, request:
-           static.Data('text/html',
-            '<h1>%s</h1><p>%s</p>' %
-              tuple(map(cgi.escape,
-                        [path,self.users.get(path, "No such user")]))))
-      def getIRCBot(self, nickname):
-          f = protocol.ReconnectingClientFactory()
-          f.protocol,f.nickname,f.getUser = IRCReplyBot,nickname,self.getUser
-          return f
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, f.getFingerFactory())
-application.listenTCP(80, server.Site(f.getResource()))
-application.connectTCP('irc.freenode.org', 6667, f.getIRCBot('finger-bot'))
-
- -

This is the first time there is client code. IRC clients often -act a lot like servers: responding to events form the network. -The reconnecting client factory will make sure that severed links -will get re-established, with intelligent tweaked exponential -backoff algorithms. The irc client itself is simple: the only -real hack is getting the nickname from the factory in connectionMade.

- - - -

Add XML-RPC Support

- -
-# Read from file, announce on the web, irc, xml-rpc
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic, irc
-from twisted.web import resource, server, static, xmlrpc
-import cgi
-class FingerProtocol(basic.LineReceiver):
-    def lineReceived(self, user):
-        self.factory.getUser(user
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m:
-            (self.transport.write(m+"\r\n"),self.transport.loseConnection()))
-class FingerSetterProtocol(basic.LineReceiver):
-      def connectionMade(self): self.lines = []
-      def lineReceived(self, line): self.lines.append(line)
-      def connectionLost(self): self.factory.setUser(*self.lines)
-class IRCReplyBot(irc.IRCClient):
-    def connectionMade(self):
-        self.nickname = self.factory.nickname
-        irc.IRCClient.connectionMade(self)
-    def privmsg(self, user, channel, msg):
-        if user.lower() == channel.lower():
-            self.factory.getUser(msg
-        ).addErrback(lambda _: "Internal error in server"
-        ).addCallback(lambda m: self.msg(user, m))
-class FingerService(app.ApplicationService):
-      def __init__(self, file, *args, **kwargs):
-          app.ApplicationService.__init__(self, *args, **kwargs)
-          self.file = file
-      def startService(self):
-          app.ApplicationService.startService(self)
-          self._read()
-      def _read(self):
-          self.users = {}
-          for line in file(self.file):
-              user, status = line.split(':', 1)
-              self.users[user] = status
-          self.call = reactor.callLater(30, self._read)
-      def stopService(self):
-          app.ApplicationService.stopService(self)
-          self.call.cancel()
-      def getUser(self, user):
-          return defer.succeed(self.users.get(u, "No such user"))
-      def getFingerFactory(self):
-          f = protocol.ServerFactory()
-          f.protocol, f.getUser = FingerProtocol, self.getUser
-          return f
-      def getResource(self):
-          r = resource.Resource()
-          r.getChild = (lambda path, request:
-           static.Data('text/html',
-            '<h1>%s</h1><p>%s</p>' %
-              tuple(map(cgi.escape,
-                        [path,self.users.get(path, "No such user")]))))
-          x = xmlrpc.XMLRPRC()
-          x.xmlrpc_getUser = self.getUser
-          r.putChild('RPC2.0', x)
-          return r
-      def getIRCBot(self, nickname):
-          f = protocol.ReconnectingClientFactory()
-          f.protocol,f.nickname,f.getUser = IRCReplyBot,nickname,self.getUser
-          return f
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, f.getFingerFactory())
-application.listenTCP(80, server.Site(f.getResource()))
-application.connectTCP('irc.freenode.org', 6667, f.getIRCBot('finger-bot'))
-
- -

In Twisted, XML-RPC support is handled just as though it was -another resource. That resource will still support GET calls normally -through render(), but that is usually left unimplemented. Note -that it is possible to return deferreds from XML-RPC methods. -The client, of course, will not get the answer until the deferred -is triggered.

- - -

Write Readable Code

- -
-# Do everything properly
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic, irc
-from twisted.web import resource, server, static, xmlrpc
-import cgi
-
-def catchError(err):
-    return "Internal error in server"
-
-class FingerProtocol(basic.LineReceiver):
-
-    def lineReceived(self, user):
-        d = self.factory.getUser(user)
-        d.addErrback(catchError)
-        def writeValue(value):
-            self.transport.write(value)
-            self.transport.loseConnection()
-        d.addCallback(writeValue)
-
-
-class FingerSetterProtocol(basic.LineReceiver):
-
-      def connectionMade(self):
-          self.lines = []
-
-      def lineReceived(self, line):
-          self.lines.append(line)
-
-      def connectionLost(self):
-          if len(self.lines) == 2:
-              self.factory.setUser(*self.lines)
-
-
-class IRCReplyBot(irc.IRCClient):
-
-    def connectionMade(self):
-        self.nickname = self.factory.nickname
-        irc.IRCClient.connectionMade(self)
-
-    def privmsg(self, user, channel, msg):
-        if user.lower() == channel.lower():
-            d = self.factory.getUser(msg)
-            d.addErrback(catchError)
-            d.addCallback(lambda m: "Status of %s: %s" % (user, m))
-            d.addCallback(lambda m: self.msg(user, m))
-
-
-class UserStatusTree(resource.Resource):
-
-    def __init__(self, service):
-        resource.Resource.__init__(self):
-        self.service = service
-
-    def render(self, request):
-        d = self.service.getUsers()
-        def formatUsers(users):
-            l = ["<li><a href="%s">%s</a></li> % (user, user)
-                for user in users]
-            return '<ul>'+''.join(l)+'</ul>'
-        d.addCallback(formatUsers)
-        d.addCallback(request.write)
-        d.addCallback(lambda _: request.finish())
-        return server.NOT_DONE_YET
-
-    def getChild(self, path, request):
-        return UserStatus(path, self.service)
-
-
-class UserStatus(resource.Resource):
-
-    def __init__(self, user, service):
-        resource.Resource.__init__(self):
-        self.user = user
-        self.service = service
-
-    def render(self, request):
-        d = self.service.getUser(self.user)
-        d.addCallback(cgi.escape)
-        d.addCallback(lambda m:
-                     '<h1>%s</h1>'%self.user+'<p>%s</p>'%m)
-        d.addCallback(request.write)
-        d.addCallback(lambda _: request.finish())
-        return server.NOT_DONE_YET
-
-
-class UserStatusXR(xmlrpc.XMLPRC):
-
-    def __init__(self, service):
-        xmlrpc.XMLRPC.__init__(self)
-        self.service = service
-
-    def xmlrpc_getUser(self, user):
-        return self.service.getUser(user)
-
-
-class FingerService(app.ApplicationService):
-
-      def __init__(self, file, *args, **kwargs):
-          app.ApplicationService.__init__(self, *args, **kwargs)
-          self.file = file
-
-      def startService(self):
-          app.ApplicationService.startService(self)
-          self._read()
-
-      def _read(self):
-          self.users = {}
-          for line in file(self.file):
-              user, status = line.split(':', 1)
-              self.users[user] = status
-          self.call = reactor.callLater(30, self._read)
-
-      def stopService(self):
-          app.ApplicationService.stopService(self)
-          self.call.cancel()
-
-      def getUser(self, user):
-          return defer.succeed(self.users.get(u, "No such user"))
-
-      def getUsers(self):
-          return defer.succeed(self.users.keys())
-
-      def getFingerFactory(self):
-          f = protocol.ServerFactory()
-          f.protocol = FingerProtocol
-          f.getUser = self.getUser
-          return f
-
-      def getResource(self):
-          r = UserStatusTree(self)
-          x = UserStatusXR(self)
-          r.putChild('RPC2.0', x)
-          return r
-
-      def getIRCBot(self, nickname):
-          f = protocol.ReconnectingClientFactory()
-          f.protocol = IRCReplyBot
-          f.nickname = nickname
-          f.getUser = self.getUser
-          return f
-
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, f.getFingerFactory())
-application.listenTCP(80, server.Site(f.getResource()))
-application.connectTCP('irc.freenode.org', 6667, f.getIRCBot('finger-bot'))
-
- -

The last version of the application had a lot of hacks. We avoided -subclassing, did not support things like user listings in the web -support, and removed all blank lines -- all in the interest of code -which is shorter. Here we take a step back, subclass what is more -naturally a subclass, make things which should take multiple lines -take them, etc. This shows a much better style of developing Twisted -applications, though the hacks in the previous stages are sometimes -used in throw-away prototypes.

- -

Write Maintainable Code

- -
-# Do everything properly, and componentize
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic, irc
-from twisted.python import components
-from twisted.web import resource, server, static, xmlrpc
-import cgi
-
-class IFingerService(components.Interface):
-
-    def getUser(self, user):
-        '''Return a deferred returning a string'''
-
-    def getUsers(self):
-        '''Return a deferred returning a list of strings'''
-
-class IFingerSettingService(components.Interface):
-
-    def setUser(self, user, status):
-        '''Set the user's status to something'''
-    
-def catchError(err):
-    return "Internal error in server"
-
-
-class FingerProtocol(basic.LineReceiver):
-
-    def lineReceived(self, user):
-        d = self.factory.getUser(user)
-        d.addErrback(catchError)
-        def writeValue(value):
-            self.transport.write(value)
-            self.transport.loseConnection()
-        d.addCallback(writeValue)
-
-
-class IFingerFactory(components.Interface):
-
-    def getUser(self, user):
-        """Return a deferred returning a string""""
-
-    def buildProtocol(self, addr):
-        """Return a protocol returning a string""""
-
-
-class FingerFactoryFromService(protocol.ServerFactory):
-
-    __implements__ = IFingerFactory,
-
-    protocol = FingerProtocol
-
-    def __init__(self, service):
-        self.service = service
-
-    def getUser(self, user):
-        return self.service.getUser(user)
-
-components.registerAdapter(FingerFactoryFromService, IFingerService)
-
-
-class FingerSetterProtocol(basic.LineReceiver):
-
-      def connectionMade(self):
-          self.lines = []
-
-      def lineReceived(self, line):
-          self.lines.append(line)
-
-      def connectionLost(self):
-          if len(self.lines) == 2:
-              self.factory.setUser(*self.lines)
-
-
-class IFingerSetterFactory(components.Interface):
-
-    def setUser(self, user, status):
-        """Return a deferred returning a string"""
-
-    def buildProtocol(self, addr):
-        """Return a protocol returning a string"""
-
-
-class FingerSetterFactoryFromService(protocol.ServerFactory):
-
-    __implements__ = IFingerSetterFactory,
-
-    protocol = FingerSetterProtocol
-
-    def __init__(self, service):
-        self.service = service
-
-    def setUser(self, user, status):
-        self.service.setUser(user, status)
-
-
-components.registerAdapter(FingerSetterFactoryFromService,
-                           IFingerSettingService)
-    
-class IRCReplyBot(irc.IRCClient):
-
-    def connectionMade(self):
-        self.nickname = self.factory.nickname
-        irc.IRCClient.connectionMade(self)
-
-    def privmsg(self, user, channel, msg):
-        if user.lower() == channel.lower():
-            d = self.factory.getUser(msg)
-            d.addErrback(catchError)
-            d.addCallback(lambda m: "Status of %s: %s" % (user, m))
-            d.addCallback(lambda m: self.msg(user, m))
-
-
-class IIRCClientFactory(components.Interface):
-
-    '''
-    @ivar nickname
-    '''
-
-    def getUser(self, user):
-        """Return a deferred returning a string"""
-
-    def buildProtocol(self, addr):
-        """Return a protocol"""
-
-
-class IRCClientFactoryFromService(protocol.ClientFactory):
-
-   __implements__ = IIRCClientFactory,
-
-   protocol = IRCReplyBot
-   nickname = None
-
-   def __init__(self, service):
-       self.service = service
-
-    def getUser(self, user):
-        return self.service.getUser()
-
-components.registerAdapter(IRCClientFactoryFromService, IFingerService)
-
-class UserStatusTree(resource.Resource):
-
-    def __init__(self, service):
-        resource.Resource.__init__(self):
-        self.putChild('RPC2.0', UserStatusXR(self.service))
-        self.service = service
-
-    def render(self, request):
-        d = self.service.getUsers()
-        def formatUsers(users):
-            l = ["<li><a href="%s">%s</a></li> % (user, user)
-                for user in users]
-            return '<ul>'+''.join(l)+'</ul>'
-        d.addCallback(formatUsers)
-        d.addCallback(request.write)
-        d.addCallback(lambda _: request.finish())
-        return server.NOT_DONE_YET
-
-    def getChild(self, path, request):
-        return UserStatus(path, self.service)
-
-components.registerAdapter(UserStatusTree, IFingerService)
-
-class UserStatus(resource.Resource):
-
-    def __init__(self, user, service):
-        resource.Resource.__init__(self):
-        self.user = user
-        self.service = service
-
-    def render(self, request):
-        d = self.service.getUser(self.user)
-        d.addCallback(cgi.escape)
-        d.addCallback(lambda m:
-                      '<h1>%s</h1>'%self.user+'<p>%s</p>'%m)
-        d.addCallback(request.write)
-        d.addCallback(lambda _: request.finish())
-        return server.NOT_DONE_YET
-
-
-class UserStatusXR(xmlrpc.XMLPRC):
-
-    def __init__(self, service):
-        xmlrpc.XMLRPC.__init__(self)
-        self.service = service
-
-    def xmlrpc_getUser(self, user):
-        return self.service.getUser(user)
-
-
-class FingerService(app.ApplicationService):
-
-    __implements__ = IFingerService,
-
-    def __init__(self, file, *args, **kwargs):
-        app.ApplicationService.__init__(self, *args, **kwargs)
-        self.file = file
-
-    def startService(self):
-        app.ApplicationService.startService(self)
-        self._read()
-
-    def _read(self):
-        self.users = {}
-        for line in file(self.file):
-            user, status = line.split(':', 1)
-            self.users[user] = status
-        self.call = reactor.callLater(30, self._read)
-
-    def stopService(self):
-        app.ApplicationService.stopService(self)
-        self.call.cancel()
-
-    def getUser(self, user):
-        return defer.succeed(self.users.get(u, "No such user"))
-
-    def getUsers(self):
-        return defer.succeed(self.users.keys())
-
-
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, IFingerFactory(f))
-application.listenTCP(80, server.Site(resource.IResource(f)))
-i = IIRCClientFactory(f)
-i.nickname = 'fingerbot'
-application.connectTCP('irc.freenode.org', 6667, i)
-
- -

In the last version, the service class was three times longer than -any other class, and was hard to understand. This was because it turned -out to have multiple responsibilities. It had to know how to access -user information, by scheduling a reread of the file ever half minute, -but also how to display itself in a myriad of protocols. Here, we -used the component-based architecture that Twisted provides to achieve -a separation of concerns. All the service is responsible for, now, -is supporting getUser/getUsers. It declares its support via the -__implements__ keyword. Then, adapters are used to make this service -look like an appropriate class for various things: for supplying -a finger factory to listenTCP, for supplying a resource to site's -constructor, and to provide an IRC client factory for connectTCP. -All the adapters use are the methods in FingerService they are -declared to use: getUser/getUsers. We could, of course, -skipped the interfaces and let the configuration code use -things like FingerFactoryFromService(f) directly. However, using -interfaces provides the same flexibility inheritance gives: future -subclasses can override the adapters.

- - - -

Advantages of Latest Version

- - - -
-class MemoryFingerService(app.ApplicationService):
-      __implements__ = IFingerService, IFingerSetterService
-
-      def __init__(self, *args, **kwargs):
-          app.ApplicationService.__init__(self, *args)
-          self.users = kwargs
-
-      def getUser(self, user):
-          return defer.succeed(self.users.get(u, "No such user"))
-
-      def getUsers(self):
-          return defer.succeed(self.users.keys())
-
-      def setUser(self, user, status):
-          self.users[user] = status
-
-application = app.Application('finger', uid=1, gid=1)
-# New constructor call
-f = MemoryFingerService(application, 'finger', moshez='Happy and well')
-application.listenTCP(79, IFingerFactory(f))
-application.listenTCP(80, server.Site(resource.IResource(f)))
-i = IIRCClientFactory(f)
-i.nickname = 'fingerbot'
-application.connectTCP('irc.freenode.org', 6667, i)
-# New: run setter too
-application.listenTCP(1079, IFingerSetterFactory(f), interface='127.0.0.1')
-
- -

Here we show just how convenient it is to implement new backends -when we move to a component based architecture. Note that here -we also use an interface we previously wrote, FingerSetterFactory, -by supporting one single method. We manage to preserve the service's -ignorance of the network.

- -

Another Backend

- -
-class LocalFingerService(app.ApplicationService):
-      __implements__ = IFingerService
-
-      def getUser(self, user):
-          return utils.getProcessOutput("finger", [user])
-
-      def getUsers(self):
-          return defer.succeed([])
-
-application = app.Application('finger', uid=1, gid=1)
-f = LocalFingerService(application, 'finger')
-application.listenTCP(79, IFingerFactory(f))
-application.listenTCP(80, server.Site(resource.IResource(f)))
-i = IIRCClientFactory(f)
-i.nickname = 'fingerbot'
-application.connectTCP('irc.freenode.org', 6667, i)
-
- -

We have already wrote this, but now we get more for less work: -the network code is completely separate from the backend.

- -

Yet Another Backend: Doing the Standard Thing

- -
-import pwd
-
-class LocalFingerService(app.ApplicationService):
-      __implements__ = IFingerService
-
-      def getUser(self, user):
-          try:
-              entry = pwd.getpwnam(user)
-          except KeyError:
-              return "No such user"
-          try:
-              f=file(os.path.join(entry[5],'.plan'))
-          except (IOError, OSError):
-              return "No such user"
-          data = f.read()
-          f.close()
-          return data
-
-      def getUsers(self):
-          return defer.succeed([])
-
-application = app.Application('finger', uid=1, gid=1)
-f = LocalFingerService(application, 'finger')
-application.listenTCP(79, IFingerFactory(f))
-application.listenTCP(80, server.Site(resource.IResource(f)))
-i = IIRCClientFactory(f)
-i.nickname = 'fingerbot'
-application.connectTCP('irc.freenode.org', 6667, i)
-
- -

Not much to say about that, except to indicate that by now we -can be churning out backends like crazy. Feel like doing a backend -for advogato, for example? Dig out the XML-RPC client support Twisted -has, and get to work!

- - -

Aspect Oriented Programming

- - - -

Use Woven

- -
-# Do everything properly, and componentize
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic, irc
-from twisted.python import components
-from twisted.web import resource, server, static, xmlrpc, microdom
-from twisted.web.woven import page, widget
-import cgi
-
-class IFingerService(components.Interface):
-
-    def getUser(self, user):
-        '''Return a deferred returning a string'''
-
-    def getUsers(self):
-        '''Return a deferred returning a list of strings'''
-
-class IFingerSettingService(components.Interface):
-
-    def setUser(self, user, status):
-        '''Set the user's status to something'''
-    
-def catchError(err):
-    return "Internal error in server"
-
-
-class FingerProtocol(basic.LineReceiver):
-
-    def lineReceived(self, user):
-        d = self.factory.getUser(user)
-        d.addErrback(catchError)
-        def writeValue(value):
-            self.transport.write(value)
-            self.transport.loseConnection()
-        d.addCallback(writeValue)
-
-
-class IFingerFactory(components.Interface):
-
-    def getUser(self, user):
-        """Return a deferred returning a string""""
-
-    def buildProtocol(self, addr):
-        """Return a protocol returning a string""""
-
-
-class FingerFactoryFromService(protocol.ServerFactory):
-
-    __implements__ = IFingerFactory,
-
-    protocol = FingerProtocol
-
-    def __init__(self, service):
-        self.service = service
-
-    def getUser(self, user):
-        return self.service.getUser(user)
-
-components.registerAdapter(FingerFactoryFromService, IFingerService)
-
-
-class FingerSetterProtocol(basic.LineReceiver):
-
-      def connectionMade(self):
-          self.lines = []
-
-      def lineReceived(self, line):
-          self.lines.append(line)
-
-      def connectionLost(self):
-          if len(self.lines) == 2:
-              self.factory.setUser(*self.lines)
-
-
-class IFingerSetterFactory(components.Interface):
-
-    def setUser(self, user, status):
-        """Return a deferred returning a string"""
-
-    def buildProtocol(self, addr):
-        """Return a protocol returning a string"""
-
-
-class FingerSetterFactoryFromService(protocol.ServerFactory):
-
-    __implements__ = IFingerSetterFactory,
-
-    protocol = FingerSetterProtocol
-
-    def __init__(self, service):
-        self.service = service
-
-    def setUser(self, user, status):
-        self.service.setUser(user, status)
-
-
-components.registerAdapter(FingerSetterFactoryFromService,
-                           IFingerSettingService)
-    
-class IRCReplyBot(irc.IRCClient):
-
-    def connectionMade(self):
-        self.nickname = self.factory.nickname
-        irc.IRCClient.connectionMade(self)
-
-    def privmsg(self, user, channel, msg):
-        if user.lower() == channel.lower():
-            d = self.factory.getUser(msg)
-            d.addErrback(catchError)
-            d.addCallback(lambda m: "Status of %s: %s" % (user, m))
-            d.addCallback(lambda m: self.msg(user, m))
-
-
-class IIRCClientFactory(components.Interface):
-
-    '''
-    @ivar nickname
-    '''
-
-    def getUser(self, user):
-        """Return a deferred returning a string"""
-
-    def buildProtocol(self, addr):
-        """Return a protocol"""
-
-
-class IRCClientFactoryFromService(protocol.ClientFactory):
-
-   __implements__ = IIRCClientFactory,
-
-   protocol = IRCReplyBot
-   nickname = None
-
-   def __init__(self, service):
-       self.service = service
-
-    def getUser(self, user):
-        return self.service.getUser()
-
-components.registerAdapter(IRCClientFactoryFromService, IFingerService)
-
-
-class UsersModel(model.MethodModel):
-
-    def __init__(self, service):
-        self.service = service
-
-    def wmfactory_users(self):
-        return self.service.getUsers()
-
-components.registerAdapter(UsersModel, IFingerService)
-
-class UserStatusTree(page.Page):
-
-    template = """<html><head><title>Users</title><head><body>
-                  <h1>Users</h1>
-                  <ul model="users" view="List">
-                  <li pattern="listItem" /><a view="Link" model="."
-                  href="dummy"><span model="." view="Text" /></a>
-                  </ul></body></html>"""
-
-    def initialize(self, **kwargs):
-        self.putChild('RPC2.0', UserStatusXR(self.model.service))
-
-    def getDynamicChild(self, path, request):
-        return UserStatus(user=path, service=self.model.service)
-
-components.registerAdapter(UserStatusTree, IFingerService)
-
-
-class UserStatus(page.Page):
-
-    template='''<html><head><title view="Text" model="user"/></heaD>
-    <body><h1 view="Text" model="user"/>
-    <p mode="status" view="Text" />
-    </body></html>'''
-
-    def initialize(self, **kwargs):
-        self.user = kwargs['user']
-        self.service = kwargs['service']
-
-    def wmfactory_user(self):
-        return self.user
-
-    def wmfactory_status(self):
-        return self.service.getUser(self.user)
-
-class UserStatusXR(xmlrpc.XMLPRC):
-
-    def __init__(self, service):
-        xmlrpc.XMLRPC.__init__(self)
-        self.service = service
-
-    def xmlrpc_getUser(self, user):
-        return self.service.getUser(user)
-
-
-class FingerService(app.ApplicationService):
-
-    __implements__ = IFingerService,
-
-    def __init__(self, file, *args, **kwargs):
-        app.ApplicationService.__init__(self, *args, **kwargs)
-        self.file = file
-
-    def startService(self):
-        app.ApplicationService.startService(self)
-        self._read()
-
-    def _read(self):
-        self.users = {}
-        for line in file(self.file):
-            user, status = line.split(':', 1)
-            self.users[user] = status
-        self.call = reactor.callLater(30, self._read)
-
-    def stopService(self):
-        app.ApplicationService.stopService(self)
-        self.call.cancel()
-
-    def getUser(self, user):
-        return defer.succeed(self.users.get(u, "No such user"))
-
-    def getUsers(self):
-        return defer.succeed(self.users.keys())
-
-
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, IFingerFactory(f))
-application.listenTCP(80, server.Site(resource.IResource(f)))
-i = IIRCClientFactory(f)
-i.nickname = 'fingerbot'
-application.connectTCP('irc.freenode.org', 6667, i)
-
- -

Here we convert to using Woven, instead of manually -constructing HTML snippets. Woven is a sophisticated web templating -system. Its main features are to disallow any code inside the HTML, -and transparent integration with deferred results.

- -

Use Perspective Broker

- -
-# Do everything properly, and componentize
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic, irc
-from twisted.python import components
-from twisted.web import resource, server, static, xmlrpc, microdom
-from twisted.web.woven import page, widget
-from twisted.spread import pb
-import cgi
-
-class IFingerService(components.Interface):
-
-    def getUser(self, user):
-        '''Return a deferred returning a string'''
-
-    def getUsers(self):
-        '''Return a deferred returning a list of strings'''
-
-class IFingerSettingService(components.Interface):
-
-    def setUser(self, user, status):
-        '''Set the user's status to something'''
-    
-def catchError(err):
-    return "Internal error in server"
-
-
-class FingerProtocol(basic.LineReceiver):
-
-    def lineReceived(self, user):
-        d = self.factory.getUser(user)
-        d.addErrback(catchError)
-        def writeValue(value):
-            self.transport.write(value)
-            self.transport.loseConnection()
-        d.addCallback(writeValue)
-
-
-class IFingerFactory(components.Interface):
-
-    def getUser(self, user):
-        """Return a deferred returning a string""""
-
-    def buildProtocol(self, addr):
-        """Return a protocol returning a string""""
-
-
-class FingerFactoryFromService(protocol.ServerFactory):
-
-    __implements__ = IFingerFactory,
-
-    protocol = FingerProtocol
-
-    def __init__(self, service):
-        self.service = service
-
-    def getUser(self, user):
-        return self.service.getUser(user)
-
-components.registerAdapter(FingerFactoryFromService, IFingerService)
-
-
-class FingerSetterProtocol(basic.LineReceiver):
-
-      def connectionMade(self):
-          self.lines = []
-
-      def lineReceived(self, line):
-          self.lines.append(line)
-
-      def connectionLost(self):
-          if len(self.lines) == 2:
-              self.factory.setUser(*self.lines)
-
-
-class IFingerSetterFactory(components.Interface):
-
-    def setUser(self, user, status):
-        """Return a deferred returning a string"""
-
-    def buildProtocol(self, addr):
-        """Return a protocol returning a string"""
-
-
-class FingerSetterFactoryFromService(protocol.ServerFactory):
-
-    __implements__ = IFingerSetterFactory,
-
-    protocol = FingerSetterProtocol
-
-    def __init__(self, service):
-        self.service = service
-
-    def setUser(self, user, status):
-        self.service.setUser(user, status)
-
-
-components.registerAdapter(FingerSetterFactoryFromService,
-                           IFingerSettingService)
-    
-class IRCReplyBot(irc.IRCClient):
-
-    def connectionMade(self):
-        self.nickname = self.factory.nickname
-        irc.IRCClient.connectionMade(self)
-
-    def privmsg(self, user, channel, msg):
-        if user.lower() == channel.lower():
-            d = self.factory.getUser(msg)
-            d.addErrback(catchError)
-            d.addCallback(lambda m: "Status of %s: %s" % (user, m))
-            d.addCallback(lambda m: self.msg(user, m))
-
-
-class IIRCClientFactory(components.Interface):
-
-    '''
-    @ivar nickname
-    '''
-
-    def getUser(self, user):
-        """Return a deferred returning a string"""
-
-    def buildProtocol(self, addr):
-        """Return a protocol"""
-
-
-class IRCClientFactoryFromService(protocol.ClientFactory):
-
-   __implements__ = IIRCClientFactory,
-
-   protocol = IRCReplyBot
-   nickname = None
-
-   def __init__(self, service):
-       self.service = service
-
-    def getUser(self, user):
-        return self.service.getUser()
-
-components.registerAdapter(IRCClientFactoryFromService, IFingerService)
-
-
-class UsersModel(model.MethodModel):
-
-    def __init__(self, service):
-        self.service = service
-
-    def wmfactory_users(self):
-        return self.service.getUsers()
-
-components.registerAdapter(UsersModel, IFingerService)
-
-class UserStatusTree(page.Page):
-
-    template = """<html><head><title>Users</title><head><body>
-                  <h1>Users</h1>
-                  <ul model="users" view="List">
-                  <li pattern="listItem" /><a view="Link" model="."
-                  href="dummy"><span model="." view="Text" /></a>
-                  </ul></body></html>"""
-
-    def initialize(self, **kwargs):
-        self.putChild('RPC2.0', UserStatusXR(self.model.service))
-
-    def getDynamicChild(self, path, request):
-        return UserStatus(user=path, service=self.model.service)
-
-components.registerAdapter(UserStatusTree, IFingerService)
-    
-
-class UserStatus(page.Page):
-
-    template='''<html><head><<title view="Text" model="user"/></heaD>
-    <body><h1 view="Text" model="user"/>
-    <p mode="status" view="Text" />
-    </body></html>'''
-
-    def initialize(self, **kwargs):
-        self.user = kwargs['user']
-        self.service = kwargs['service']
-
-    def wmfactory_user(self):
-        return self.user
-
-    def wmfactory_status(self):
-        return self.service.getUser(self.user)
-
-class UserStatusXR(xmlrpc.XMLPRC):
-
-    def __init__(self, service):
-        xmlrpc.XMLRPC.__init__(self)
-        self.service = service
-
-    def xmlrpc_getUser(self, user):
-        return self.service.getUser(user)
-
-
-class IPerspectiveFinger(components.Interface):
-
-    def remote_getUser(self, username):
-        """return a user's status"""
-
-    def remote_getUsers(self):
-        """return a user's status"""
-
-
-class PerspectiveFingerFromService(pb.Root):
-
-    __implements__ = IPerspectiveFinger,
-
-    def __init__(self, service):
-        self.service = service
-
-    def remote_getUser(self, username):
-        return self.service.getUser(username)
-
-    def remote_getUsers(self):
-        return self.service.getUsers()
-
-components.registerAdapter(PerspectiveFingerFromService, IFingerService)
-
-
-class FingerService(app.ApplicationService):
-
-    __implements__ = IFingerService,
-
-    def __init__(self, file, *args, **kwargs):
-        app.ApplicationService.__init__(self, *args, **kwargs)
-        self.file = file
-
-    def startService(self):
-        app.ApplicationService.startService(self)
-        self._read()
-
-    def _read(self):
-        self.users = {}
-        for line in file(self.file):
-            user, status = line.split(':', 1)
-            self.users[user] = status
-        self.call = reactor.callLater(30, self._read)
-
-    def stopService(self):
-        app.ApplicationService.stopService(self)
-        self.call.cancel()
-
-    def getUser(self, user):
-        return defer.succeed(self.users.get(u, "No such user"))
-
-    def getUsers(self):
-        return defer.succeed(self.users.keys())
-
-
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, IFingerFactory(f))
-application.listenTCP(80, server.Site(resource.IResource(f)))
-i = IIRCClientFactory(f)
-i.nickname = 'fingerbot'
-application.connectTCP('irc.freenode.org', 6667, i)
-application.listenTCP(8889, pb.BrokerFactory(IPerspectiveFinger(f))
-
- -

We add support for perspective broker, Twisted's native remote -object protocol. Now, Twisted clients will not have to go through -XML-RPCish contortions to get information about users.

- -

Support HTTPS

- -
-# Do everything properly, and componentize
-from twisted.internet import protocol, reactor, defer, app
-from twisted.protocols import basic, irc
-from twisted.python import components
-from twisted.web import resource, server, static, xmlrpc, microdom
-from twisted.web.woven import page, widget
-from twisted.spread import pb
-from OpenSSL import SSL
-import cgi
-
-class IFingerService(components.Interface):
-
-    def getUser(self, user):
-        '''Return a deferred returning a string'''
-
-    def getUsers(self):
-        '''Return a deferred returning a list of strings'''
-
-class IFingerSettingService(components.Interface):
-
-    def setUser(self, user, status):
-        '''Set the user's status to something'''
-    
-def catchError(err):
-    return "Internal error in server"
-
-
-class FingerProtocol(basic.LineReceiver):
-
-    def lineReceived(self, user):
-        d = self.factory.getUser(user)
-        d.addErrback(catchError)
-        def writeValue(value):
-            self.transport.write(value)
-            self.transport.loseConnection()
-        d.addCallback(writeValue)
-
-
-class IFingerFactory(components.Interface):
-
-    def getUser(self, user):
-        """Return a deferred returning a string""""
-
-    def buildProtocol(self, addr):
-        """Return a protocol returning a string""""
-
-
-class FingerFactoryFromService(protocol.ServerFactory):
-
-    __implements__ = IFingerFactory,
-
-    protocol = FingerProtocol
-
-    def __init__(self, service):
-        self.service = service
-
-    def getUser(self, user):
-        return self.service.getUser(user)
-
-components.registerAdapter(FingerFactoryFromService, IFingerService)
-
-
-class FingerSetterProtocol(basic.LineReceiver):
-
-      def connectionMade(self):
-          self.lines = []
-
-      def lineReceived(self, line):
-          self.lines.append(line)
-
-      def connectionLost(self):
-          if len(self.lines) == 2:
-              self.factory.setUser(*self.lines)
-
-
-class IFingerSetterFactory(components.Interface):
-
-    def setUser(self, user, status):
-        """Return a deferred returning a string"""
-
-    def buildProtocol(self, addr):
-        """Return a protocol returning a string"""
-
-
-class FingerSetterFactoryFromService(protocol.ServerFactory):
-
-    __implements__ = IFingerSetterFactory,
-
-    protocol = FingerSetterProtocol
-
-    def __init__(self, service):
-        self.service = service
-
-    def setUser(self, user, status):
-        self.service.setUser(user, status)
-
-
-components.registerAdapter(FingerSetterFactoryFromService,
-                           IFingerSettingService)
-    
-class IRCReplyBot(irc.IRCClient):
-
-    def connectionMade(self):
-        self.nickname = self.factory.nickname
-        irc.IRCClient.connectionMade(self)
-
-    def privmsg(self, user, channel, msg):
-        if user.lower() == channel.lower():
-            d = self.factory.getUser(msg)
-            d.addErrback(catchError)
-            d.addCallback(lambda m: "Status of %s: %s" % (user, m))
-            d.addCallback(lambda m: self.msg(user, m))
-
-
-class IIRCClientFactory(components.Interface):
-
-    '''
-    @ivar nickname
-    '''
-
-    def getUser(self, user):
-        """Return a deferred returning a string"""
-
-    def buildProtocol(self, addr):
-        """Return a protocol"""
-
-
-class IRCClientFactoryFromService(protocol.ClientFactory):
-
-   __implements__ = IIRCClientFactory,
-
-   protocol = IRCReplyBot
-   nickname = None
-
-   def __init__(self, service):
-       self.service = service
-
-    def getUser(self, user):
-        return self.service.getUser()
-
-components.registerAdapter(IRCClientFactoryFromService, IFingerService)
-
-
-class UsersModel(model.MethodModel):
-
-    def __init__(self, service):
-        self.service = service
-
-    def wmfactory_users(self):
-        return self.service.getUsers()
-
-components.registerAdapter(UsersModel, IFingerService)
-
-class UserStatusTree(page.Page):
-
-    template = """<html><head><title>Users</title><head><body>
-                  <h1>Users</h1>
-                  <ul model="users" view="List">
-                  <li pattern="listItem" /><a view="Link" model="."
-                  href="dummy"><span model="." view="Text" /></a>
-                  </ul></body></html>"""
-
-    def initialize(self, **kwargs):
-        self.putChild('RPC2.0', UserStatusXR(self.model.service))
-
-    def getDynamicChild(self, path, request):
-        return UserStatus(user=path, service=self.model.service)
-
-components.registerAdapter(UserStatusTree, IFingerService)
-
-class UserStatus(page.Page):
-
-    template='''<html><head><title view="Text" model="user"/></heaD>
-    <body><h1 view="Text" model="user"/>
-    <p mode="status" view="Text" />
-    </body></html>'''
-
-    def initialize(self, **kwargs):
-        self.user = kwargs['user']
-        self.service = kwargs['service']
-
-    def wmfactory_user(self):
-        return self.user
-
-    def wmfactory_status(self):
-        return self.service.getUser(self.user)
-
-class UserStatusXR(xmlrpc.XMLPRC):
-
-    def __init__(self, service):
-        xmlrpc.XMLRPC.__init__(self)
-        self.service = service
-
-    def xmlrpc_getUser(self, user):
-        return self.service.getUser(user)
-
-
-class IPerspectiveFinger(components.Interface):
-
-    def remote_getUser(self, username):
-        """return a user's status"""
-
-    def remote_getUsers(self):
-        """return a user's status"""
-
-
-class PerspectiveFingerFromService(pb.Root):
-
-    __implements__ = IPerspectiveFinger,
-
-    def __init__(self, service):
-        self.service = service
-
-    def remote_getUser(self, username):
-        return self.service.getUser(username)
-
-    def remote_getUsers(self):
-        return self.service.getUsers()
-
-components.registerAdapter(PerspectiveFingerFromService, IFingerService)
-
-
-class FingerService(app.ApplicationService):
-
-    __implements__ = IFingerService,
-
-    def __init__(self, file, *args, **kwargs):
-        app.ApplicationService.__init__(self, *args, **kwargs)
-        self.file = file
-
-    def startService(self):
-        app.ApplicationService.startService(self)
-        self._read()
-
-    def _read(self):
-        self.users = {}
-        for line in file(self.file):
-            user, status = line.split(':', 1)
-            self.users[user] = status
-        self.call = reactor.callLater(30, self._read)
-
-    def stopService(self):
-        app.ApplicationService.stopService(self)
-        self.call.cancel()
-
-    def getUser(self, user):
-        return defer.succeed(self.users.get(u, "No such user"))
-
-    def getUsers(self):
-        return defer.succeed(self.users.keys())
-
-
-class ServerContextFactory:
-
-    def getContext(self):
-        """Create an SSL context.
-
-        This is a sample implementation that loads a certificate from a file
-        called 'server.pem'."""
-        ctx = SSL.Context(SSL.SSLv23_METHOD)
-        ctx.use_certificate_file('server.pem')
-        ctx.use_privatekey_file('server.pem')
-        return ctx
-
-
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, IFingerFactory(f))
-site = server.Site(resource.IResource(f))
-application.listenTCP(80, site)
-application.listenSSL(443, site, ServerContextFactory())
-i = IIRCClientFactory(f)
-i.nickname = 'fingerbot'
-application.connectTCP('irc.freenode.org', 6667, i)
-application.listenTCP(8889, pb.BrokerFactory(IPerspectiveFinger(f))
-
- -

All we need to do to code an HTTPS site is just write a context -factory (in this case, which loads the certificate from a certain file) -and then use the listenSSL method. Note that one factory (in this -case, a site) can listen on multiple ports with multiple protocols.

- -

Finger Proxy

- -
-class FingerClient(protocol.Protocol):
-
-    def connectionMade(self):
-        self.transport.write(self.factory.user+"\r\n")
-        self.buf = []
-
-    def dataReceived(self, data):
-        self.buf.append(data)
-
-    def connectionLost(self):
-        self.factory.gotData(''.join(self.buf))
-
-
-class FingerClientFactory(protocol.ClientFactory):
-
-    protocol = FingerClient
-
-    def __init__(self, user):
-        self.user = user
-        self.d = defer.Deferred()
-
-    def clientConnectionFailed(self, _, reason):
-        self.d.errback(reason)
-
-    def gotData(self, data):
-        self.d.callback(data)
-
-
-def finger(user, host, port=79):
-    f = FingerClientFactory(user)
-    reactor.connectTCP(host, port, f)
-    return f.d
-  
-class ProxyFingerService(app.ApplicationService):
-    __implements__ = IFingerService
-
-    def getUser(self, user):
-        user, host = user.split('@', 1)
-        ret = finger(user, host)
-        ret.addErrback(lambda _: "Could not connect to remote host")
-        return ret
-
-    def getUsers(self):
-        return defer.succeed([])
-
-application = app.Application('finger', uid=1, gid=1)
-f = ProxyFingerService(application, 'finger')
-application.listenTCP(79, IFingerFactory(f))
-
- -

Writing new clients with Twisted is much like writing new servers. -We implement the protocol, which just gathers up all the data, and -give it to the factory. The factory keeps a deferred which is triggered -if the connection either fails or succeeds. When we use the client, -we first make sure the deferred will never fail, by producing a message -in that case. Implementing a wrapper around client which just returns -the deferred is a common pattern. While being less flexible than -using the factory directly, it is also more convenient.

- -

Organization

- - - -
-from twisted.internet import app
-from finger import FingerService, IIRCclient, ServerContextFactory, \
-                   IFingerFactory, IPerspectiveFinger
-from twisted.web import resource, server
-from twisted.spread import pb
-
-application = app.Application('finger', uid=1, gid=1)
-f = FingerService('/etc/users', application, 'finger')
-application.listenTCP(79, IFingerFactory(f))
-r = resource.IResource(f)
-r.templateDirectory = '/usr/share/finger/templates/'
-site = server.Site(r)
-application.listenTCP(80, site)
-application.listenSSL(443, site, ServerContextFactory())
-i = IIRCClientFactory(f)
-i.nickname = 'fingerbot'
-application.connectTCP('irc.freenode.org', 6667, i)
-application.listenTCP(8889, pb.BrokerFactory(IPerspectiveFinger(f))
-
- - - -

Easy Configuration

- -

We can also supply easy configuration for common cases

- -
-# in finger.py moudle
-def updateApplication(app, **kwargs):
-     f = FingerService(kwargs['users'], application, 'finger')
-     application.listenTCP(79, IFingerFactory(f))
-     r = resource.IResource(f)
-     r.templateDirectory = kwargs['templates']
-     site = server.Site(r)
-     app.listenTCP(80, site)
-     if kwargs.get('ssl'):
-         app.listenSSL(443, site, ServerContextFactory())
-     if 'ircnick' in kwargs:
-         i = IIRCClientFactory(f)
-         i.nickname = kwargs['ircnick']
-         ircServer = kwargs['ircserver']
-         application.connectTCP(ircserver, 6667, i)
-     if 'pbport' in kwargs:
-         application.listenTCP(int(kwargs['pbport']),
-                               pb.BrokerFactory(IPerspectiveFinger(f))
-
- -

And we can write simpler files now:

- -
-# simple-finger.tpy
-from twisted.internet import app
-import finger 
-
-application = app.Application('finger', uid=1, gid=1)
-finger.updateApplication(application,
-   users='/etc/users',
-   templatesDirectory='/usr/share/finger/templates',
-   ssl=1,
-   ircnick='fingerbot',
-   ircserver='irc.freenode.net',
-   pbport=8889
-)
-
- -

Note: the finger user still has ultimate power: he can use -updateApplication, or he can use the lower-level interface if he has -specific needs (maybe an ircserver on some other port? maybe we -want the non-ssl webserver to listen only locally? etc. etc.) -This is an important design principle: never force a layer of abstraction: -allow usage of layers of abstractions.

- -

The pasta theory of design:

- - - -

Plugins

- -

So far, the user had to be somewhat of a programmer to use this. -Maybe we can eliminate even that? Move old code to -"finger/service.py", put empty "__init__.py" and...

- -
-# finger/tap.py
-from twisted.python import usage
-from finger import service
-
-class Options(usage.Options):
-
-    optParams = [
-      ['users', 'u', '/etc/users'],
-      ['templatesDirectory', 't', '/usr/share/finger/templates'],
-      ['ircnick', 'n', 'fingerbot'],
-      ['ircserver', None, 'irc.freenode.net'],
-      ['pbport', 'p', 8889],
-    ]
-
-    optFlags = [['ssl', 's']]
-
-def updateApplication(app, config):
-    service.updateApplication(app, **config)
-
- -

And register it all:

- -
-#finger/plugins.tml
-register('Finger', 'finger.tap', type='tap', tapname='finger')
-
- -

And now, the following works

- -
-% mktap finger --users=/usr/local/etc/users --ircnick=moshez-finger
-% sudo twistd -f finger.tap
-
- -

OS Integration

- -

If we already have the "finger" package installed, we can achieve -easy integration:

- -

on Debian--

- -
-% tap2deb --unsigned -m "Foo " --type=python finger.tpy
-% sudo dpkg -i .build/*.deb
-
- -

On Red Hat [or Mandrake]

- -
-% tap2rpm --type=python finger.tpy #[maybe other options needed]
-% sudo rpm -i .build/*.rpm
-
- -

Will properly register configuration files, init.d sripts, etc. etc.

- -

If it doesn't work on your favourite OS: patches accepted!

- -

Summary

- - - -

Motto

- - - - - diff --git a/docs/historic/2003/haifux/notes.html b/docs/historic/2003/haifux/notes.html deleted file mode 100644 index c35afa8a879..00000000000 --- a/docs/historic/2003/haifux/notes.html +++ /dev/null @@ -1,60 +0,0 @@ -Notes -

Notes

- -

[translated roughly from Hebrew]

- -

Introduction

- -