While working on a NNTP interface for the mailing list archives of Mailman, I had to decide which Python networking library I use for the NNTP protocol. There were two obvious candidates: Twisted and asyncore.
Twisted is the big asynchronous python library. It has many features and supports many protocols out of the box. There is even a twisted-news module, that implements a NNTP server. But there a several downsides of Twisted. Because it supports so much protocols and features, it is a big library and would be a huge new dependency of the mailman core (under the assumption, that the NNTP archive will make it into mailman core). It also does not support Python 3 and stated that they does not plan to support Python 3 in near future. I am aware of the fact, that mailman also does not fully support Python 3 at the moment. But Twisted as dependency would be a hard blocker.
In contrast to Twisted, asyncore is contained in the standard library of Python. So currently all versions of Python support it. But it does not have so many features then Twisted:
- Like Twisted (with its reactor pattern) asyncore provides a asynchronous loop to abstract the socket handling. It does that using the select() interface of the OS.
- asyncore also provides a thin wrapper (called dispatcher) for the low-level socket object from Python, that could register in this loop and gets notified, it the status of the socket changes (if for example a connection gets established or date could be read form the socket).
- asynchat is an extension for asyncore (also contained in the standard library). asynchat makes it easy to parse network protocols whose elements are separated by special strings or have a variable length.
One main difference between these two frameworks is the fact, that Twisted already contains a NNTP implementation. I looked over the source code. Mainly it is undocumented and it is even to find some documentation how to use it. The server-side protocol parser does not try to respect the current NNTP RFC in any way. It misses a few commands that stated as mandatory in the RFC (like CAPABILITIES) and also response with wrong response codes in certain situations.
One main advantage of Twisted is, that every protocol could be combined with many frontends (for example local loops, unix domain sockets, ssl sockets). But for the use case of the NNTP archive server, we does not need so many features. asyncore could be easily extended with tlslite to support SSL connections. I am aware of the fact, that tlslite currently does not support Python3, but there are patches available that fixes the outstanding problems (strings as exceptions and usage of hashlib). I think that fixing that problems would be easier than fixing Twisted. asyncore even supports IPv6 out of the box, as it is only a thin wrapper for the socket object. Twisted in contrast to that has an open bug report for more than 5 years now and did not manage to implement it.
During evaluation of the twisted-news module, I notice that they have a test suite. But they are testing very few properties. Only some commands are executed during the test. The main part was just testing the example database for keeping the news entries. But this part of the module was the most irrelevant part for mailman. Mailman has to keep the mails in a separate database, so that different archivers could access it. So the example database of the twisted-news module would not be needed.
During evaluation of Twisted it was very frustrating. Twisted requires a lot of boilerplate code, only to setup a listening socket and the asynchronous loop. The usage of the different factories does not get very clear and the documentation is not very good. asyncore on the other side requires only some code that looks like the default socket binding code in python and the execution of asyncore.loop() and every thing works.
The decision I had to make now was, whether to fix and extend the current twisted-news code or write an own version based on asyncore. Because of the mentioned facts, I decided to write my own NNTP protocol parser with asynchat. With asynchat it was extremely easy to parse the different elements of the NNTP protocol and the code looks now very clean and a lot of easier than the Twisted code (my code contains one method for each command, the Twisted code contains at least three for each one). The code abstracts the parsing from the concrete handling of the commands. Additional to that the code tries to keep as close as possible to the RFC to ensure a wide portability with the clients.
Comments !