Python

Changes that caught my eye in Python 3.2

Samat Jain — Wed, 23 Feb 2011 01:30:03 +0000

Python 3.2 was released a few days ago. Reading the What’s New document, some of the stuff that caught my eye…

argparse is a new command-line option parsing module that replaces optparse. I found optparse, now deprecated, a pain to use—argparse looks a lot better. argparse can be used in older programs today with a third-party module (E.g. argparse Debian package). I actively avoided adding command-line parsing to my applications because optparse was such a pain—something now much less of a problem.

The new concurrent.futures module seems to be a port of Java’s java.util.concurrent package. I haven’t looked into it yet, but with the GIL, how many people really care about Python threads? The simple examples given can easily be done with Python’s multiprocessing module, for both processes and threads, though without the Java paradigm. The threading module also has a new threading.Barrier class.

WSGI for Python 3 appears to be finalized. I’m looking forward to finally porting my WSGI web applications to Python 3. The email module also received the same makeover that WSGI needed for Python 3.

ElementTree has been updated to v1.3. Not that I care—lxml’s namespace handling is much better (which is to say, it’s just terrible as opposed to near unusable, as is with pure ElementTree).

Quite possibly the most badass addition to Python 3.2: the functools.lru_cache decorator. I use this pattern all the time, but have never gotten around to making it a generic class or decorator I could reuse across applications. With Python 3.2, I no longer need to. The idea: decorate a slow-performing function (where it makes sense) with functools.lru_cache, and Python will instantly and easily memoize the return values of that function making subsequent calls to the function faster.

html.escape() is a simple function that escapes HTML markup for you with the appropriate HTML/XML entities (I forget how I used to do this usually; something in one of the CGI utils, I think? Or maybe stuff in my template engine…).

This is just from a quick reading of the release notes—it’s very likely I missed something. What’d I miss? And what do you like that’s new in Python 3.2?

Topic:

Programming

Python

gpxsplitter: Split GPX files with their waypoints

Samat Jain — Wed, 16 Feb 2011 06:54:52 +0000

gpxsplitter splits multi-track GPX files, containing waypoints, into individual one-track GPX files with their respective waypoints.

GPX files containing multiple tracks and waypoints jumbled together are produced on export by many GPS units, particularly MTK chipset-based devices such as the Qstarz Q1000 and Transystem i-Blue 474. Separating tracks and their associated waypoints was a headache until gpxsplitter came along. It’s meant to be run first-thing after downloading data from your unit via gpsbabel or mtkbabel. It’s a quick little script written in Python 2.x, with dependencies on mxDateTime and lxml.

You can get it from the gpxsplitter repository on gitorious, and the GpxSplitter wiki page is the one-stop place that will collect information about it.

I thought about turning this into a web service, where users can upload their GPX files and have them split, but I’d like to know the demand for such a service before writing it. Ideally, gpxsplitter should be part of gpsbabel or something… but yeah, I’ll save the XML parsing in C for a very, very rainy day.

There are probably any number of bugs. If you find one, please let me know—and send a testcase too!

Topic:

Python

OpenStreetMap

Python-like tuple unpacking for PHP

Samat Jain — Wed, 29 Oct 2008 21:55:47 +0000

Python provides a neat way for functions to return multiple arguments via “tuple unpacking”. For example:

def blah:
  return ('one', 'two')

rval_1, rval_2 = blah()

The same can be done in PHP relatively easily via the list construct:

function blah()
{
  return array('one', 'two'); }

list($rval_1, $rval_2) = blah();

Topic:

Programming

Python

PHP

GPG public key signing post-party automation with KMail

Samat Jain — Mon, 19 Feb 2007 03:24:32 +0000

This past Ubucon’s key signing party was my first key signing party. One thing I noticed–signing keys after a key signing party is a boring repetitive task. Summarized from the Ubuntu wiki entry on typical key signing post-party protocol:

Retrieve all public keys of key signing party participants, using gpg –-recv-key
Compare the hardcopy fingerprint from the keysigning party to the fingerprint of the retrieved public keys, using gpg –-fingerprint
Sign the key, using gpg –-sign
Send the signed key back, either by
- E-mail: export the key, then e-mail it to the key owner, using gpg –-export -a | mail -s “Your signed key” user@example.com
- Key server: send the key to a public keyserver, using gpg –send-keys

This is incredibly monotonous—and people have to wonder why Web of Trust-based encryption is not more popular?

The Debian signing-party package provides the utility caff to automate some of this. It’s not very friendly to “desktop” users, however:

it’s a CLI application
it requires a local MTA (/usr/sbin/sendmail in particular), or an “open” SMTP server, with no support for authenticated SMTP or SMTP/SSL
the configuration file syntax is Perl and confusing; there are also few examples on the Internet

You could add authenticated SMTP or SMTP/SSL support to the script, but having to know how to hack Perl definitely disqualifies caffe from being a desktop-friendly application.

So, I hacked together my own key signing party script in Python that would send signed keys back to people via KMail. To use it, create a text file listing all key IDs you wish to sign, one per line. Pipe the contents of this list into the script:

cat list-of-ids.txt | key-signing-party-batch-process-via-kmail.py

The script will download each key, ask you to verify the fingerprint, and then sign it. It then will open a KMail composer window, pre-filled with the key owner’s e-mail address, a friendly template message (customizable in the script), and attached key. Review each e-mail to make sure it is kosher, and click send. Other than continuing to be a CLI program, I think this is much friendlier–the only manual work done is the creation of list of keys to sign, comparing fingerprints (this could be automated, but it seems in the spirit of the Web of Trust-based systems not to), and clicking send in a familiar desktop e-mail client.

Now for some notes…

It uses the DCOP automation features of KDE’s Kmail to send messages. You could similarly use Evolution and D-Bus, but I don’t use Evolution so I can’t contribute that bit of functionality. Mozilla’s Thunderbird unfortunately does not yet support any kind of automation features (as far as I know, anyway), so you’re completely out of luck if you use it.

DCOP with Python is a complete, utter, pain. The easy way to drag-and-drop boiler-plate code with kdcop did not work, as it appears the APIs have changed. A problem with KDE/Python dcopext’s module and multiple identically-named-functions sealed the deal for me and I gave up trying to use DCOP with Python, and instead settled for a hack of using the shell instead. I’m looking forward the one Linux desktop IPC protocol to rule them all, D-Bus, to debut in KDE4.

My script does not provide all the functionality of caffe. It, for example, does not encrypt the messages and their keys back to their owners. There doesn’t appear to be an easy way to do this with KMail and DCOP, so it’s a feature that will have to wait.

Topic:

T-Mobile WiFi Hotspot login script

Samat Jain — Sat, 29 Jul 2006 06:53:05 +0000

T-Mobile’s WiFi Hotspot service, thankfully, forgoes a proprietary authentication mechanism for a solution that while cross platform (i.e. it works with Linux), can be extremely annoying. On opening a web browser and attempting to go to any website, you’re required to login on an SSL-protected website with your account username and password before you can use the connection. If your web browser automatically tries to open many pages on startup, such as when you’re using the Session Saving extension for Firefox, you get T-Mobile’s Hotspot login page in every tab–extremely annoying!

I’ve written a small Python script that can login programatically without use of a web browser.

Topic:

Linux

Python

Wireless

A quick shell include for setting paths for programs installed in non-traditional locations

Samat Jain — Wed, 28 Jun 2006 15:34:05 +0000

A page in the Beyond Linux from Scratch manual describes environment variables that should be set when installing software in a non-traditional location (e.g. your home directory).

I've written a sh/bash include that can be included from .bashrc to set these variables, as well as PYTHON_PATH for separately installed Python libraries:

#!/bin/bash

PREFIX=$HOME/usr

export PATH="$PREFIX/bin:$PATH"
export PYTHONPATH="$PREFIX/lib/python2.4/site-packages:$PYTHON_PATH"
export MANPATH="$PREFIX/man:$MANPATH"
export INFOPATH="$PREFIX/info:$INFOPATH"
export LD_LIBRARY_PATH="$PREFIX/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="$PREFIX/lib/pkgconfig:$PKG_CONFIG_PATH"
export CPPFLAGS="-I$PREFIX/includes $CPPFLAGS"
export LDFLAGS="-L$PREFIX/lib $LDFLAGS"

Topic:

Linux

Python

Bash Shell

A take on Drupal's taxonomy system from the Plone/Python camp

Samat Jain — Fri, 02 Jun 2006 10:23:13 +0000

Over at the Plone Blog is the article Death and Taxonomies; it reviews Drupal’s taxonomy (aka category) system.

The author basically comes to the conclusion I have, a conclusion that is shared by many in the Drupal community but not so much outside of it: Drupal’s taxonomy system is amazing.

For most people and their uses, it is completely overengineered and complicated to use. This fits into Drupal’s marketing stance that it is a content management framework rather than just a CMS. It is very generic, and with some custom programming can be adapted to anything–the possibilities are limitless. With many web development projects (that I don’t want to write in PHP) I think about, I wish I had the facility of Drupal’s taxonomy system.

Of course, there are problems, which the review goes into: there are too many hierarchal relationships in Drupal, all competing with each other. There is the menu system, the book module, and hierarchal taxonomies. The key to being a Drupal master is know when to use which and how to use them, something I’ve definitely not mastered. And that is part of the problem–why should you need to?

Topic:

Python

Drupal

Calculating bandwidth from a combined-format web server log

Samat Jain — Thu, 25 May 2006 09:03:21 +0000

Given a combined web server access log, such as the ones generated by Apache, it can be useful to know the total amount of data transfer of all requests in that log. This task is simple: extract the field listing the number of bytes sent for a request, and add them all up. For something so simple, there is an odd lack of examples or pre-made scripts that do this. Or, at least, I couldn’t find any.

I wrote my solution, calculate-data-transfer.py, in Python:

Topic:

Python

Apache

A better way to separate Apache log files by virtual host domains

Samat Jain — Tue, 23 May 2006 08:02:00 +0000

Apache’s “combined” log format is one the most common log formats used in access logging, containing useful fields such as referrer and user agent. Unfortunately, it does not contain a field listing the the virtual host for whom a request was formed. With Apache, this is easily rectified by defining a custom logging format and post-processing logs to maintain compatibility.

Topic:

Python

Apache

Python 2.5 alpha released

Samat Jain — Thu, 06 Apr 2006 00:00:00 +0000

Python 2.5’s first alpha has been released. From a quick look at the What’s New in Python 2.5 document, what I thought was interesting…

Topic:

Programming

Python

Looking at Python web development frameworks

Samat Jain — Mon, 25 Jul 2005 20:05:00 +0000

I’ve a lot of interactive websites I’ve been wanting to do lately. After learning Python earlier this year, I’m wanting to write all these projects in Python versus something conventional like PHP. I’ve come to the conclusion that web development with Python sucks; it’s no wonder it’s not being used very much. There are too many frameworks, many doing similar things in different ways, or doing totally dissimilar things in totally different ways. I’ve been spending the last week reviewing several different frameworks and deciding which ones I thought were best.

Topic:

Programming

Python

Splitting a Sequence into Subsequences with Python

Samat Jain — Sun, 05 Jun 2005 22:50:00 +0000

I’m writing a load balancing algorithm for this cluster I’m working on, and have a bunch of things in a Python list sequence. So came the issue, how to divide the list into sublists for each node in the cluster to compute? I came up with the algorithm:

def split_seq_stride(l, n):
  """Splits a sequence l into a list of subsequences containing at least n
  elements each, not preserving order. The first few subsequences may contain
  n+1 elements, containing the last few elements. (Algorithm is good for load
  balancing)"""
  r=[]
  k=len(l)/n
  [r.append(l[i::k]) for i in range(k)]
  return r

which behaves like:




  
    
      l = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
      print split_seq_stride(l, 3)
      [[1, 4, 7, 10], [2, 5, 8], [3, 6, 9]]

Assuming each task is an equal amount of work, this algorithm is a good load balancing algorithm. it prevents any node from having to do any less significant work than the others.

Probably useful for someone: what about splitting up a sequence in-order, with the last subsequence containing fewer elements? This code segment does just that:


def split_seq(l, n):
  """Splits a sequence l into a list of subsequences containing at most n
  elements each, preserving order. The last subsequence may contain less
  than n elements."""
  r=[]
  [r.append(l[s:s+n]) for s in range(0, len(l), n)]
  return r

which behaves like:




  
    
      l = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
      print split_seq(l, 3)
      [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

Thanks to yason and others on EFnet’s #python for pointers.

Topic:

Programming

Python

High Performance Computing