Changes that caught my eye in Python 3.2

Python 3.2 was released a few days ago. Reading the What’s New document, some of the stuff that caught my eye…

argparse is a new command-line option parsing module that replaces optparse. I found optparse, now deprecated, a pain to use—argparse looks a lot better. argparse can be used in older programs today with a third-party module (E.g. argparse Debian package). I actively avoided adding command-line parsing to my applications because optparse was such a pain—something now much less of a problem.

The new concurrent.futures module seems to be a port of Java’s java.util.concurrent package. I haven’t looked into it yet, but with the GIL, how many people really care about Python threads? The simple examples given can easily be done with Python’s multiprocessing module, for both processes and threads, though without the Java paradigm. The threading module also has a new threading.Barrier class.

WSGI for Python 3 appears to be finalized. I’m looking forward to finally porting my WSGI web applications to Python 3. The email module also received the same makeover that WSGI needed for Python 3.

ElementTree has been updated to v1.3. Not that I care—lxml’s namespace handling is much better (which is to say, it’s just terrible as opposed to near unusable, as is with pure ElementTree).

Quite possibly the most badass addition to Python 3.2: the functools.lru_cache decorator. I use this pattern all the time, but have never gotten around to making it a generic class or decorator I could reuse across applications. With Python 3.2, I no longer need to. The idea: decorate a slow-performing function (where it makes sense) with functools.lru_cache, and Python will instantly and easily memoize the return values of that function making subsequent calls to the function faster.

html.escape() is a simple function that escapes HTML markup for you with the appropriate HTML/XML entities (I forget how I used to do this usually; something in one of the CGI utils, I think? Or maybe stuff in my template engine…).

This is just from a quick reading of the release notes—it’s very likely I missed something. What’d I miss? And what do you like that’s new in Python 3.2?

gpxsplitter: Split GPX files with their waypoints

gpxsplitter splits multi-track GPX files, containing waypoints, into individual one-track GPX files with their respective waypoints.

GPX files containing multiple tracks and waypoints jumbled together are produced on export by many GPS units, particularly MTK chipset-based devices such as the Qstarz Q1000 and Transystem i-Blue 474. Separating tracks and their associated waypoints was a headache until gpxsplitter came along. It’s meant to be run first-thing after downloading data from your unit via gpsbabel or mtkbabel. It’s a quick little script written in Python 2.x, with dependencies on mxDateTime and lxml.

You can get it from the gpxsplitter repository on gitorious, and the GpxSplitter wiki page is the one-stop place that will collect information about it.

I thought about turning this into a web service, where users can upload their GPX files and have them split, but I’d like to know the demand for such a service before writing it. Ideally, gpxsplitter should be part of gpsbabel or something… but yeah, I’ll save the XML parsing in C for a very, very rainy day.

There are probably any number of bugs. If you find one, please let me know—and send a testcase too!

Play WebM in Internet Explorer 9

Update: Google now offers a WebM plugin for Internet Explorer 9, much easier than what I’ve detailed below.

Google’s recent announcement deprecating H.264 for Chrome (see my thoughts on it) means it’s likely that WebM will become the defacto standard for the HTML5 video tag, supported by Internet Explorer 9. Unfortunately, Internet Explorer 9 does not (yet) ship with WebM, despite a lot of misleading PR indicating some kind of “compatibility”.

So, how do you play WebM with Internet Explorer 9?

The easiest way is to use the DirectShow filter pack from Xiph.org. Download and install the installer, available for both 32-bit and 64-bit Windows, and not only will you be able to play WebM/VP8, but also Ogg/Theora, Vorbis, Speex, and FLAC. It’s an royalty-free, open-source standards smörgåsbord!

What do you do next? Of course, submit feedback! Click Send Feedback under Internet Explorer’s Tools menu, and simply ask Microsoft: please support WebM!

Clarification: Don’t install the Support for HTML <video> tag option. It installs an ActiveX control, which requires some extra markup (see the release notes).

Note: Internet Explorer 9 is a beta, as well as Xiph’s DirectShow filters. IE9 doesn’t support a lot of <video> tag features, so many demos out there on the Internet don’t work.

Google Chrome deprecates H.264: the right move, but little change for HTML5 video

Google has decided to deprecate H.264 in Chrome. This is nothing but good for the future of web video. With support in three major browsers (Firefox 4, Chrome, and Opera) it means that WebM/VP8, instead of H.264, will become the defacto codec for HTML5 video.

I’ve talked to several people who think that this move has killed HTML5 video. I’m not sure I follow the logic — little has changed, except what will become the dominant codec.

You can say it’s made Flash the least common denominator, which ignores the fact that Flash already IS the least common denominator for web video.

Regarding codec fragmentation, little is changed there too: Microsoft’s Internet Explorer 9 and Apple’s various Webkit products still do not have WebM/VP8 support. Content providers wanting to support HTML5 still need to encode to both H.264 and WebM.

With the codec fragmentation problem as yet unsolved, do content providers have any reason to use HTML5 video when Flash still is the least common denominator? Well, Flash is no longer included with Windows 7 or Mac OS X (and was never included with any reputable Linux distribution). Are content providers still willing to force users to download plugins, when they can just use the dominant HTML5 video codec?

I don’t have the answers to these questions, nor does anyone else. Nobody said that the problem of open video would be solved easily or overnight. But focusing on WebM is, in my opinion, a step in the right direction.

In the meanwhile, WebM is winning, so why don’t you start encoding your videos to WebM now? On SamatsWiki I’ve a sparse page on encoding to WebM (which will work with stock Debian/Ubuntu tools), as well as one on encoding to Ogg Theora. If you’re on Linux, the easiest way to convert videos is OggConvert, an easy-to-use GNOME-based GUI. Publishing them on the Web is just as easy. Check out the HTML5 video chapter in Mark Pilgrim’s Dive Into HTML5, or Jakub Steiner’s How to get your clips on the web.

Hardware review of the Hewlett-Packard ProLiant N36L Microserver

[flickr-photo:id=5204509633,size=m]

Low-power systems are popular with enthusiasts everywhere. From the Linksys NSLU2 (thoughtfully also known as “the slug”), and the various Marvell SheevaPlug devices, there isn’t a shortage of options. With all of them, however, you need to make compromises—be it having to deal with ARM’s tics, lack of I/O expansion, bad performance, or lackadaisical manufacturers.

If you’re willing to compromise on: size, but still be much smaller than your average PC; power, but also consume less power than your average PC; performance, but still run circles around an ARM-based device—then take a look at the Hewlett-Packard ProLiant N36L “Microserver”. Introduced September 2010, reviews and photos of this system are few and far between. In this article, I review the hardware aspects of the N36L, while in another, I review its software aspects [coming soon].

Internals

[flickr-photo:id=5204528809,size=m]

The N36L is powered by an x86-based AMD Athlon II Neo processor running at 1.3 GHz intended for low-power systems like netbooks. While it has a slower clockspeed, this AMD CPU typically benchmarks faster than Intel’s Atom 1.6 GHz CPU. For the enterprise crowd, the Athlon II Neo is a 64-bit processor and supports hardware-accelerated virtualization and nested paging. This CPU is ideal for partitioning lightly-used services into lightweight VMs. With two DDR3 DIMM slots, the N36L can accommodate up to 8 GiB of RAM.

Graphics is provided by an integrated ATI Mobility Radeon HD 4200 (which also supports GPGPU/OpenCL via proprietary drivers), and the Gigabit NIC is a Broadcom NetXtreme BCM5723.

[flickr-photo:id=5205136774,size=m]

The mainboard provides a respectable amount of expansion. It has two PCIe slots, an x16 (you could easily use a discrete graphics card, though you’d have to be picky about dimensions) and an x1. Adjacent the x1 slot is an x4 slot, supposedly for use with HP’s proprietary management card. You could probably hack a conventional x4 card into the slot, but I rather HP have made the x4 slot usable and used the x1 slot for it’s proprietary add-ons (does a management card really need more than PCIe x1?).

The chassis’ disk racks connect via a mini-SAS connector. There’s one internal SATA connector for the 5.25” bay, but the system’s eSATA connector faces outward so your dreams of easily putting six drives in this tiny system are dashed.

There’s an internal USB 2.0 port, a common feature on servers. It makes running an OS off a USB flash drive that much easier—sequestered internally, such a drive won’t accidentally get knocked off.

Externals

The frontside of the N36L is… “server-like”, whatever that means. Along the top are LED indicators for disk and network activity, as well as the system’s backlit power button. There are four USB 2.0 ports along the right side, and an HP logo that glows blue when the system is on. The chassis door is metal (not plastic!), and has a lock.

[flickr-photo:id=5204512873,size=m]

The backside of the N36L is austere. The only ports: two USB 2.0 ports, one D-sub VGA port, a Gigabit Ethernet port, and one eSATA port. There’s a security Kensington lock slot, as well as an “expander slot” for HP’s proprietary management card. The power supply, fortunately, is integrated (power bricks are a pet peeve of mine), and uses a standard AC power cord.

There are two fans: a 120 mm fan for the system’s main cooling, and a 40 mm fan internal to the PSU. Fortunately, both are quiet; HP rates the system at 21 dB. It’s not silent, but it is quiet. There are no top or side vents; air is drawn in through the front and exhausted out the back.

[flickr-photo:id=5204520585,size=m]

Unlike other PCs, the N36L does not use Phillips-head screws for the user-accessible bits. Two sizes of Torx screws are used (I’m unsure of the size), and HP was pleasant enough to include a Torx screwdriver that snaps into the inside of the machine’s front door. Screws for hard disks and the optical disk drive are also screwed into convenient holes in the front door—no little baggies of screws to lose here! There is a single thumbscrew on the top-back to remove the top cover, and two thumbscrews hold the motherboard plate in place.

[flickr-photo:id=5204523307,size=m]

Other than the handle mechanism which has a metal spring, the N36L’s disk caddies are simple plastic affairs. The plastic does not appear to be particularly high quality, but since the only purpose of the things is to hold disks (and not face the environment), it probably good enough.

How much power does the N36L consume? Using my Kill-a-Watt, I measured 60 W on startup, which settled down to 45 W or so after booting and idling. This unfortunately is a much more than I’d have liked, but with four spinning disks I suppose it’s reasonable.

Cons

I’m not trying to be pessimist by not including a Pros list, but honestly, if you need one at this point you probably don’t need this machine. However, there are some cons I found annoying:

  • Low height clearance for RAM. I found this out the hard way when my heatspreader-equipped DIMMs would not fit
  • In the USA, at least, the N36L ships with 1 GiB of RAM, and either 160 GB or 250 GB disk… Which I immediately tossed for 8 GiB of RAM and four Western Digital 2 TB Green series disks. HP could have easily knocked $50 off the price by not including RAM and disk.
  • No SATA cable included for the 5.25” bay. This is a minor quip, and was probably done to save that last extra $0.50—but it makes the decision to include the RAM and the disk seem that much more strange.

Conclusion

Why did I get an N36L? The short list:

  • It’s x86-based. I didn’t want to muck about with ARM—its benefits for me are few.
  • It can hold four 3.5” disks, and with Gigabit Ethernet functions as a great, inexpensive NAS
  • Does not come with an operating system—yes, you are NOT paying Microsoft’s Windows tax! Also, all of the hardware in the N36L is well-supported by Linux and free software. Most other systems in this class force a Windows Home Server license on you.
  • Cheap. I bought the N36L for $320 USD. It’s more expensive than most ARM-based alternatives, but simultaneously much more powerful.

If you’re looking for photos, see my HP ProLiant N36L set on Flickr. And, if you liked this article, please support this site and consider buying the N36L via affiliate link through Amazon (which only has the 250 GB disk model) or Newegg (which has both the 160 GB model and 250 GB model).

Bing Imagery Misaligned at Lower Zooms

Microsoft’s Bing aerial imagery (that’s recently been donated to OSM for use in tracing) is offset by a few meters in some places.

For example, earlier this week I traced buildings with imagery from USDA’s NAIP program. Besides a history of being well-rectified, a GPS track in the parking lot confirms that I was spot on. However, on Bing with a low zoom, they’re misaligned:

[inline-old:Bing-Misaligned.jpg]

Zoom in a bit, and they magically align again:

[inline-old:Bing-Aligned.jpg]

Apparently, different, well-rectified imagery is used at higher zooms. If I’ve found one problem, it then follows that it exists elsewhere. Be careful when tracing!

High-resolution text console with uvesafb and Debian

While you may rarely use the console on your server, it’s nice to have a high-resolution display just to see that many more columns and rows. Linux’s vesa module (via the vga= parameter) has been around for a while and made this possible, provided you kept up with what VGA mode number to use and don’t mind the spotty hardware compatibility.

While KMS is the way to do this in the future, it doesn’t help us with the drivers and hardware we have now. A new kernel module, uvesafb, mainlined in 2.6.24, is another, new option. In addition to specifying modes in a more user-friendly way (e.g. 1280x1024-32 for 32-bit color, with a 1280x1024 resolution), hardware compatibility is better—in particular, you can now get a high-resolution text console with NVIDIA display adapters.

In the following, I describe how to use uvesafb on Debian and derivative distributions (e.g. Ubuntu). The instructions assume kernel 2.6.27 or higher (Debian 6.0 (squeeze) and Ubuntu 8.10 (Intrepid Ibex), or later).

Read more…

OpenStreetMap “Geolocate me” user script

[inline-old:OpenStreetMap-Geolocate.png]

OpenStreetMap Geolocate is a user script that adds a “Geolocate me” link next to the OpenStreetMap.org search box. If your browser supports it and you’ve granted permission, clicking on this link will center your map window to your location, as reported by your browser via the HTML 5 geolocation API.

Say you’ve taken your laptop to a new cafe or conference—as soon as you open up OpenStreetMap, you can hit the “Geolocate me” link and quickly see what’s around you, without fiddling with search or endlessly dragging the slippy map. Or, better yet, quickly add what’s missing.

This definitely needs to be built into the OpenStreetMap website.

On most browsers, the geolocation API uses Google Location Services or Skyhook, which determine your location based on nearby wireless 802.11 access points. However, some browsers, like Firefox 3.6 on Linux, can talk to gpsd and your GPS unit, so geolocation can get quite accurate.

I’ve tested it on Firefox 3.6, Chrome 5, and Opera 10.60 (which, interestingly enough, is the first non-beta of Opera that supports geolocation). I’ve been told it also works on Safari 5.

I should make a note in the interest of accuracy: geolocation isn’t actually part of “HTML 5”—it’s a product of the W3C Geolocation Working Group. However, the need to be accurate didn’t keep the XML out of AJAX, and by and far geolocation is one of the technologies people think about when they hear HTML5.

This entry is cross-posted on my OpenStreetMap user diary.

Transitioning to a 4096-bit RSA OpenPGP key

I created a new GnuPG key two months ago (see key ID 0x4A456FBA). Now is a good a time as any to publicly announce it. Information for the key:

pub   4096R/4A456FBA 2009-05-08 [expires: 2015-01-01]
      Key fingerprint = E95D 7465 5B35 C5F6 B3B6  68CC 20C6 F0A6 4A45 6FBA
uid                  Samat K Jain 
uid                  Samat K Jain 
uid                  Samat K Jain 
uid                  Samat K Jain 
sub   4096R/8D18D72F 2009-05-15 [expires: 2015-01-01]

All this information (as well as the downloadable public key itself) is available on my CryptoKeys wiki page.

The new key uses 4096-bit RSA keys for both digital signatures and encryption. The change is prompted by questions regarding SHA-1’s viability, detailed by Daniel Gillmore. The concern is not new, as Bruce Schneier reported SHA-1 weaknesses back in 2005. The concerns have simply become worse, and they’re likely to become worse. So much so that the US government’s NIST has recommended the phasing out of SHA-1 by the end of 2010. GnuPG’s maintainers don’t trust SHA-1 either, as upstream GnuPG now defaults to RSA as well.

In this space was a paragraph (or four) describing a little bit more in detail the interaction between encryption algorithms (e.g. RSA, DSA), encryption keys, and hash algorithms (e.g. SHA-1/SHA-160, SHA-512), etc. But as an end-user, I don’t care, and I don’t think other end users need to care either. With encryption, I follow the mantra: use the defaults; more than likely you don’t have a clue what you’re doing if you stray. If you use OpenPGP and use an older DSA-based key (2048-bit RSA is safe), keep in mind there may be issues soon regarding it’s security, and you should switch to DSA-2 or RSA (the new default) instead.

Since SHA-1 hasn’t actually been broken yet, I’ve decided to set an expiration date on my old key (0x1A1993D3), rather than outright revoke it. [

Microsoft’s Hyper-V contribution is not outside their agenda

If you pay attention to Linux-related news, you may have heard that Microsoft has contributed code adding Hyper-V acceleration to the Linux kernel. This event is not something that falls outside of their corporate agenda (whether it falls out of their strategy, I’ll let Steve Balmer voice).

Hyper-V is Microsoft’s hypervisor, included with the server editions of Windows (somewhat similar to VMware Workstation or Sun’s VirtualBox). It lets you run other guest operating systems within the currently running one (called the host OS). Typically, virtualizing guest OSes is slow. To improve performance, rather than virtualizing everything, special drivers and software can be installed into the guest OS to make certain things faster (such as graphics, disk I/O, etc).

The popular Linux hypervisors (Xen, KVM, etc) don’t have special drivers like these for Windows, so they won’t be able to run Windows particularly quickly. With Microsoft’s contribution, Linux now will ship with built-in acceleration for Microsoft’s hypervisor, making Linux run that much faster. If you were an IT shop that simultaneously needed to maximize performance and run both Linux and Windows, would you:

  1. Run an open-source Linux hypervisor, and virtualize Windows (slow)
  2. Run Microsoft’s hypervisor, included with expensive Windows Server licenses, and virtualize Linux (fast)

The answer’s clear. Microsoft’s kernel contribution brings them good PR and satisfies real-world customer demands, while continuing to promote their agenda to make running Windows seem like the best choice. Smart move!