I can’t claim to be remotely unusual in this respect though – there’s a whole subculture of programmers (“hackers”) who can relate to this mindset. Sometimes when I build a new tool, I get the impulse to share it with the world in case it turns out to be of use to anyone else. In the past, these hacks have ended up on my software web page, but I think this blog might be a better medium.

So, without further ado, here’s a cute hack I just built: bping, a wrapper around ping(8) which makes it do a lot of beeping – one beep per packet received, with pitch going up an octave for every doubling of the response latency. (See below for installation instructions.) Why did I want this?

I use ethernet over power to connect the machines in my bedroom to the equipment in the lounge. In a high-rise apartment block with about 100 wireless networks fighting over the same spectrum band, this is (normally) a much more reliable option. However yesterday the connections in my home network started behaving very weirdly. I tried pinging various machines on the network from each other to narrow down the problem, but it was annoying to have to keep going between rooms to visually monitor the output from the various pings when it would have been quicker to be able to hear the quality of the connections being tested. So I wrote bping, which also turns a laptop into a sort of Geiger counter for wifi.

Currently bping uses bip (here’s a suggested approach to installation) which in turns uses sox to make the beeps. This works fine on Fedora 15 but unfortunately for some reason takes longer than one second to run on Ubuntu 11.10 and openSUSE 12.1, regardless of how short the beep is. I think it’s something to do with pulseaudio, but I haven’t bothered to figure it out yet. Answers on a e-postcard please …

By the way, many of my hacks are already publically available on my github account, but most aren’t documented yet. The current exceptions are:

• my UNIX shell environment and tools, and
• my utilities and configuration files for git.

I’ll try to document some of the others soon. Watch this space!

So plan B was to set up port redirection from the host to the guest, e.g. so that ssh’ing to localhost port 2222 would redirect to the guest’s port 22.

After a quick google, some fiddling with iptables, and a glance at the libvirt Networking wiki page, I was still having no luck. Then it hit me – my guest was using user-mode networking, and rather than getting its DHCP-allocated IP from the libvirtd-launched dnsmasq instance on the host, was receiving a hardcoded allocation of 10.0.2.15 from the host which is on 10.0.2.2. This can be extremely puzzling at first, because no network commands run on the host (such as ifconfig, iptables, brctl, route) will reveal this magic address, yet the host is still accessible from the guest via it.

After a lot more googling, I stumbled across a technique for configuring host to guest port redirection on a running VM. This sounded very promising, but virt-manager refused to accept the magic Control-Alt-2 key combination to switch to QEMU monitor mode. It turns out that this is no accident. However, since libvirt 0.8.8, the QEMU monitor can be accessed via virsh.
Note that the --hmp option is required, otherwise the monitor expects the command in JSON format, so omitting it leads to errors like error: internal error cannot parse json ... lexical error: invalid char in json text.

The final hurdle was figuring out the correct monitor command. The host_net_redir command as mentioned in the above article is no longer recognized. Luckily the QEMU monitor interface helped me out here – I spotted an encouraging sounding command hostfwd_add:

# virsh qemu-monitor-command --hmp sles11 'help hostfwd_add'
hostfwd_add [vlan_id name] [tcp|udp]:[hostaddr]:hostport-[guestaddr]:guestport -- redirect TCP or UDP connections from host to guest (requires -net user)

and google confirmed that the latter had superceded the former.

So finally we have the complete solution:

# virsh qemu-monitor-command --hmp sles11 'hostfwd_add ::2222-:22'
# ssh -p 2222 localhost
Password:
Last login: Mon Jan 23 00:37:44 2012
linux-mnsh:~ #

Hooray!

UPDATE: just found another very simple solution – add a new NIC to the VM which doesn’t use user-mode networking. Then it will get a IP (on 192.168.100.0/24 by default) which is still NAT’d but also routable via virbr0 on the host, meaning no redirection is necessary; just ssh directly to the guest’s IP from the host. A minor disadvantage of this is that the guest won’t be directly reachable from outside the host, but that’s unlikely to be an issue in most scenarios.

Reaction from the desktop development community has been swift and mostly unapologetic. Mark Shuttleworth, the founder of Canonical, responded “Once again, Linus is underestimating the importance of aesthetics in computing. You only have to look at Apple to see that people place more importance on visual beauty than the kind of efficient work processes that a flexible and reliable desktop environment enable. This is why we’re pushing our new Unity launcher as a mandatory part of Ubuntu. We’re confident that people will quickly overcome the initial shock of everything taking longer to find and access because they’ll be too busy admiring how beautiful it looks.” He then cited the latest 11.10 release as an example of this. “If you look in the release notes for [Oneiric Ocelot], you’ll see a new Alt+Tab switcher at the top of the list of highlights, and below it other radical changes such as renaming ‘Places’ to ‘Lenses’. Frankly, most people lap up this whizzbang shit, and as long as it looks cooler than their friend’s Windows 7 netbook they’ll be willing to tolerate some minor annoyances which are unavoidable when making immature software a critical component of the desktop. Sure, we could prioritise boring bug-fixing over innovation, but that just doesn’t excite the teenagers on the web forums, and we have to think about the next generation of users. Besides, if you want a dumbed down system that mostly works, there’s always Mac OS X.”

Havoc Pennington, a GNOME developer well-known for initiating the war on Linux desktop flexibility by drastically reducing the number of preferences and replacing GNOME’s default window manager, the high-performance scriptable Sawfish, with Metacity, commented: “It’s about achieving the right work/play balance. If your desktop allowed you to get stuff done too quickly, it would just increase your stress levels. Some ‘power’ users think they want to be able to stream-line their workflows, but we know better, so we are doing them a favour by making this customizability harder. After all, everyone needs basically the same things. Rather than trying to be different, these people should instead learn to enjoy the cute visuals and focus more on having fun. Life’s more than just work, work, work, you know.”

The KDE camp has been slightly less vociferous, perhaps because it’s old hat for them – back in 2008 they pioneered the concept of intrusive redesigns and ended up the wrong end of one of Linus’ rants as a result. “With KDE 4.0, we did our best to prevent people achieving real work, and I think we largely succeeded”, one of the KDE team leaders recalls. “I mean, there was a significant period of time where neither the KDE3 version of knetworkmanager nor its KDE4 rewrite worked properly, so for many wireless networks, the only way you could connect was to disable NetworkManager and write a shell-script to interface directly with wpa-supplicant and ifup. And that’s just one small example.”

Despite Torvalds’ comments, the move towards form over function has been witnessed elsewhere outside the desktop software space. For example, Apple have introduced the MacWheel, a move so bold that it makes innovations such as Unity and the GNOME Shell look positively conservative. However there is no clear industry-wide consensus; in fact companies such as Ebay and Sony are beginning to experiment with rejecting both form and function, turning conventional wisdom on its head.

Anyway, in the mean time a quick google brought up the following solution:

• http://www.dgf64art.com/2011/07/05/installing-amazon-mp3-downloader-on-ubuntu-11-04/ (unfortunately this link no longer works)

Unfortunately it doesn’t work – the step which installs the manually downloaded .deb files fails due to broken dependencies. However further googling found a post from 2008 which revealed a technique based on the very useful getlibs utility.

So here’s my solution:

1. Download the 32-bit Amazon downloader app for Ubuntu 9.10.
2. Run sudo dpkg -i --force-all AmazonMP3DownloaderInstall.deb
3. Run sudo apt-get install getlibs if you don’t already have getlibs installed.
4. Run sudo getlibs /usr/bin/amazonmp3 and answer yes to the confirmation.

At this point if you try to run /usr/bin/amazonmp3 you’ll probably hit Ubuntu bug 781870. The workaround is as follows:

export GDK_PIXBUF_MODULE_FILE=/usr/lib32/gdk-pixbuf-2.0/2.10.0/loaders.cache
/usr/bin/amazonmp3

You’ll still get an error that it’s trying to load the 64-bit version of libgvfsdbus.so thanks to Ubuntu bug 369498. I had hopes that export GIO_EXTRA_MODULES=/usr/lib32/gio/modules would fix this, but it seems that this variable only gets honoured too late. However, apparently this issue doesn’t stop the program working so can be ignored.

Another option is to use Banshee’s built-in Amazon downloader, but even without all the politics surrounding Ubuntu’s version of Banshee this didn’t suit my tastes.

UPDATE: Wow. Just found out Amazon doesn’t support re-downloading stuff you’ve already bought. This is truly pathetic, especially considering their Android app kind of implements a locker service. From now on I’ll be using 7digital whenever I can – unfortunately their selection isn’t as big though. The quest for the perfect music services continues … :-/

I’ve been a fan of freeDB for years. It’s a great way of crowd-sourcing CD title/artist/track information and is a huge help when converting CDs into part of your digital music collection (“ripping”).

However, more recently I have noticed that the majority of times I submit a new CD to freeDB, it gets rejected due to a discid collision. This is due to a fundamental limitation in the discid hashing algorithm which freeDB inherited from CDDB – it’s only a 32-bit number, of which a mere 8 bits are used as a checksum for the individual track starting times. So it’s no surprise that we’re getting collisions galore, at an increasing frequency as the database continually grows. Even worse, CDDB attempts to deal with collisions by making CD entries in the database uniquely retrievable by (discid, category) pairs, where category is one of only 11 musical genres. Of course this is woefully inadequate, because there are countless genres and most music defies classification anyway. They attempted to deal with this by calling the 11th category “misc”, but that still has the problem of restricting entries to one unique discid per genre. Unsurprisingly this has caused a huge number of collisions, especially in the “misc” category. As a result, people have been re-submitting collided entries into the wrong genre, simply because having an entry with the wrong genre in the database is still better than not having it at all.

Gracenote, the eventual owner of CDDB have developed a new generation database imaginatively called CDDB2 which adds a much richer meta-data structure. Gracenote has taken advantage of this to clean up the mess caused by attempting to shoe-horn classical CDs into an inadequate schema, and license the results to Apple for iTunes. Unfortunately that’s no use to those of us who recognise the value of freedom over vendor lock-in.

It seems that the freeDB server software hasn’t been updated since 2006, so presumably there’s not much of an active community left. So there’s a ripe opportunity for a smart philanthropist hacker to breathe new life into this valuable project. Sounds ideal for Google Summer of Code task, for instance. As this is largely a lazyweb blog post, here are my thoughts on what needs to be done; it’s unlikely I’ll ever manage to prioritise it above other things already on my plate:

• Design a new collision-proof hashing algorithm. It should produce at least 128-bit hashes, and include as much information about the contents of the physical CD as possible, namely:
  1. number of tracks
  2. starting times of all tracks
  3. total playing time

  This algorithm could be as simple as calculating the MD5 digest of a delimiter-separated concatenation of the above items represented as integers.

  Notice that this should be limited to information which can be retrieved very quickly; for instance producing MD5 digests of the contents of each track takes too long to be useful in practice.

• Design the next level of the CDDB protocol (which at the time of writing would be level 7), which allows additional querying by this new 128-bit (or larger) digest.
• Extend the existing freeDB server software to support this new level whilst remaining backwards-compatible with existing clients. In other words, database entries should be retrievable both by the old (32-bit discid, category) pair and the new digest. This would require iterating once through all existing entries to recalculate the new digest for each.
• (Optional) Extend one or more F/OSS clients to use the new protocol level, and advocate other clients to do the same …

For bonus points, you could extend the database schema in a similar way to CDDB2, and then start a crowd-sourcing project for cleaning up the database with respect to all those pesky classical tracks which have distinct composer / performer metadata.

So, any takers? You’d win the admiration and gratitude of a few, the satisfaction of knowing you helped slightly improve the lives of millions, and a place in heaven

This is absolutely appalling support – all FIVE methods I tried for contacting BT regarding this issue failed:

1. I responded to the email regarding incident number [CENSORED] providing a time and method for calling me as requested, but noone ever called back.
2. I rang 0800 731 0286 and selected option 1 – it rang for several minutes with no answer then hung me up.
3. There is a bug with your login process on btvoip.custhelp.com where the password dialog box is not shown alongside the username dialog box, so I could not log in.  Additionally  requesting a new password failed, as did re-registration.
4. Clicking “Contact Us” on btvoip.custhelp.com does nothing.
5. Clicking “Live Chat” on btvoip.custhelp.com results in the error ”There was a problem connecting to the Chat Server”.

All this is in regard to attempting to switch my landline back from Orange to BT.  Now I am beginning to wonder why I should bother.  At least I can talk to Orange, even if they get everything wrong.

If this ever reaches a human being, please call me as requested in my original email!

This feels good.  Next up is to purge my massive TODO list of out of date entries and then re-evaluate what’s left …

Well, this bad habit has been stressing me out for a LONG time now. I’m a long-term fan of Gretchen Rubin’s Happiness Project, and the other day stumbled across one of her older posts entitled Measure what you want to manage. I’d been wanting to graph the size of my inboxes over time, to get some grip on how bad my backlog actually is, and her post gave me the nudge to actually sort it out.

As usual, Ruby makes it almost ridiculously easy. There’s a beautiful written gem called gmail which uses IMAP to talk to your gmail account. So then it’s just a matter of writing a little program to append a line of gmail folder sizes to a CSV file every time it gets run:

#!/usr/bin/env ruby

require 'pathname'
require 'rubygems'

# sudo gem install gmail
# Note: this is an improved version of Daniel Parker's ruby-gmail
# http://rubydoc.info/gems/gmail/0.3.4/frames
require 'gmail'

# sudo gem install fastercsv
require 'faster_csv'

USERNAME = 'your.account.name.goes.here@gmail.com'
PASSWORD = 'your.password.goes.here'

CSV_FILE = Pathname(ENV['HOME']).join(
  "choose", "a", "path", "to", "gmail-counts.csv"
)

labels = FasterCSV.open(CSV_FILE, 'r').shift[2..-1]

FasterCSV.open(CSV_FILE, 'a') do |csv|
  Gmail.connect(USERNAME, PASSWORD) do |gmail|
    now = Time.now
    counts = [ now.to_i.to_s, now.strftime("%Y/%m/%d.%H:%M:%S") ]
    labels.each do |label|
      unread = label.gsub!(/ unread$/, '')
      folder = gmail.mailbox(label)
      count = unread ? folder.count(:unread) : folder.count
      puts "%s: %d (%d unread, %d read)" % \
        [ label, count, folder.count(:unread), folder.count(:read) ]
      counts << count
    end
    csv << counts
  end
end

Then create gmail-counts.csv in the folder referenced in the script, whose header line contains a list of the labels you want counted (append unread if you want the unread count rather than the total, and prefix “special” gmail folders with [Google Mail]/. Here’s an example of the header followed by a single line of folder counts:

epoch,datetime,INBOX,INBOX unread,[Google Mail]/Drafts,music
1290873125,11/27/2010.15:52:05,555,20,12,1596

Then make sure the program is automatically run on a regular basis somehow. On Linux this is as simple as adding a new line into your crontab. My quietrun utility comes in handy for this:

0 6,12,18 * * * quietrun /path/to/script/count-gmail

Then you can use Google Docs or your favourite spreadsheet / charting application to plot some nice graphs. I used this script which is written in the Ploticus graphing language; here is the result:

Easy!

UPDATE 5 months later! (15th May 2011): I finally made it!

  <profiles>
    <profile>
      <id>unix</id>
      <activation>
        <os>
          <family>unix</family>
        </os>
      </activation>
      <build>
        <plugins>
          <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <executions>
              <execution>
                <id>set-run-file-perms</id>
                <phase>generate-resources</phase>
                <goals>
                  <goal>exec</goal>
                </goals>
                <configuration>
                  <executable>chmod</executable>
                  <arguments>
                    <argument>0755</argument>
                    <argument>${project.build.directory}/foo.sh</argument>
                  </arguments>
                </configuration>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

Dear god. 34 lines and a plug-in, just to change the permissions on a file in a platform-specific way??

I should add that the above was written by an extremely smart guy who is a top-notch programmer; no, I don’t think the author is at fault here. Even if there’s a more concise/portable way of achieving the same result in Maven (and there might well be – I admit I’m still a Maven newbie), there’s still the undeniable fact that XML is horrendously verbose, and any code written in it is by nature unnecessarily difficult to maintain. To this end I applaud the ongoing efforts supporting the use of YAML to implement the Maven POM.

It’s worth seeing what the above would look like if we wrote it in rake:

require 'pathname'

desc "Make binary executable"
task :chmod do
  File.new(Pathname.new(build_dir) + "foo.sh").chmod(0755)
end

I don’t think I need to make a case for which is more legible or maintainable. Oh, and the Ruby version is cross-platform.

To continue an anti-XML rant which has been made countless times already: what the ant and Maven people don’t seem to realise is that XML is not a real programming language and is therefore not expressive enough to deal with many cases that a build system needs. The clue’s in the name, guys! “M” is for “markup” not “Turing-complete“. That’s why every time you need to do something vaguely unusual for which there isn’t an ant taskdef or Maven plugin, you have to write hundreds of lines more Java/XML just to cope with that case. That’s why Maven needs so many damn plugins.

The accidental silver lining to this is that because it takes so much effort to accomplish simple tasks, Maven developers find themselves compelled to reuse and share plugins, and to be fair, Maven has some good ideas on how to do this, even if the implementation isn’t always the best. For example, the built-in plug-in repository management and plug-in dependency management seem to work nicely, but unfortunately for some reason it has a propensity to download plug-ins on most runs, far more frequently than any sensible caching layer should.

DSL issues aside, I’m not convinced by the fixed lifecycle philosophy behind Maven either. I wonder if it was borne out of frustration with the lack of proper dependency checking in ant.

That said, I do like how Maven encourages standardization of the build lifecycle and phase namespace thereof, since newcomers to a project immediately know some familiar entry points. But the same could be said of 99% of projects which use Make and use standard rule target names such as install and clean. And I suspect that many developers suffer when they try to shoe-horn their own project’s build requirements into Maven’s standard lifecycle.

My concern with this phased approach is that it is too linear.  The expectation is that a build process is a one-dimensional sequence of steps, and you get to choose your starting point but not much else.  This seems fundamentally wrong to me.  A build dependency tree is well understood to be a DAG, and any build system which doesn’t model this properly seems to me to be burying its head in the sand.  On the other hand, if it does model it properly, which includes implementing proper dependency resolution, the required build lifecycle should emerge naturally without having to dictate that generate-sources comes before compile which comes before test and so on.

I’ve had some ideas of what the ideal build system looks like, and how to get there from the conventional Java world. More on that soon.

# list all disabled repos
rypper -d

# list all enabled repos with autorefresh off
rypper -e -R

# list all repos which have anything to do with KDE
rypper -x kde

# list priority and URIs for all repos whose alias contains 'home:'
rypper -a home: l -pu

# enable autorefresh on all OpenSUSE Build Service repos
rypper -u download.opensuse.org -R mr -r

# remove all repos on external USB HDD mounted on /media/disk
rypper -u /media/disk rr

I only got one response so I assumed it wasn’t that useful to other people.

However here I am at BrainShare EMEA in Amsterdam surrounded by like-minded geeks, and in a fit of enthusiasm / procrastination, I have finally got round to learning how to build openSUSE Build Service packages, and so proudly (?) present my first OBS package release: the 1-click install version of rypper. Enjoy!

UPDATE 21 March 2011: based on feedback from SLE Product Management, I have submitted a formal request to have zypper extended to include this kind of functionality.

• You can’t sort search results by popularity
• You can’t restrict search results to answered or unanswered questions

Well, that pretty much renders it totally useless right there, for both customers and companies, unless the community you’re searching is tiny, in which case why would you need the site anyway?

Oh yeah, and the undocumented boolean AND searches don’t work.

Staggering.  (I’m praying someone will tell me I’m wrong about at least one of these.)

Update 2010/03/07: got a reply from getsatisfaction on search results sorting – I was right but there is a kind of lame alternative.

Just got a Milestone and am impressed with the hardware but disappointed with Motorola’s attitude towards European customers regarding the closed bootloader. [An official statement from the manager of the technical team behind the MOTODEV program at Motorola] says “This practice is driven by a number of different business factors” but does not explain further – what are you hiding, Motorola? You seem to be underestimating

• (a) the potential of a technology community which promotes collaboration rather than hindering it (why do you think Android grew so quickly in the first place?), and

• (b) the importance of regular, open communication with your customers.

Sadly if this continues, I expect HTC and your other competitors will leave you far behind.

We believe that cyclists should be able to download their power data to the computer of their choice, analyze it in whatever way they see fit, and share their methods of analysis with others.

If only companies like Suunto “got it” rather than shipping miserable excuses for software such as Suunto Training Manager… *sigh*. Anyway, looking forward to getting my media PC fixed and trying this out soon… maybe I’ll even find some time to help improve it

Four hours trying to figure out why the hell my changes to .java source files weren’t showing up in the compiled binaries, debugging an unholy mess of ant XML files, before I finally realised how badly ant’s dependency checking sucks … Then 5 minutes of googling for ‘ant sucks’ and I find two excellently written rants which confirm my discovery (you might need to consult google’s cache for the latter).

I shouldn’t have been surprised.  Anything which uses XML as a domain-specific language should have already set the alarm bells ringing, but my excitement at learning something new initially blotted out the dull headache caused by hacking in XML.  Bah!

Finally started using my CompuTrainer a bit more. Just rode 20km in the lounge with the computer pacing me at 200 Watts. After so much riding with guys way stronger than me, the temptation to draft the computer is difficult to resist. However in the last 500 metres it spontaneously decided to make a sprint for the finish line – the bloody cheek! Obviously I couldn’t allow a bunch of transistors to beat me so I sprinted after it and hit the finish line 0.05 seconds ahead, narrowly avoiding embarassment. Next step is to start doing brick sessions …

By the way, in case anyone googles for computrainer linux, I am actually running it under VMware Workstation on LinuxMCE 0710 which is based on Ubuntu Gutsy. It works great with the unfortunate exception of the Interactive Real Course Videos (IRCV), which seems to be due to missing support in VMware Workstation for VMR9. I’ve posted about this in both VMware Communities and the RacerMate Forum but noone has been able to suggest a solution so far.

UPDATE 15th February 2010 – great news! Check out this fantastic new piece of software called Golden Cheetah!

View Larger Map

View Larger Map

There were other things, like korganizer getting slower and more buggy, phonon not working right, plasma’s bizarre customisation UI and tendency to leak memory like a sieve, a device notifier pop-up which often makes new devices invisible and unclickable … but enough is enough, life is too short. GNOME does what I need (which isn’t much, thanks to the flexibility of openbox).

• A real programming language (PHP) rather than some lame imitation which doesn’t even have proper flow control structures. Of course this also means it’s very easy to set cookies, use CGI parameters etc.
• More more out-of-the-box flexibility and functionality on the backend
• Unbelievably easy to set up on my own webserver
• Over 4000 plugins! including ones for SyntaxHighlighter and Google Analytics
• Countless themes, many of which support fluid width content columns.

So I’m totally sold; now I just have to make different RSS feeds available for my different audiences (none of which exist yet…)

1. In MyTracks, copy the track to SD card as a KML file
2. Plug the phone in via USB and mount the SD card’s filesystem
3. Copy the KML file from the kml directory to a local directory.
4. Open the KML file in an editor, search for 7f0000ff (light orange) and change it to ffcc66cc (purple) or something more visible
5. Ensure the track name is correct
6. Go to http://maps.google.co.uk/ (ensuring I’m signed into my google account)
7. Click “My Maps”
8. Click “Create new map”
9. Click “Import” and upload the KML file
10. Edit the route if necessary and save changes.
11. Sometimes it’s necessary to click on another route and back to this new one, in order to make the “View in Google Earth” link appear.
12. Copy the URL of the “View in Google Earth” KML link and paste it into the Google Maps *search* bar (yes that’s right, *not* your browser address bar)
13. Click “Link” near the top right
14. Share the link with friends and/or paste the embeddable HTML into a new blog post.

The copy of the “View in Google Earth” KML link back into the search bar in step 12 is necessary to stop Google Maps from splitting the route up into multiple pages, which would only show shorter segments (“lines”) of the overall route at one time.