Worked All States – by my standards, at least

So I’ve managed to talk to at least one person in every US state. This is Quite A Big Deal for a new(ish) ham. To be more specific, I’ve done this using one mode – PSK31 – which makes it slightly better nerd-tuned.

But you’re going to just have to take my word for it now. In order to get a certificate, I have to get the 50-ish responds to log their details either on eQSL or ARRL’s Logbook of the World. And this is a bit more difficult.

On eQSL, I’m at 49/50. My lone North Dakotan QSO was from someone mobile from out of state, and eQSL doesn’t handle that well. North Dakota is very sparsely populated, and there’s very seldom anyone on the air from there. Every time I look on PSK Reporter, the emptiness of the state glares at me …

LotW is more of a problem. It says I still have ten more states to go. While it’s a very elegant system, the setup process for LotW is just a bit too complex for most people.

So here are my maps:

VA3PID Worked All States
VA3PID Worked All States: West
VA3PID Worked All States: Central
VA3PID Worked All States: East

I’d like to apologise to Vermont, whose presence is rather noted by its absence from the map. I’m actually surprised that more states ended up with all of their QSOs appearing inside, as:

  • Towns tend to be by rivers, so that’s where you’ll find more hams
  • Rivers often mark state boundaries
  • Ham locations are given as grid squares, which are a few kilometres across
  • Rivers are wiggly, and don’t respect arbitrary boundaries.

Absent making a blocky, 8-bit like political map, we’ll have to make do with these failings sometimes.

keep it short, keep me sweet

Most of my radio communications use a text mode called PSK-31. It’s popular and survives a lot of interference, but fast it isn’t. I tested sending 1000 random characters, and it took 5′ 33″. That’s about this speed:

Because the mode is so slow, I prefer to keep my transmissions and macros short, stopping just shy of Morse’s telegraphically incomprehensible VY FB CPY OM. My heart sinks when I’m in a QSO, and I see the beginning of a huge piece of macro text being fired at me. A one-way long conversation (like the dude who sent the entire wikipedia article about his hometown) isn’t a ragchew, it’s a barrage. If someone started wittering on at you for 15 minutes without a chance for you to say anything, no-one would want to talk to them. So please, check your macros, think before you type, and we’ll have more and more pleasant contacts.

QRM Cat

Was having a nice chat with Gary, WB0RUR, this morning on 40m PSK-31. His signal was clear, he’s a very experienced operator — yet he was jumping all over the waterfall with every transmission. I couldn’t understand why, but after his signal jumped, stopped, started, moved again, I caught a very brief TX: “QRM … cat

Gary explained:

“Sorry about that … he stepped on the keyboard and stopped my transmission and also bumped the VFO … so I’m probably moving all over frequency.”

QRM would be a great name for a cat.

pdf qsl cards for all!

I’m right chuffed that people have started sending me PDF QSL cards in return. Here are just a few samples:

(Okay, so Federico’s was a JPEG; near enough …)

Thanks to all who sent them.

I ♥ 10m

And this is why I do ham radio:

Worked a pileup on UA0FO, Sergey on Sakhalin. Yeah, those islands just north of Japan … y’know, the whole part of the world my antenna doesn’t even point near.

(picture from Philip Gladstone’s entirely awesome PSKReporter)

Chirp, the universal HT programmer

CHIRP is rather good. It replaces all the horrid proprietary HT programming software with one cross-platform, cross-radio solution. It allowed me to program my new Kenwood from data from my Wouxun. It uses transparent file formats, and can import from everything. It’s great!

Update: Whoa! It can now program my FT-857D! I just uploaded all the repeaters within 50km, and there’s a bunch going on on a few of them.

24 hours after sending 286 digital QSL cards …

Of the 286 secure digital QSL cards I sent, 8 were undeliverable. I’ve had 25 responses so far:

  • Fourteen positive
  • Three wondering if I hadn’t seen their confirmation on eQSL/QRZ/LoTW
  • Two that don’t accept QSLs of any kind
  • One language issue; I have “Report” when “RST/RSQ” might be clearer internationally
  • One completely neutral
  • One auto responder
  • One with no record of a QSO in their log
  • One saying their callsign has been hijacked and that they are no longer an active ham
  • One negative

Of the positive ones:

  • Two have also sent real QSL cards (thanks, guys, but really, you shouldn’t have …)
  • Two asked how I did it.

Pretty good and quick response, I thought.

Secure digital QSL cards, part 2: shell script

Following on from Creating secure digital QSL cards with your LoTW certificate, here’s a Bash script to generate encrypted signed PDF QSLs. You will need to edit the certificate file name, the QSL blank file name, your call sign, your LoTW password and the PDF encryption password. After doing so, please keep the script safe, as whoever has your LoTW password can pretend to be you.

The only checks that this script doesn’t do (and probably should) are if you have pdftk and PortableSigner executables in your path. PortableSigner is rather weird the way it runs; you need to specify full paths for all files, or it dies.

The script is called like this:

mkdqsl.sh callsign date utc mhz mode report

for example:

mkdqsl.sh VE3KPK 2011-10-02 2341 7.03581 CW 499

Code below the fold.

Creating secure digital QSL cards with your LoTW certificate

Hams have sent out QSL cards since the hobby began. It may be less popular than it used to be, because mail seems slow now (especially when QSL bureaux are used), and there are online alternatives like eQSL, QRZ.com, and the ARRL‘s Logbook of the World (LoTW).

LoTW has been criticized for being too complex. It uses an X.509 public key signature to verify your submitted log entries, and a QSL is confirmed when the other party sends in a matching entry. In a way, it’s rather elegant, as the LoTW server does the work of matching the entries, so QSLs “magically” appear in your log. You don’t get a fancy QSL card in the mail or on your screen, and I think a lot of users miss that.

X.509 might be a bit unweildy, but props to ARRL for setting up a industry standard, robust (-ish; X.509 has its flaws), general purpose signing infrastructure. Since other file formats — notably PDF — support X.509 signing, you can use your LoTW certificate to make other data tamper-evident.

Here I document a method of creating a digital QSL card that can be e-mailed, and subsequently verified by the recipient as being legitimate. Any alterations to the file will break the signature, and the file will just appear as a regular PDF (or not display properly). The process can be used to sign any other Acrobat file. There are probably more streamlined ways to do this, but I only came up with it last night as the beginning of a scriptable solution.

You will need:

  • Your ARRL LoTW electronic certificate file. This is called callsign.p12; mine is VA3PID.P12
  • A means of making a QSL card as a PDF fill-in form. I used Scribus; it’s a free but powerful DTP system.
  • Pdftk, “the” PDF toolkit, to apply the QSL details to the PDF form. If you’re on Windows, you might want the Pdftk GUI.
  • PortableSigner, a java application for signing Acrobat files using an X.509 certificate.

(Quite a bit of what follows was learnt from the two pages Your first PDF form with Scribus – Scribus Wiki and filling in forms with pdftk, so thanks to the authors of those for the guidance.)

First, make your QSL card. Since you’re not going to print this, it can be any size you want, but postcard size is standard. At the very least, create spaces for the recipient callsign, the date, the time, frequency, mode used, and signal report.

Under these headings, I’ve made six PDF text form fields. Scribus creates form fields like text frames/boxes. I used plain text fields (which are selected by this icon: ), centred text, and with the name of the field set from the PDF Options → Field Properties context menu. Each field needs a different name. I used callsign, date, utc, mhz, mode, and report.

Save your QSL card as PDF. It might be an idea to check it to see if the form fields are really there and editable:

Now it starts to get really nerdy. Adobe specified the Forms Data Format (FDF) to allow PDF form data to be slung around. FDF looks a bit like PostScript or raw PDF:

%FDF-1.2

1 0 obj
<<
/FDF << /Fields 2 0 R>>
>>
endobj
2 0 obj
[<< /T (callsign) /V (VE3KPK) >>
<< /T (date) /V (2011-10-02) >>
<< /T (mhz) /V (7.03581) >>
<< /T (mode) /V (CW) >>
<< /T (report) /V (499) >>
<< /T (utc) /V (2341) >>
]
endobj
trailer
<< /Root 1 0 R >>
%%EOF

If you think of the T & V values above as Tag and Value, you can see that the file defines callsign=VE3KPK, date=2011-10-02, mode=CW, and so on. This step can be easily scripted. If you’re not sure what the fields are called, pdftk has a “dump_data_fields” option that spits out the field names as plain text.

pdftk is also used to put the data from the FDF file into the PDF template. It’s a slightly hairy command line:

pdftk QSL-blank.pdf fill_form VE3KPK.fdf output QSL-VE3KPK.pdf flatten

Here the source file is QSL-blank.pdf, the FDF data is VE3KPK.fdf, and the output is to QSL-VE3KPK.pdf. The flatten option turns the pdf form into regular, uneditable PDF. (These details are from a real QSO, by the way, and by utter coincidence I’m writing this in a hotel in Ken VE3KPK’s home town in Northern Ontario.)

Checking that this worked in Adobe Reader:

For many people this is probably enough (or perhaps, too much already!) but I really want to have a digital QSL card that will stand up to some scrutiny. This is where your LoTW certificate file comes in.

PortableSigner is a java application for signing PDF files. It seems quite happy signing the files made in this workflow. It can run from the command line, or as a windowed application:

You use your .p12 signature file and its password to sign the PDF. Once the file is signed, you can send it to your contact, and they can prove (and hopefully, any certification/contest agency will agree) that the contact was confirmed.

Viewing the signed document is deeply unimpressive:

It’s not such a big deal that Reader says that “the validity of the certification is unknown”. It’s just that Adobe doesn’t have the ARRL’s certificate loaded into everyone’s reader (what!? mock outrage!) and so it doesn’t match a certificate it knows about. You have to dig a bit deeper into the signature panel to check out who is responsible for this.

Well, that’s a start; at least it was signed by someone with my e-mail address. There’s more under Show Signature Properties:

There’s the ARRL signature in there. Buried deep in another properties tree is my callsign; can’t find it today, but I saw it in there last night. Either way, the digital QSL PDF is now signed and certified that it came from me, as an ARRL LoTW user.

K3NG Keyer complete

I finally build K3NG’s Arduino CW Keyer and put it in a nice box. Here’s how it looks:

K3NG Arduino keyer, complete

That’s a SparkFun Arduino Project Enclosure with two buttons (one command, one macro), a CW speed control potentiometer, and a simple 3.5mm audio jack for keying the transmitter.

Enclosure base, showing piezo and potentiometer

I’ve glued a cheapo piezo (UHU All Purpose glue is my piezo glue of choice; cyanoacrylate is too brittle) onto the base, and cut a hole in the side for the speed pot. The piezo gives a clear enough side tone that I can copy CW (or Hellschreiber, as before) with audio output into Fldigi. The beeper’s got a fierce third harmonic, but that’s part of its charm. K3NG has a more complex speaker circuit, but this is simple and self-contained.

Enclosure lid, with buttons and resistors

Two momentary switches ($2 from Active Surplus) make up the control interface.

Arduino+Protoboard, and the magic of a 2N2222 switch

The clever bit is an Arduino Duemilanove Uno (my oldest board; it deserves a proper useful home which doesn’t seem to have the serial startup problems my Duemilanove had) with an Adafruit Proto Shield on top. The only “clever” componentry on that is a solitary 2N2222 switching transistor.

It works pretty well. The only thing that doesn’t seem to be stable is the memory button; it seems to choose randomly from any of the first four memories, so I might accidentally send an SK when I meant a CQ. For now, until I work out what’s wrong, I’ll stick to keyboard input of the macros.

K3NG Arduino Keyer

I’m pretty amazed that the above image is even vaguely readable. It’s Hellschreiber, generated by Anthony K3NG Good’s Arduino CW Keyer. What you’re seeing, though, is Hellschreiber from the keyer’s sidetone generator being fed through a piezo glued to a paper cup (and not just any paper cup) being picked up by Fldigi on my laptop’s microphone. This isn’t what you’d call a quality signal path, and it’s a tribute to the mode’s robustness that it can be made out at all.

Anthony has packed an absurd amount into this keyer. There isn’t enough memory on a stock 32K Arduino for all the features to be enabled. I’m planning to use it as a CW keyer alongside Fldigi as the decoder. Despite all the features that can be built in, I’d just be using it as a serial to Morse converter, with perhaps a couple of memory keys for calling CQ and the like.

I do have a slight problem with it while it’s breadboarded, though. The wiring’s so sensitive that the control circuit triggers if I put my hand near it, let alone touch the command button. I’ll have to do something about that. I can’t breadboard for toffee.

Ham Radio Talk links

The ham radio talk went quite well last week; the usual TLUG (and Perl Mongers) suspects plus a couple of knowledgeable hams. For some reason, LibreOffice didn’t format the links correctly, so here they are as real links:

analogue photo transfer over radio

Last night I picked up a weird signal on 14.074 MHz that wasn’t quite Olivia, and wasn’t quite Contestia. After a while I worked out it was MFSK16; but only after I started calling CQ nearby in Olivia and the original caller, Don (W5VRX) in Arlington, TX, told me what it was.

Don told me that MFSK could send pictures, and that Fldigi supported it well. It’s a bit slow, can be subject to noise, but you can get quite clean results. Here’s what Don sent me:

Slightly Soggy Field Day

Scarborough Amateur Radio Club set up in Heber Down yesterday for Field Day. We could have had better weather.

GOTA table
Joy works PSK31 on 15m
Calling CQ on 40m
Chris sets up
The Chrismobile!
Field at Heber Downs
Field at Heber Downs

All the bands were humming last night. Unfortunately, all the field day stations only wanted to talk to other field day stations, so I got precisely 1 contact all night. Even got a promising signal from Azerbaijan wiped out by an FDer over-eager for ARRL brownie points.

Parsing ADIF with Perl

In ham radio, we’re plagued with a data log standard called ADIF, the Amateur Data Interchange Format. It certainly is amateur, in the bad sense of the word. It looks like someone once saw SGML in a fever dream, and wrote down what little they remembered.

Anyway, the following Perl snippet will parse an ADIF file into an array of hashes. It was based on some code from PerlMonks that kinda worked. This works for all the file (singular) I tested.

#!/usr/bin/perl -w
# modified from perlmonks - TedPride - http://www.perlmonks.org/?node_id=559222
use strict;

my ( $temp, @results ) = '';

### Fast forward past header
while (<>) {
  last if m/<eoh>\s+$/i;
}

### While there are records remaining...
while (<>) {
  $temp .= $_;

  ### Process if end of record tag reached
  if (m/<eor>\s+$/i) {
    my %hash;
    $temp =~ s/\n//g;
    $temp =~ s/<eoh>.*//i;
    $temp =~ s/<eor>.*//i;
    my @arr = split( '<', $temp );
    foreach (@arr) {
      next if (/^$/);
      my ( $key, $val ) = split( '>', $_ );
      $key =~ s/:.*$//;
      $hash{$key} = $val unless ( $key eq '' );
    }
    push @results, \%hash;
    $temp = '';
  }
}

# example: just pull out CALL and GRIDSQUARE for each record that has them
foreach (@results) {
  next unless exists( $_->{GRIDSQUARE} );
  print join( "\t", $_->{CALL}, $_->{GRIDSQUARE} ), "\n";
}

exit;

If you want some real code to manipulate ADIF files, adifmerg works.

The Accidental DXer

Update: I have a much better Ham Radio QSO Map now.

I had modest expectations when I set up my antenna and rig. I might’ve expected to work some of Canada, the northern US states, and maybe far western Europe, and that’s about it. But this map, extracted from my logs, shows different:

New Zealand, Ukraine, Patagonia, Greenland, Brazil, the Azores … way beyond what I expected.

I know the map’s not quite right. The lines should really be curved to be great circle lines on a Mercator map. Also, the NZ contact path was probably roughly SW through Hawaii. This round world doesn’t work on a flat page.