Tag: unicode

I (U+1F494, BROKEN HEART) UTF-8

Something has gone very wrong with the database encoding on this blog after a recent update, so all my lovely UTF-8 characters have gone mojibake.

Trying to find ways to fix it. It may have to be manual. Remember, kids: have backups before letting WordPress upgrade!

Here’s the Python equivalent of what I think the database has done:

bytes("I ???? UTF-8", encoding='utf-8').decode(encoding='cp1252')
'I 💔 UTF-8'

Quite why my hosting thought a character encoding from last century was appropriate, I’ll never know.

Update, November 2023: kinda-sort fixed the backend, but the encoding is still weird — can we…?

𒐳 / ༳ == ( ⑽ – 𐹭 ) * ( 𒐲 / 𐅉 ), of course

I just got brian d. foy’s Learning Perl 6 from the library. It’s a pretty good book, though it’ll take a good few readings for some of Perl 6’s features to stick.

Since Perl 6 is built using Unicode from the ground up, it does two rather wonderful things when dealing with numbers:

  1. regular expressions match numerals beyond 0–9: ٤ is as much four as 4
  2. numeric constants can (pretty much) be expressed in terms of Unicode values in your Perl 6 source code. Assigning π to a variable does what you think it does. Dividing by ¼ is the same as multiplying by, well, ٤.

So herewith a table (probably incomplete, and very unlikely to render properly for you) of Unicode glyphs accepted by Perl 6 as numeric values:

Value Glyphs
-0.5 ༳
0 0 ٠ ۰ ߀ ० ০ ੦ ૦ ୦ ௦ ౦ ౸ ೦ ൦ ๐ ໐ ༠ ၀ ႐ ០ ៰ ᠐ ᥆ ᧐ ᪀ ᪐ ᭐ ᮰ ᱀ ᱐ ⁰ ₀ ↉ ⓪ ⓿ 〇 ꘠ ꛯ ꣐ ꤀ ꧐ ꩐ ꯰ ０ 𐆊 𐒠 𑁦 𑃰 𑄶 𑇐 𑛀 𝟎 𝟘 𝟢 𝟬 𝟶 🄀 🄁
0.0625 ৴ ୵ ꠳
0.1 â…’
0.111111 â…‘
0.125 ৵ ୶ ⅛ ꠴ 𒑟
0.142857 ⅐
0.166667 â…™ ð’‘¡
0.1875 ৶ ୷ ꠵
0.2 â…•
0.25 ¼ ৷ ୲ ൳ ꠰ 𐅀 𐹼 𒑠 𒑢
0.333333 ⅓ 𐹽 𒑚 𒑝
0.375 ⅜
0.4 â…–
0.5 ½ ୳ ൴ ༪ ⳽ ꠱ 𐅁 𐅵 𐅶 𐹻
0.6 â…—
0.625 ⅝
0.666667 ⅔ 𐅷 𐹾 𒑛 𒑞
0.75 ¾ ৸ ୴ ൵ ꠲ 𐅸
0.8 â…˜
0.833333 ⅚ 𒑜
0.875 â…ž
1 1 ¹ ١ ۱ ߁ १ ১ ੧ ૧ ୧ ௧ ౧ ౹ ౼ ೧ ൧ ๑ ໑ ༡ ၁ ႑ ፩ ១ ៱ ᠑ ᥇ ᧑ ᧚ ᪁ ᪑ ᭑ ᮱ ᱁ ᱑ ₁ ⅟ Ⅰ ⅰ ① ⑴ ⒈ ⓵ ❶ ➀ ➊ 〡 ㆒ ㈠ ㊀ ꘡ ꛦ ꣑ ꤁ ꧑ ꩑ ꯱ １ 𐄇 𐅂 𐅘 𐅙 𐅚 𐌠 𐏑 𐒡 𐡘 𐤖 𐩀 𐩽 𐭘 𐭸 𐹠 𑁒 𑁧 𑃱 𑄷 𑇑 𑛁 𒐕 𒐞 𒐬 𒐴 𒑏 𒑘 𝍠 𝟏 𝟙 𝟣 𝟭 𝟷 🄂
1.5 ༫
2 2 ² ٢ ۲ ߂ २ ২ ੨ ૨ ୨ ௨ ౨ ౺ ౽ ೨ ൨ ๒ ໒ ༢ ၂ ႒ ፪ ២ ៲ ᠒ ᥈ ᧒ ᪂ ᪒ ᭒ ᮲ ᱂ ᱒ ₂ Ⅱ ⅱ ② ⑵ ⒉ ⓶ ❷ ➁ ➋ 〢 ㆓ ㈡ ㊁ ꘢ ꛧ ꣒ ꤂ ꧒ ꩒ ꯲ ２ 𐄈 𐅛 𐅜 𐅝 𐅞 𐏒 𐒢 𐡙 𐤚 𐩁 𐭙 𐭹 𐹡 𑁓 𑁨 𑃲 𑄸 𑇒 𑛂 𒐀 𒐖 𒐟 𒐣 𒐭 𒐵 𒑊 𒑐 𒑖 𒑙 𝍡 𝟐 𝟚 𝟤 𝟮 𝟸 🄃
2.5 ༬
3 3 ³ ٣ ۳ ߃ ३ ৩ ੩ ૩ ୩ ௩ ౩ ౻ ౾ ೩ ൩ ๓ ໓ ༣ ၃ ႓ ፫ ៣ ៳ ᠓ ᥉ ᧓ ᪃ ᪓ ᭓ ᮳ ᱃ ᱓ ₃ Ⅲ ⅲ ③ ⑶ ⒊ ⓷ ❸ ➂ ➌ 〣 ㆔ ㈢ ㊂ ꘣ ꛨ ꣓ ꤃ ꧓ ꩓ ꯳ ３ 𐄉 𐒣 𐡚 𐤛 𐩂 𐭚 𐭺 𐹢 𑁔 𑁩 𑃳 𑄹 𑇓 𑛃 𒐁 𒐈 𒐗 𒐠 𒐤 𒐥 𒐮 𒐯 𒐶 𒐷 𒐺 𒐻 𒑋 𒑑 𒑗 𝍢 𝟑 𝟛 𝟥 𝟯 𝟹 🄄
3.141592653589793 π
3.5 ༭
4 4 ٤ ۴ ߄ ४ ৪ ੪ ૪ ୪ ௪ ౪ ೪ ൪ ๔ ໔ ༤ ၄ ႔ ፬ ៤ ៴ ᠔ ᥊ ᧔ ᪄ ᪔ ᭔ ᮴ ᱄ ᱔ ⁴ ₄ Ⅳ ⅳ ④ ⑷ ⒋ ⓸ ❹ ➃ ➍ 〤 ㆕ ㈣ ㊃ ꘤ ꛩ ꣔ ꤄ ꧔ ꩔ ꯴ ４ 𐄊 𐒤 𐩃 𐭛 𐭻 𐹣 𑁕 𑁪 𑃴 𑄺 𑇔 𑛄 𒐂 𒐉 𒐏 𒐘 𒐡 𒐦 𒐰 𒐸 𒐼 𒐽 𒐾 𒐿 𒑌 𒑒 𒑓 𝍣 𝟒 𝟜 𝟦 𝟰 𝟺 🄅
4.5 ༮
5 5 ٥ ۵ ߅ ५ ৫ ੫ ૫ ୫ ௫ ౫ ೫ ൫ ๕ ໕ ༥ ၅ ႕ ፭ ៥ ៵ ᠕ ᥋ ᧕ ᪅ ᪕ ᭕ ᮵ ᱅ ᱕ ⁵ ₅ Ⅴ ⅴ ⑤ ⑸ ⒌ ⓹ ❺ ➄ ➎ 〥 ㈤ ㊄ ꘥ ꛪ ꣕ ꤅ ꧕ ꩕ ꯵ ５ 𐄋 𐅃 𐅈 𐅏 𐅟 𐅳 𐌡 𐒥 𐹤 𑁖 𑁫 𑃵 𑄻 𑇕 𑛅 𒐃 𒐊 𒐐 𒐙 𒐢 𒐧 𒐱 𒐹 𒑍 𒑔 𒑕 𝍤 𝟓 𝟝 𝟧 𝟱 𝟻 🄆
5.5 ༯
6 6 ٦ ۶ ߆ ६ ৬ ੬ ૬ ୬ ௬ ౬ ೬ ൬ ๖ ໖ ༦ ၆ ႖ ፮ ៦ ៶ ᠖ ᥌ ᧖ ᪆ ᪖ ᭖ ᮶ ᱆ ᱖ ⁶ ₆ Ⅵ ⅵ ↅ ⑥ ⑹ ⒍ ⓺ ❻ ➅ ➏ 〦 ㈥ ㊅ ꘦ ꛫ ꣖ ꤆ ꧖ ꩖ ꯶ ６ 𐄌 𐒦 𐹥 𑁗 𑁬 𑃶 𑄼 𑇖 𑛆 𒐄 𒐋 𒐑 𒐚 𒐨 𒑀 𒑎 𝍥 𝟔 𝟞 𝟨 𝟲 𝟼 🄇
6.5 ༰
7 7 ٧ ۷ ߇ ७ ৭ ੭ ૭ ୭ ௭ ౭ ೭ ൭ ๗ ໗ ༧ ၇ ႗ ፯ ៧ ៷ ᠗ ᥍ ᧗ ᪇ ᪗ ᭗ ᮷ ᱇ ᱗ ⁷ ₇ Ⅶ ⅶ ⑦ ⑺ ⒎ ⓻ ❼ ➆ ➐ 〧 ㈦ ㊆ ꘧ ꛬ ꣗ ꤇ ꧗ ꩗ ꯷ ７ 𐄍 𐒧 𐹦 𑁘 𑁭 𑃷 𑄽 𑇗 𑛇 𒐅 𒐌 𒐒 𒐛 𒐩 𒑁 𒑂 𒑃 𝍦 𝟕 𝟟 𝟩 𝟳 𝟽 🄈
7.5 ༱
8 8 ٨ ۸ ߈ ८ ৮ ੮ ૮ ୮ ௮ ౮ ೮ ൮ ๘ ໘ ༨ ၈ ႘ ፰ ៨ ៸ ᠘ ᥎ ᧘ ᪈ ᪘ ᭘ ᮸ ᱈ ᱘ ⁸ ₈ Ⅷ ⅷ ⑧ ⑻ ⒏ ⓼ ❽ ➇ ➑ 〨 ㈧ ㊇ ꘨ ꛭ ꣘ ꤈ ꧘ ꩘ ꯸ ８ 𐄎 𐒨 𐹧 𑁙 𑁮 𑃸 𑄾 𑇘 𑛈 𒐆 𒐍 𒐓 𒐜 𒐪 𒑄 𒑅 𝍧 𝟖 𝟠 𝟪 𝟴 𝟾 🄉
8.5 ༲
9 9 ٩ ۹ ߉ ९ ৯ ੯ ૯ ୯ ௯ ౯ ೯ ൯ ๙ ໙ ༩ ၉ ႙ ፱ ៩ ៹ ᠙ ᥏ ᧙ ᪉ ᪙ ᭙ ᮹ ᱉ ᱙ ⁹ ₉ Ⅸ ⅸ ⑨ ⑼ ⒐ ⓽ ❾ ➈ ➒ 〩 ㈨ ㊈ ꘩ ꛮ ꣙ ꤉ ꧙ ꩙ ꯹ ９ 𐄏 𐒩 𐹨 𑁚 𑁯 𑃹 𑄿 𑇙 𑛉 𒐇 𒐎 𒐔 𒐝 𒐫 𒑆 𒑇 𒑈 𒑉 𝍨 𝟗 𝟡 𝟫 𝟵 𝟿 🄊
10 ௰ ൰ ፲ Ⅹ ⅹ ⑩ ⑽ ⒑ ⓾ ❿ ➉ ➓ 〸 ㈩ ㉈ ㊉ 𐄐 𐅉 𐅐 𐅗 𐅠 𐅡 𐅢 𐅣 𐅤 𐌢 𐏓 𐡛 𐤗 𐩄 𐭜 𐭼 𐹩 𑁛 𝍩
11 Ⅺ ⅺ ⑪ ⑾ ⒒ ⓫
12 Ⅻ ⅻ ⑫ ⑿ ⒓ ⓬
13 ⑬ ⒀ ⒔ ⓭
14 ⑭ ⒁ ⒕ ⓮
15 ⑮ ⒂ ⒖ ⓯
16 ৹ ⑯ ⒃ ⒗ ⓰
17 ᛮ ⑰ ⒄ ⒘ ⓱
18 ᛯ ⑱ ⒅ ⒙ ⓲
19 ᛰ ⑲ ⒆ ⒚ ⓳
20 ፳ ⑳ ⒇ ⒛ ⓴ 〹 ㉉ 𐄑 𐏔 𐡜 𐤘 𐩅 𐭝 𐭽 𐹪 𑁜 𝍪
21 ㉑
22 ㉒
23 ㉓
24 ㉔
25 ㉕
26 ㉖
27 ㉗
28 ㉘
29 ㉙
30 ፴ 〺 ㉊ ㉚ 𐄒 𐅥 𐹫 𑁝 𝍫
31 ㉛
32 ㉜
33 ㉝
34 ㉞
35 ㉟
36 ㊱
37 ㊲
38 ㊳
39 ㊴
40 ፵ ㉋ ㊵ 𐄓 𐹬 𑁞 𝍬
41 ㊶
42 ㊷
43 ㊸
44 ㊹
45 ㊺
46 ㊻
47 ㊼
48 ㊽
49 ㊾
50 ፶ Ⅼ ⅼ ↆ ㉌ ㊿ 𐄔 𐅄 𐅊 𐅑 𐅦 𐅧 𐅨 𐅩 𐅴 𐌣 𐩾 𐹭 𑁟 𝍭
60 ፷ ㉍ 𐄕 𐹮 𑁠 𝍮
70 ፸ ㉎ 𐄖 𐹯 𑁡 𝍯
80 ፹ ㉏ 𐄗 𐹰 𑁢 𝍰
90 ፺ 𐄘 𐍁 𐹱 𑁣 𝍱
100 ௱ ൱ ፻ Ⅽ ⅽ 𐄙 𐅋 𐅒 𐅪 𐏕 𐡝 𐤙 𐩆 𐭞 𐭾 𐹲 𑁤
200 𐄚 𐹳
300 𐄛 𐅫 𐹴
400 𐄜 𐹵
500 Ⅾ ⅾ 𐄝 𐅅 𐅌 𐅓 𐅬 𐅭 𐅮 𐅯 𐅰 𐹶
600 𐄞 𐹷
700 𐄟 𐹸
800 𐄠 𐹹
900 𐄡 𐍊 𐹺
1000 ௲ ൲ Ⅿ ⅿ ↀ 𐄢 𐅍 𐅔 𐅱 𐡞 𐩇 𐭟 𐭿 𑁥
2000 𐄣
3000 𐄤
4000 𐄥
5000 ↁ 𐄦 𐅆 𐅎 𐅲
6000 𐄧
7000 𐄨
8000 𐄩
9000 𐄪
10000 ፼ ↂ 𐄫 𐅕 𐡟
20000 𐄬
30000 𐄭
40000 𐄮
50000 ↇ 𐄯 𐅇 𐅖
60000 𐄰
70000 𐄱
80000 𐄲
90000 𐄳
100000 ↈ
216000 𒐲
432000 𒐳
Inf ∞

So the title of this post really is accepted as a valid Perl 6 expression in the REPL:

$ perl6
To exit type 'exit' or '^D'
> 𒐳 / ༳ == ( ⑽ - 𐹭 ) * ( 𒐲 / 𐅉 )
True

What does it evaluate to? Well:

  • 𒐳 ‘CUNEIFORM NUMERIC SIGN SHAR2 TIMES GAL PLUS MIN’ represents 432000
  • ༳ ‘TIBETAN DIGIT HALF ZERO’ represents -½
  • ⑽ ‘PARENTHESIZED NUMBER TEN’ represents 10
  • 𐹭 ‘RUMI NUMBER FIFTY’ represents 50
  • 𒐲 ‘CUNEIFORM NUMERIC SIGN SHAR2 TIMES GAL PLUS DISH’ represents 216000
  • 𐅉 ‘GREEK ACROPHONIC ATTIC TEN TALENTS’ represents 10.

Definitely into just because you can doesn’t mean you should territory, and a feature to make the Pythonistas reach for the Zantac again, poor dears.

Ⓗⓞⓦ ⓣⓞ ⓑⓔ ⓐⓝⓝⓞⓨⓘⓝⓖ ⓦⓘⓣⓗ Ⓟⓔⓡⓛ ⓐⓝⓓ Ⓤⓝⓘⓒⓞⓓⓔ

It’s been so long since I’ve programmed in Perl. Twelve years ago, it was my life, but what with the Raspberry Pi intervening, I hadn’t used it in a while. It’s been so long, in fact, that I wasn’t aware of the new language structures available since version 5.14. Perl’s Unicode support has got a lot more robust, and I’m sick of Python’s whining about codecs when processing anything other than ASCII anyway. So I thought I’d combine re-learning some modern Perl with some childish amusement.

What I came up with was a routine to convert ASCII alphanumerics ([0-9A-Za-z]) to Unicode Enclosed Alphanumerics ([⓪-⑨Ⓐ-Ⓩⓐ-ⓩ]) for advanced lulz purposes. Ⓘ ⓣⓗⓘⓝⓚ ⓘⓣ ⓦⓞⓡⓚⓢ ⓡⓐⓣⓗⓔⓡ ⓦⓔⓛⓛ:

#!/usr/bin/perl
# annoying.pl - ⓑⓔ ⓐⓝⓝⓞⓨⓘⓝⓖ ⓦⓘⓣⓗ ⓤⓝⓘⓒⓞⓓⓔ
# created by scruss on 2014-05-18

use v5.14;
# fun UTF8 tricks from http://stackoverflow.com/questions/6162484/
use strict;
use utf8;
use warnings;
use charnames qw( :full :short );
sub annoyify;

die "usage: $0 ", annoyify('string to print like this'), "\n" if ( $#ARGV < 0 );
say annoyify( join( ' ', @ARGV ) );
exit;

# 💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩💩

sub annoyify() {
    # convert ascii to chars in circles
    my $str = shift;
    my @out;
    foreach ( split( '', $str ) ) {
        my $c = ord($_);             # remember, can be > 127 for UTF8
        if ( $c == charnames::vianame("DIGIT ZERO") )
	{
            # 💩💩💩 sigh; this one's real special ... 💩💩💩
            $c = charnames::vianame("CIRCLED DIGIT ZERO");
        }
        elsif ($c >= charnames::vianame("DIGIT ONE")
            && $c <= charnames::vianame("DIGIT NINE") )
        {
            # numerals, 1-9 only (grr)
            $c =
              charnames::vianame("CIRCLED DIGIT ONE") +
              $c -
              charnames::vianame("DIGIT ONE");
        }
        elsif ($c >= charnames::vianame("LATIN CAPITAL LETTER A")
            && $c <= charnames::vianame("LATIN CAPITAL LETTER Z") )
        {
            # upper case
            $c =
              charnames::vianame("CIRCLED LATIN CAPITAL LETTER A") +
              $c -
              charnames::vianame("LATIN CAPITAL LETTER A");
        }
        elsif ($c >= charnames::vianame("LATIN SMALL LETTER A")
            && $c <= charnames::vianame("LATIN SMALL LETTER Z") )
        {
            # lower case
            $c =
              charnames::vianame("CIRCLED LATIN SMALL LETTER A") +
              $c -
              charnames::vianame("LATIN SMALL LETTER A");
        }
        else {
            # pass thru non-ascii chars
        }
        push @out, chr($c);
    }
    return join( '', @out );
}

Yes, I really did have to do that special case for ⓪; ⓪…⑨ are not contiguous like ASCII 0…9. ⓑⓞⓞ!

Compose yourself, Raspberry Pi!

NB: If you’re gonna complain about the text encoding in this article, please know that the database went and broke itself: I (U+1F494, BROKEN HEART) UTF-8

Years ago, I worked in multilingual dictionary publishing. I was on the computing team, so we had to support the entry and storage of text in many different languages. Computers could display accented and special characters, but we were stuck with 8-bit character sets. This meant that we could only have a little over 200 distinct characters display in the same font at the same time. We’d be pretty much okay doing French & English together, but French & Norwegian started to get a little trying, and Italian & Greek couldn’t really be together at all.

We were very fortunate to be using Sun workstations in the editorial office. These were quite powerful Unix machines, which means that they were a fraction of the speed and capabilities of a Raspberry Pi. Suns had one particularly neat feature:

Compose_key_on_Sun_Type_5c_keyboard(source: Compose key, Wikipedia.)

That little key marked “Compose”  (to the right of the space bar) acted as a semi-smart typewriter backspace key: if you hit Compose, then the right key combination, an accented character or symbol would appear. Some of the straightforward compose key sequences are:

  Compose +    
Accent First key Second key Result Example
Acute ‘ e é café
Grave ` a à déjà
Cedilla , c ç soupçon
Circumflex ^ o ô hôtel
Umlaut “ u ü küche
Ring o a å Håkon
Slash / L Ł Łukasiewicz
Tilde ~ n ñ mañana

Like every (non-embedded) Linux system I’ve used, the Raspberry Pi running Raspbian can use the compose key method for entering extra characters. I’m annoyed, though, that almost every setup tutorial either says to disable it, or doesn’t explain what it’s for. Let me fix that for you …

Setup

Run raspi-config

sudo raspi-config

and go to the configure_keyboard “4 Internationalisation Options” → “I3 Change Keyboard Layout” section. Your keyboard’s probably mostly set up the way you want it, so hit the Tab key and select <Ok> until you get to the Compose key section:

raspi-config: Compose key selectionChoose whatever is convenient. The combined keyboard and trackpad I use (a SolidTek KB-3910) with my Raspberry Pi has a couple of “Windows® Logo” keys, and the one on the right works for me. Keep the rest of the keyboard options the same, and exit raspi-config. After the message

Reloading keymap. This may take a short while
[ ok ] Setting preliminary keymap...done.

appears, you now have a working Compose key.

Using the Compose key

raspi-config hints (‘On the text console the Compose key does not work in Unicode mode …’) that Compose might not work everywhere with every piece of software. I’ve tested it across quite a few pieces of software — both on the text console and under LXDE — and support seems to be almost universal. The only differences I can find are:

  • Text Console — (a. k. a. the texty bit you see after booting) Despite raspi-config’s warning, accented alphabetical characters do seem to work (é è ñ ö ø Ã¥, etc). Most symbols, however, don’t (like ± × ÷ …). The currency symbol for your country is a special case. In Canada, I need to use Compose for € and £, but you’ve probably got a key for that.
  • LXDE — (a. k. a. the mousey bit you see after typing ‘startx’) All characters and symbols I’ve tried work everywhere, in LXTerminal, Leafpad, Midori, Dillo (browser), IDLE, and FocusWriter (a very minimal word processor).

Special characters in Python's IDLE
Special characters in Python’s IDLE

Some Compose key sequences — Leafpad
Some Compose key sequences — Leafpad

To find out which key sequences do what, the Compose key – Wikipedia page is a decent start. I prefer the slightly friendlier Ubuntu references GtkComposeTable and Compose Key, or the almost unreadable but frighteningly comprehensive UTF-8 (Unicode) compose sequence reference (which is essentially mirrored on your Raspberry Pi as the file /usr/share/X11/locale/en_US.UTF-8/Compose). Now go forth and work that Compose key like a boß.

(If you’re on a Mac and feeling a bit left out, you can do something similar with the Option key. Here’s how: Extended Keyboard Accent Codes for the Macintosh. On Windows®? Out of luck, I’m afraid WinCompose!)