Develop a language to better describe how transliteration systems work

[ronaldtse]

The current method of relying on a mix of character replacements, contextual detection via regular expression, pre/post processing and options don't work that well.

Unicode has a language for expressing transliteration systems but it has its flaws.
https://www.unicode.org/cldr/charts/32/transforms/index.html

We should come up with a better language to describe how transliteration systems work.

ping @chaaklau

[ronaldtse]

The Unicode language looks like this:

From https://github.com/unicode-org/cldr/blob/master/common/transforms/ru_Latn-ru-BGN.xml:

		<transform source="ru_Latn" target="ru" variant="BGN" direction="forward" draft="contributed" alias="Latin-Russian/BGN ru-t-ru-latn-m0-bgn">
			<tRule><![CDATA[
# BGN/PCGN 1947 System for Russian, in direction ru_Latn → ru
# https://umnb4e1uguqx605uhk9jj.julianrbryant.com/gns/html/Romanization/Romanization_Russian.pdf
$prime = ʹ;
$doublePrime = ʺ;
$wordBoundary = [^[:L:][:M:][:N:]];
$upperConsonant = [БВГДЖЙКЛМНПРСТФХЦЧШЩЭ];
$lowerConsonant = [бвгджйклмнпрстфхцчшщэ];
$consonant = [$upperConsonant $lowerConsonant];
::NFC;
[:Upper:] {$prime} [^[:Lower:]] → Ь;
$prime → ь;
[:Upper:] {$doublePrime} [^[:Lower:]] → Ъ;
$doublePrime → ъ;
K[Hh] → Х;
k[Hh] → х;
T·S → ТС;
T·s → Тс;
t·S → тС;
t·s → тс;
T[Ss] → Ц;
t[Ss] → ц;
C[Hh] → Ч;
c[Hh] → ч;
S[Hh]·C[Hh] → ШЧ;
S[Hh]·c[Hh] → Шч;
s[Hh]·C[Hh] → шЧ;
s[Hh]·c[Hh] → шч;
S[Hh][Cc][Hh] → Щ;
s[Hh][Cc][Hh] → щ;
S[Hh] → Ш;
s[Hh] → ш;
Y[Ee] → Е;
y[Ee] → е;
Y[Ëë] → Ё;
y[Ëë] → ё;
Y[Uu] → Ю;
y[Uu] → ю;
Y[Aa] → Я;
y[Aa] → я;
{yy} $wordBoundary → ый;
$wordBoundary {Y} [^aeëiouyAEËIOUY] → Ы;
$wordBoundary {y} [^aeëiouyAEËIOUY] → ы;
$consonant {Y} → Ы;
$consonant {y} → ы;
Y → Й;
y → й;
$wordBoundary {E} → Э;
$wordBoundary {e} → э;
·E → Э;
·e → э;
E → Е;
e → е;
A → А;
a → а;
B → Б;
b → б;
V → В;
v → в;
G → Г;
g → г;
D → Д;
d → д;
Ë → Ё;
ë → ё;
Z[Hh] → Ж;
z[Hh] → ж;
Z → З;
z → з;
I → И;
i → и;
K → К;
k → к;
L → Л;
l → л;
M → М;
m → м;
N → Н;
n → н;
O → О;
o → о;
P → П;
p → п;
R → Р;
r → р;
S → С;
s → с;
T → Т;
t → т;
U → У;
u → у;
F → Ф;
f → ф;
·Y → Ы;
·y → ы;
· → ;
			]]></tRule>
		</transform>
	</transforms>

From: https://github.com/unicode-org/cldr/blob/master/common/transforms/Latin-ConjoiningJamo.xml

		<transform source="Latin" target="ConjoiningJamo" direction="both" visibility="internal">
			<tRule><![CDATA[
# Follows the Ministry of Culture and Tourism romanization: see https://umn0mtkzghdxczp6hhuxm.julianrbryant.com/korea/kor_loca.asp?code=A020303
# https://umn0mtkzgh2kwmpmhkae4.julianrbryant.com/cldr/transliteration_guidelines.html#Korean
#- N.B. DO NOT put any filters, NFD, etc. here -- those are aliased in
#- the INDEX file.  This transliterator is, by itself, not
#- instantiated.  It is used as a part of Latin-Jamo, Latin-Hangul, or
#- inverses thereof.
# Transliteration from Latin characters to Korean script is done in
# two steps: Latin to Jamo, then Jamo to Hangul.  The Jamo-Hangul
# transliteration is done algorithmically following Unicode 3.0
# section 3.11.  This file implements the Latin to Jamo
# transliteration using rules.
#   Initial only: IEUNG BB DD JJ R
#   Final only: BS GS L LB LG LH LM LP LS LT NG NH NJ
#   Initial and Final: B C D G GG H J K M N P S SS T
$Gi = ᄀ;
$KKi = ᄁ;
$Ni = ᄂ;
$Di = ᄃ;
$TTi = ᄄ;
$Li = ᄅ;
$Mi = ᄆ;
$Bi = ᄇ;
$PPi = ᄈ;
$Si = ᄉ;
$SSi = ᄊ;
...
# These are simple, since they can't chain. Note that we don't handle extreme cases like [ga][eo][e][o]
$sep ← a {} [$E $EO $EU];
$sep ← [^aow] e {} [$O $OE];
$sep ← [^aowy] e {} [$U $UI];
$sep ← [^ey] o {} [$E $EO $EU];
$sep ← [^y] u {} [$I];
# Similar to the above, but with an intervening $IEUNG.
$sep ← [^$latinMedial] [y] e {} $IEUNG [$O $OE];
$sep ← [^$latinMedial] e {} $IEUNG [$O $OE $U];
$sep ← [^$latinMedial] [o a] {} $IEUNG [$E $EO $EU];
$sep ← [^$latinMedial] [w y] a {} $IEUNG [$E $EO $EU];
# Single finals followed by IEUNG.  The jamo sequence A Xf IEUNG E,
# where Xi also exists, must be transliterated as "ax-e" to prevent
# the round trip conversion to A Xi E.
$sep ← $latinMedialEnd b {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd d {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd g {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd h {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd j {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd k {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd m {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd n {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd p {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd s {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd t {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd l {} $IEUNG $jamoMedial;
# Double finals followed by IEUNG.  Similar to the single finals
# followed by IEUNG.  Any latin consonant pair X Y, between medials,
# that we would split by Latin-Jamo, we must handle when it occurs as
# part of A XYf IEUNG E, to prevent round trip conversion to A Xf Yi E
$sep ← $latinMedialEnd b s {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd k k {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd g s {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd l b {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd l g {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd l h {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd l m {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd l p {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd l s {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd l t {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd n g {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd n h {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd n j {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd s s {} $IEUNG $jamoMedial;
$sep ← $latinMedialEnd ch {} $IEUNG $jamoMedial;
# Split doubles.  Text of the form A Xi Xf E, where XXi also occurs,
# we transliterate as "ax-xe" to prevent round trip transliteration as
# A XXi E.
$sep ← $latinMedialEnd j {} $Ji $jamoMedial;
$sep ← $latinMedialEnd k {} $Ki $jamoMedial;
$sep ← $latinMedialEnd s {} $Si $jamoMedial;
# XYY.  This corresponds to the XYY rule in Latin-Jamo.  By default
# Latin-Jamo maps "xyy" to Xf YYi, to keep YY together.  As a result,
# "xyy" forms that correspond to XYf Yi must be transliterated as
# "xy-y".
$sep ← $latinMedialEnd b s {} [$Si $SSi];
$sep ← $latinMedialEnd g s {} [$Si $SSi];
$sep ← $latinMedialEnd l b {} [$Bi];
$sep ← $latinMedialEnd l g {} [$Gi];
$sep ← $latinMedialEnd l s {} [$Si $SSi];
$sep ← $latinMedialEnd n g {} [$Gi];
$sep ← $latinMedialEnd n j {} [$Ji $JJi];
# $sep ← $latinMedialEnd l  {} [$PPi];
# $sep ← $latinMedialEnd l  {} [$TTi];
$sep ← $latinMedialEnd l p {} [$Pi];
$sep ← $latinMedialEnd l t {} [$Ti];
$sep ← $latinMedialEnd k {} [$KKi $Ki];
$sep ← $latinMedialEnd p {} $Pi;
$sep ← $latinMedialEnd t {} $Ti;
$sep ← $latinMedialEnd c {} [$Hi];
# Deletion of IEUNG is handled below.
#----------------------------------------------------------------------
# Latin-Jamo
# [Basic, context-free Jamo-Latin rules are embedded here too.  See
# above.]
# Split digraphs: Text of the form 'axye', where 'xy' is a final
# digraph, 'x' is a final (by itself), 'y' is an initial, and 'a' and
# 'e' are medials, we want to transliterate this as A Xf Yi E rather
# than A XYf IEUNG E.  We do NOT include text of the form "axxe",
# since that is handled differently below.  These rules are generated
# programmatically from the jamo data.
$jamoMedial {b s} $latinMedial → $Bf $Si;
$jamoMedial {g s} $latinMedial → $Gf $Si;
$jamoMedial {l b} $latinMedial → $L $Bi;
$jamoMedial {l g} $latinMedial → $L $Gi;
$jamoMedial {l h} $latinMedial → $L $Hi;
$jamoMedial {l m} $latinMedial → $L $Mi;
$jamoMedial {l p} $latinMedial → $L $Pi;
$jamoMedial {l s} $latinMedial → $L $Si;
$jamoMedial {l t} $latinMedial → $L $Ti;
$jamoMedial {n g} $latinMedial → $Nf $Gi;
$jamoMedial {n h} $latinMedial → $Nf $Hi;
$jamoMedial {n j} $latinMedial → $Nf $Ji;
# Single consonants are initials: Text of the form 'axe', where 'x'
# can be an initial or a final, and 'a' and 'e' are medials, we want
# to transliterate as A Xi E rather than A Xf IEUNG E.
$jamoMedial {b} $latinMedial → $Bi;
$jamoMedial {ch} $latinMedial → $CHi;
$jamoMedial {d} $latinMedial → $Di;
$jamoMedial {g} $latinMedial → $Gi;
$jamoMedial {h} $latinMedial → $Hi;
$jamoMedial {j} $latinMedial → $Ji;
$jamoMedial {k} $latinMedial → $Ki;
$jamoMedial {m} $latinMedial → $Mi;
$jamoMedial {n} $latinMedial → $Ni;
$jamoMedial {p} $latinMedial → $Pi;
$jamoMedial {s} $latinMedial → $Si;
$jamoMedial {t} $latinMedial → $Ti;
$jamoMedial {l} $latinMedial → $Li;
# Doubled initials.  The sequence "axxe", where XX exists as an initial
# (XXi), and also Xi and Xf exist (true of all digraphs XX), we want
# to transliterate as A XXi E, rather than split to A Xf Xi E.
$jamoMedial {p p} $latinMedial → $PPi;
$jamoMedial {t t} $latinMedial → $TTi;
$jamoMedial {j j} $latinMedial → $JJi;
$jamoMedial {k k} $latinMedial → $KKi;
$jamoMedial {s s} $latinMedial → $SSi;
# XYY.  Because doubled consonants bind more strongly than XY
# consonants, we must handle the sequence "axyy" specially.  Here XYf
# and YYi must exist.  In these cases, we map to Xf YYi rather than
# XYf.
# However, there are two special cases.
$jamoMedial {lp} p p → $LP;
$jamoMedial {lt} t t → $LT;
# End special cases
$jamoMedial {b} s s → $Bf;
$jamoMedial {g} s s → $Gf;
$jamoMedial {l} b b → $L;
$jamoMedial {l} g g → $L;
$jamoMedial {l} s s → $L;
$jamoMedial {l} t t → $L;
$jamoMedial {l} p p → $L;
$jamoMedial {n} g g → $Nf;
$jamoMedial {n} j j → $Nf;
# Finals: Attach consonant with preceding medial to preceding medial.
# Do this BEFORE mapping consonants to initials.  Longer keys must
# precede shorter keys that they start with, e.g., the rule for 'bs'
# must precede 'b'.
# [BASIC Jamo-Latin FINALS handled here.  Order irrelevant within this
# block for Jamo-Latin.]
$jamoMedial {bs} ↔ $BS;
$jamoMedial {b} ↔ $Bf;
$jamoMedial {ch} ↔ $Cf;
$jamoMedial {c} → $Cf;
$jamoMedial {d} ↔ $Df;
$jamoMedial {kk} ↔ $GGf;
$jamoMedial {gs} ↔ $GS;
$jamoMedial {g} ↔ $Gf;
$jamoMedial {h} ↔ $Hf;
$jamoMedial {j} ↔ $Jf;
$jamoMedial {k} ↔ $Kf;
$jamoMedial {lb} ↔ $LB;  $jamoMedial {lg} ↔ $LG;
$jamoMedial {lh} ↔ $LH;
$jamoMedial {lm} ↔ $LM;
$jamoMedial {lp} ↔ $LP;
$jamoMedial {ls} ↔ $LS;
$jamoMedial {lt} ↔ $LT;
$jamoMedial {l} ↔ $L;
$jamoMedial {m} ↔ $Mf;
$jamoMedial {ng} ↔ $NG;
$jamoMedial {nh} ↔ $NH;
$jamoMedial {nj} ↔ $NJ;
$jamoMedial {n} ↔ $Nf;
$jamoMedial {p} ↔ $Pf;
$jamoMedial {ss} ↔ $SSf;
$jamoMedial {s} ↔ $Sf;
$jamoMedial {t} ↔ $Tf;
# Initials: Attach single consonant to following medial.  Do this
# AFTER mapping finals.  Longer keys must precede shorter keys that
# they start with, e.g., the rule for 'gg' must precede 'g'.
# [BASIC Jamo-Latin INITIALS handled here.  Order irrelevant within
# this block for Jamo-Latin.]
{kk} $latinMedial ↔ $KKi;
{g} $latinMedial ↔ $Gi;
{n} $latinMedial ↔ $Ni;
{tt} $latinMedial ↔ $TTi;
{d} $latinMedial ↔ $Di;
{l} $latinMedial ↔ $Li;
{m} $latinMedial ↔ $Mi;
{pp} $latinMedial ↔ $PPi;
{b} $latinMedial ↔ $Bi;
{ss} $latinMedial ↔ $SSi;
{s} $latinMedial ↔ $Si;
{jj} $latinMedial ↔ $JJi;
{j} $latinMedial ↔ $Ji;
{ch} $latinMedial ↔ $CHi;
{c} $latinMedial → $CHi;
{k} $latinMedial ↔ $Ki;
{t} $latinMedial ↔ $Ti;
{p} $latinMedial ↔ $Pi;
{h} $latinMedial ↔ $Hi;
# 'r' in final position.  Because of the equivalency of the 'l' and
# 'r' jamo (the glyphs are the same), we try to provide the same
# equivalency in Latin-Jamo.  The 'l' to 'r' conversion is handled
# below.  If we see an 'r' in an apparent final position, treat it
# like 'l'.  For example, "karka" =→ Ki A R EU Ki A without this rule.
# Instead, we want Ki A L Ki A.
# Initial + Final: If we match the next rule, we have initial then
# final consonant with no intervening medial.  We insert the null
# vowel BEFORE it to create a well-formed syllable.  (In the next rule
# we insert a null vowel AFTER an anomalous initial.)
# Initial + X: This block matches an initial consonant not followed by
# a medial.  We insert the null vowel after it.  We handle double
# initials explicitly here; for single initial consonants we insert EU
# (as Latin) after them and let standard rules do the rest.
# BREAKS ROUND TRIP INTEGRITY
kk → $KKi $EU;
tt → $TTi $EU;
pp → $PPi $EU;
ss → $SSi $EU;
jj → $JJi $EU;
ch → $CHi $EU;
([lbdghjkmnpst]) → | $1 eu;
# X + Final: Finally we have to deal with a consonant that can only be
# interpreted as a final (not an initial) and which is preceded
# neither by an initial nor a medial.  It is the start of the
# syllable, but cannot be.  Most of these will already be handled by
# the above rules.  'bs' splits into Bi EU Sf.  Similar for 'gs' 'ng'
# 'nh' 'nj'.  The only problem is 'l' and digraphs starting with 'l'.
# For this isolated case, we could add a null initial and medial,
# which would give "la" =→ IEUNG EU L IEUNG A, for example.  A more
# economical solution is to transliterate isolated "l" (that is,
# initial "l") to "r".  (Other similar conversions of consonants that
# occur neither as initials nor as finals are handled below.)
l → | r;
# Medials.  If a medial is preceded by an initial, then we proceed
# normally.  As usual, longer keys must precede shorter ones.
# [BASIC Jamo-Latin MEDIALS handled here.  Order irrelevant within
# this block for Jamo-Latin.]
#
# a e i o u
# ae
# eo eu
# oe
# ui
# wa we wi
# wae
# yae ya yeo ye yo yu
$jamoInitial {ae} ↔ $AE;
$jamoInitial {a} ↔ $A;
$jamoInitial {eo} ↔ $EO;
$jamoInitial {eu} ↔ $EU;
$jamoInitial {e} ↔ $E;
$jamoInitial {i} ↔ $I;
$jamoInitial {oe} ↔ $OE;
$jamoInitial {o} ↔ $O;
$jamoInitial {ui} ↔ $UI;
$jamoInitial {u} ↔ $U;
$jamoInitial {wae} ↔ $WAE;
$jamoInitial {wa} ↔ $WA;
$jamoInitial {wo} ↔ $WO;
$jamoInitial {we} ↔ $WE;
$jamoInitial {wi} ↔ $WI;
$jamoInitial {yae} ↔ $YAE;
$jamoInitial {ya} ↔ $YA;
$jamoInitial {yeo} ↔ $YEO;
$jamoInitial {ye} ↔ $YE;
$jamoInitial {yo} ↔ $YO;
$jamoInitial {yu} ↔ $YU;
# We may see an anomalous isolated 'w' or 'y'.  In that case, we
# interpret it as 'wi' and 'yu', respectively.
# BREAKS ROUND TRIP INTEGRITY
$jamoInitial {w} → | wi;
$jamoInitial {y} → | yu;
# Otherwise, insert a null consonant IEUNG before the medial (which is
# still an untransliterated latin vowel).
($latinMedial) → $IEUNG | $1;
# Convert non-jamo latin consonants to equivalents.  These occur as
# neither initials nor finals in jamo.  'l' occurs as a final, but not
# an initial; it is handled above.  The following letters (left hand
# side) will never be output by Jamo-Latin.
f → | p;
q → | k;
v → | b;
x → | ks;
z → | s;
r → | l;
c → | k;
# Delete separators (Latin-Jamo).
$sep → ;
# Delete null consonants (Jamo-Latin).  Do NOT delete null EU vowels,
# since these may also occur in text.
← $IEUNG;

[ronaldtse]

The definition of Unicode's "transformation language" is here:
https://www.unicode.org/reports/tr35/tr35-general.html#Transforms

[chaaklau]

Here are some initial thoughts.

A comparison between Unicode LDML and interscript

The current method of relying on a mix of character replacements, contextual detection via regular expression, pre/post processing and options don't work that well.

interscript breaks down transliteration into three parts:

• Pre-processing (rules)
• Character Replacement (characters) only allows context-less, global replacement
• Post-processing (postrules)

The transforms in Unicode LDML (Locale Data Markup Language) do not distinguish these three stages. Instead, there can be an unlimited number of passes, and any transform rule can be context-sensitive. They are equivalent in most of the cases.

LDML is an unrestrictive version of interscript.

---

There are two LDML features that currently interscript does not have:

1. Variables
   Predefined names for groups, e.g. $gammaLike = [ΓΚΞΧγκξχϰ] ;, $lowerConsonant = [бвгджйклмнпрстфхцчшщэ];, etc.
2. Bidirectionality

These are features that help readability of the transliteration rules, but does not solve issues that we have.

---

Another crucial difference is on how transformation is done.

• interscript loops through all mappings under rules, characters, and postrules, and applies them globally to the entire string.
• LDML walks through the string from start to finish, and looks for relevant rules within the same block, and move the cursor to the next position after replacement. LDML therefore provides a | operator that indicates up to which point the cursor should move after a transformation rule is applied.

interscript is good when we have long strings and few rules (e.g. Latn <-> Cyrl) (O(r log s), r: number of rules, s: length of the string), but will have a slow runtime for Asian scripts with over 10000 rules (e.g. Hani or Hang). The | operator makes transformation rules less transparent and should be avoided.

---

Proposed features:

1. Variable-based operations (something like $Vowels instead of [AaEeIiOoUu])
2. Markup tags for entities (GeoNames, Person Names, Currency, etc.)
3. A way to specify pre-processing engines, e.g. use module ABC to preprocess (normalize, segment, tag name entities) the text before performing transliteration tasks below
4. Allow separate transliteration rules for different types of entity, e.g. title case for spans marked as GeoNames, hyphenate syllables within Person Names, etc.
5. Rules to specify unhandled cases (e.g. combinations not supported by the transliteration system)
6. Functions to support loose transliteration systems (interscript/interscript#39)
7. Support for dictionary-based transliteration (interscript/interscript#31)
8. Options: return best guess or all possible transliterations

[ronaldtse]

Ping @chaaklau any updates here? Thanks!

[ronaldtse]

@chaaklau please also have a look at this: https://scripts.sil.org/cms/scripts/page.php?item_id=TECkitIntro

[chaaklau]

@chaaklau please also have a look at this: https://scripts.sil.org/cms/scripts/page.php?item_id=TECkitIntro

This seems to be an encoding-conversion tool. This might be useful if one needs to handle non-Unicode data (e.g. most Burmese data are in zawgyi1).

[ronaldtse]

Unicode seems to have updated their charts to YAML, and the language used.

Features:

1. Differentiate unidirectional and bidirectional transforms.

   • ሀ > ha
   • ኢ < i
   • ለ <> lə

2. Fully utilize Unicode characters and their composites.

   • `t͡ʃ``
   • symbols: < [ˈˌ\\. ̯̃]
   • < [ ̃ ̰ ̋ ́ ̄ ̀ ̏ ̌ ̂ ˥ ˦ ˧ ˨ ˩ ꜜ ꜛ ↗ ↘ ]

3. Context separators {} are used to separate characters.

   • [p{pʼ}]

4. Character group definitions like regexes.

   • [aeəiɨou]

5. Variable assignment

   • $IPA_VOWEL = [aeəiɨou];

6. Comments

   • # ኢስላማባድ < Islamabad /islamabad/

7. Lookbacks

   • $1 w < {($IPA_VOWEL ː?) ̯} $IPA_VOWEL; # /ewowa/ < /e̯o̯a/
   • ($IPA_CONSONANT) ([jw]? $IPA_VOWEL) \\u135D > $1 ː $2 ː;
   • {($IPA_VOWEL ː?)} [[:L:]] > $1 \\.
   • $1 wa \\u135D < ($LABIALIZABLE_BEFORE_A) ː waː; # ቷ፝ < [tːʷaː]

8. Flow control. Seems that their rules are applied one by one on the previous results.

   • ::null()
   • ::(null)
   • ::null
   • ::(NFC)
   • ::(NFD)

Extract from CLDR transforms/am-am_FONIPA.yml

--- 
:transforms: 
  - 
    :direction: both
    :rules: 
      - "$IPA_VOWEL = [aeəiɨou];"
      - "$IPA_CONSONANT = [mnɲɴ p{pʼ}bt{tʼ}dk{kʼ}ɡʔʕ fvs{sʼ}zʃʒxh lr {t͡ʃ}{t͡ʃʼ}{d͡ʒ}];"
      - "$LABIALIZABLE_BEFORE_A = [p{pʼ}t{tʼ} {t͡ʃ}{t͡ʃʼ}{d͡ʒ}{d͡ʒʼ} s{sʼ}zʃʒ fv r];"
      - "< [ ʼ  ͡  ͜  ̯];"
      - "::(null);"
      - "ሀ > ha;"
      - "ሀ < hə;"
... 
    :source: am
    :target: am_FONIPA
    :variant: ~

[ronaldtse]

Proposed features:

1. Variable-based operations (something like $Vowels instead of [AaEeIiOoUu])
2. Markup tags for entities (GeoNames, Person Names, Currency, etc.)
3. A way to specify pre-processing engines, e.g. use module ABC to preprocess (normalize, segment, tag name entities) the text before performing transliteration tasks below
4. Allow separate transliteration rules for different types of entity, e.g. title case for spans marked as GeoNames, hyphenate syllables within Person Names, etc.
5. Rules to specify unhandled cases (e.g. combinations not supported by the transliteration system)
6. Functions to support loose transliteration systems (Test against GeoNames data (Hong Kong) #39)
7. Support for dictionary-based transliteration (Chinese: consider the concept of dictionaries #31)
8. Options: return best guess or all possible transliterations

Fully agree with @chaaklau on the desired features. One more:

9. Bi-directional indication of rules so we can see what systems can support bi-directional conversions.

[ronaldtse]

Additional requirement: conversion systems often have "options", e.g. character set with diacritics, character set without diacritics. Reversible vs non-reversible. We need to support them.

[ronaldtse]

The DSL needs to support all the current operations of existing maps, and also support the concept of stages, etc.

From @webdev778 :

regarding the DSL, as much as I understand it, it is about changing the whole engine into something like interscript 2.
or this DSL can compile the maps to the current engine, but I don't know, this would be limited if the semantics change at least.
the new engine would support most operation of the old engine, with the exception, that the regexp support could be a bit limited.
because we won't use regexp but on demand we will be able to expand it

[ronaldtse]

From @ronaldtse

We also need an Atom for the "stage".

For example, Korean transliteration go through these stages: Hanja => Hangeul => Jamo => Rules => Latn
Thai transliteration goes like this: Thai script => trained model for inferring to pronunciation => IPA script => Rules => Latn

The data passed through each stage may be data structures, not plain text

For example, if we want to do Japanese transliteration with proper nouns (capitalization), we will need to do: Kanji => semantic enhancement (detect proper nouns) => each token go through Hira/Kata => Rules => Latn (with capitalization)

We will need to port all the maps (we can ask the authors of the original maps to update them for v2)
For the DSL, we want to have a language independent DSL, that is not bound to Ruby. It could be realized in YAML, JSON or other formats.
A sample of that is like GitHub Actions syntax for workflows, it’s realized in YAML

I think eventually we need the DSL with 2 levels of support. Some runtimes can support all mechanisms, including the deep learning models. Some runtimes only support rules.
We want the maps to be portable, and can be defined by people outside the interscript project

My main concern is about what is "interchangable" and portability.

• Interchangeable - definition is human readable and can be used by multiple runtimes (whether compiled or not)
• Portability - the user can run the software without needing to install compilation tools

[ronaldtse]

Sample:

[ronaldtse]

Moved this issue back to main interscript/interscript since the interscript/lcs repo is being archived.