Up  

Module Uucd

module Uucd: sig .. end
Unicode character database decoder.

Uucd decodes the data of the Unicode character database from its XML representation. It provides high-level (but not necessarily efficient) access to the data so that efficient representations can be extracted.

Uucd decodes the representation described in the Annex #42 of Unicode 6.3.0. Subsequent versions may be decoded as long as no new cases are introduced in parsed enumerated properties.

Consult the basics.

Note. All strings returned by the module are UTF-8 encoded.

Release 1.0.0 — Unicode version 6.3.0 — Daniel Bünzli <daniel.buenzl i@erratique.ch>

References



Code points


type cp = int
The type for Unicode code points, ranges from 0x0000 to 0x10_FFFF.
val is_cp : int -> bool
is_cp n is true iff n a Unicode code point.
val is_scalar_value : int -> bool
is_scalar_value n is true iff n is a Unicode scalar value.
module Cpmap: Map.S  with type key = cp
Code point maps.

Properties

Properties are referenced by their name and property values by their abbreviated name. To understand their semantics refer to the standard.

type props
The type for sets of properties.
type 'a prop
The type for properties with property value of type 'a.
val find : props -> 'a prop -> 'a option
find ps p is the value of property p in ps, if any.
val unknown_prop : string * string -> string prop
unknown_prop (ns, n) is a property read from an XML attribute whose expanded name is (ns, n). This can be used to access a property unknown to the module.

Non Unihan properties

In alphabetical order.

val age : [ `Unassigned | `Version of int * int ] prop
val alphabetic : bool prop
val ascii_hex_digit : bool prop
val bidi_class : [ `AL
       | `AN
       | `B
       | `BN
       | `CS
       | `EN
       | `ES
       | `ET
       | `FSI
       | `L
       | `LRE
       | `LRI
       | `LRO
       | `NSM
       | `ON
       | `PDF
       | `PDI
       | `R
       | `RLE
       | `RLI
       | `RLO
       | `S
       | `WS ] prop
val bidi_control : bool prop
val bidi_mirrored : bool prop
val bidi_mirroring_glyph : cp option prop
val bidi_paired_bracket : [ `Cp of cp | `Self ] prop
val bidi_paired_bracket_type : [ `C | `N | `O ] prop
val block : [ `ASCII
       | `Aegean_Numbers
       | `Alchemical
       | `Alphabetic_PF
       | `Ancient_Greek_Music
       | `Ancient_Greek_Numbers
       | `Ancient_Symbols
       | `Arabic
       | `Arabic_Ext_A
       | `Arabic_Math
       | `Arabic_PF_A
       | `Arabic_PF_B
       | `Arabic_Sup
       | `Armenian
       | `Arrows
       | `Avestan
       | `Balinese
       | `Bamum
       | `Bamum_Sup
       | `Batak
       | `Bengali
       | `Block_Elements
       | `Bopomofo
       | `Bopomofo_Ext
       | `Box_Drawing
       | `Brahmi
       | `Braille
       | `Buginese
       | `Buhid
       | `Byzantine_Music
       | `CJK
       | `CJK_Compat
       | `CJK_Compat_Forms
       | `CJK_Compat_Ideographs
       | `CJK_Compat_Ideographs_Sup
       | `CJK_Ext_A
       | `CJK_Ext_B
       | `CJK_Ext_C
       | `CJK_Ext_D
       | `CJK_Radicals_Sup
       | `CJK_Strokes
       | `CJK_Symbols
       | `Carian
       | `Chakma
       | `Cham
       | `Cherokee
       | `Compat_Jamo
       | `Control_Pictures
       | `Coptic
       | `Counting_Rod
       | `Cuneiform
       | `Cuneiform_Numbers
       | `Currency_Symbols
       | `Cypriot_Syllabary
       | `Cyrillic
       | `Cyrillic_Ext_A
       | `Cyrillic_Ext_B
       | `Cyrillic_Sup
       | `Deseret
       | `Devanagari
       | `Devanagari_Ext
       | `Diacriticals
       | `Diacriticals_For_Symbols
       | `Diacriticals_Sup
       | `Dingbats
       | `Domino
       | `Egyptian_Hieroglyphs
       | `Emoticons
       | `Enclosed_Alphanum
       | `Enclosed_Alphanum_Sup
       | `Enclosed_CJK
       | `Enclosed_Ideographic_Sup
       | `Ethiopic
       | `Ethiopic_Ext
       | `Ethiopic_Ext_A
       | `Ethiopic_Sup
       | `Geometric_Shapes
       | `Georgian
       | `Georgian_Sup
       | `Glagolitic
       | `Gothic
       | `Greek
       | `Greek_Ext
       | `Gujarati
       | `Gurmukhi
       | `Half_And_Full_Forms
       | `Half_Marks
       | `Hangul
       | `Hanunoo
       | `Hebrew
       | `High_PU_Surrogates
       | `High_Surrogates
       | `Hiragana
       | `IDC
       | `IPA_Ext
       | `Imperial_Aramaic
       | `Indic_Number_Forms
       | `Inscriptional_Pahlavi
       | `Inscriptional_Parthian
       | `Jamo
       | `Jamo_Ext_A
       | `Jamo_Ext_B
       | `Javanese
       | `Kaithi
       | `Kana_Sup
       | `Kanbun
       | `Kangxi
       | `Kannada
       | `Katakana
       | `Katakana_Ext
       | `Kayah_Li
       | `Kharoshthi
       | `Khmer
       | `Khmer_Symbols
       | `Lao
       | `Latin_1_Sup
       | `Latin_Ext_A
       | `Latin_Ext_Additional
       | `Latin_Ext_B
       | `Latin_Ext_C
       | `Latin_Ext_D
       | `Lepcha
       | `Letterlike_Symbols
       | `Limbu
       | `Linear_B_Ideograms
       | `Linear_B_Syllabary
       | `Lisu
       | `Low_Surrogates
       | `Lycian
       | `Lydian
       | `Mahjong
       | `Malayalam
       | `Mandaic
       | `Math_Alphanum
       | `Math_Operators
       | `Meetei_Mayek
       | `Meetei_Mayek_Ext
       | `Meroitic_Cursive
       | `Meroitic_Hieroglyphs
       | `Miao
       | `Misc_Arrows
       | `Misc_Math_Symbols_A
       | `Misc_Math_Symbols_B
       | `Misc_Pictographs
       | `Misc_Symbols
       | `Misc_Technical
       | `Modifier_Letters
       | `Modifier_Tone_Letters
       | `Mongolian
       | `Music
       | `Myanmar
       | `Myanmar_Ext_A
       | `NB
       | `NKo
       | `New_Tai_Lue
       | `Number_Forms
       | `OCR
       | `Ogham
       | `Ol_Chiki
       | `Old_Italic
       | `Old_Persian
       | `Old_South_Arabian
       | `Old_Turkic
       | `Oriya
       | `Osmanya
       | `PUA
       | `Phags_Pa
       | `Phaistos
       | `Phoenician
       | `Phonetic_Ext
       | `Phonetic_Ext_Sup
       | `Playing_Cards
       | `Punctuation
       | `Rejang
       | `Rumi
       | `Runic
       | `Samaritan
       | `Saurashtra
       | `Sharada
       | `Shavian
       | `Sinhala
       | `Small_Forms
       | `Sora_Sompeng
       | `Specials
       | `Sundanese
       | `Sundanese_Sup
       | `Sup_Arrows_A
       | `Sup_Arrows_B
       | `Sup_Math_Operators
       | `Sup_PUA_A
       | `Sup_PUA_B
       | `Sup_Punctuation
       | `Super_And_Sub
       | `Syloti_Nagri
       | `Syriac
       | `Tagalog
       | `Tagbanwa
       | `Tags
       | `Tai_Le
       | `Tai_Tham
       | `Tai_Viet
       | `Tai_Xuan_Jing
       | `Takri
       | `Tamil
       | `Telugu
       | `Thaana
       | `Thai
       | `Tibetan
       | `Tifinagh
       | `Transport_And_Map
       | `UCAS
       | `UCAS_Ext
       | `Ugaritic
       | `VS
       | `VS_Sup
       | `Vai
       | `Vedic_Ext
       | `Vertical_Forms
       | `Yi_Radicals
       | `Yi_Syllables
       | `Yijing ] prop
val canonical_combining_class : int prop
val cased : bool prop
val case_folding : [ `Cps of cp list | `Self ] prop
val case_ignorable : bool prop
val changes_when_casefolded : bool prop
val changes_when_casemapped : bool prop
val changes_when_lowercased : bool prop
val changes_when_nfkc_casefolded : bool prop
val changes_when_titlecased : bool prop
val changes_when_uppercased : bool prop
val composition_exclusion : bool prop
val dash : bool prop
val decomposition_mapping : [ `Cps of cp list | `Self ] prop
val decomposition_type : [ `Can
       | `Com
       | `Enc
       | `Fin
       | `Font
       | `Fra
       | `Init
       | `Iso
       | `Med
       | `Nar
       | `Nb
       | `None
       | `Sml
       | `Sqr
       | `Sub
       | `Sup
       | `Vert
       | `Wide ] prop
val default_ignorable_code_point : bool prop
val deprecated : bool prop
val diacritic : bool prop
val east_asian_width : [ `A | `F | `H | `N | `Na | `W ] prop
val expands_on_nfc : bool prop
val expands_on_nfd : bool prop
val expands_on_nfkc : bool prop
val expands_on_nfkd : bool prop
val extender : bool prop
val fc_nfkc_closure : [ `Cps of cp list | `Self ] prop
val full_composition_exclusion : bool prop
val general_category : [ `Cc
       | `Cf
       | `Cn
       | `Co
       | `Cs
       | `Ll
       | `Lm
       | `Lo
       | `Lt
       | `Lu
       | `Mc
       | `Me
       | `Mn
       | `Nd
       | `Nl
       | `No
       | `Pc
       | `Pd
       | `Pe
       | `Pf
       | `Pi
       | `Po
       | `Ps
       | `Sc
       | `Sk
       | `Sm
       | `So
       | `Zl
       | `Zp
       | `Zs ] prop
val grapheme_base : bool prop
val grapheme_cluster_break : [ `CN | `CR | `EX | `L | `LF | `LV | `LVT | `PP | `RI | `SM | `T | `V | `XX ]
       prop
val grapheme_extend : bool prop
val grapheme_link : bool prop
val hangul_syllable_type : [ `L | `LV | `LVT | `NA | `T | `V ] prop
val hex_digit : bool prop
val hyphen : bool prop
val id_continue : bool prop
val id_start : bool prop
val ideographic : bool prop
val ids_binary_operator : bool prop
val ids_trinary_operator : bool prop
val indic_syllabic_category : [ `Avagraha
       | `Bindu
       | `Consonant
       | `Consonant_Dead
       | `Consonant_Final
       | `Consonant_Head_Letter
       | `Consonant_Medial
       | `Consonant_Placeholder
       | `Consonant_Repha
       | `Consonant_Subjoined
       | `Modifying_Letter
       | `Nukta
       | `Other
       | `Register_Shifter
       | `Tone_Letter
       | `Tone_Mark
       | `Virama
       | `Visarga
       | `Vowel
       | `Vowel_Dependent
       | `Vowel_Independent ] prop
val indic_matra_category : [ `Bottom
       | `Bottom_And_Right
       | `Invisible
       | `Left
       | `Left_And_Right
       | `NA
       | `Overstruck
       | `Right
       | `Top
       | `Top_And_Bottom
       | `Top_And_Bottom_And_Right
       | `Top_And_Left
       | `Top_And_Left_And_Right
       | `Top_And_Right
       | `Visual_Order_Left ] prop
val iso_comment : string prop
val jamo_short_name : string prop
val join_control : bool prop
val joining_group : [ `Ain
       | `Alaph
       | `Alef
       | `Alef_Maqsurah
       | `Beh
       | `Beth
       | `Burushaski_Yeh_Barree
       | `Dal
       | `Dalath_Rish
       | `E
       | `Farsi_Yeh
       | `Fe
       | `Feh
       | `Final_Semkath
       | `Gaf
       | `Gamal
       | `Hah
       | `Hamza_On_Heh_Goal
       | `He
       | `Heh
       | `Heh_Goal
       | `Heth
       | `Kaf
       | `Kaph
       | `Khaph
       | `Knotted_Heh
       | `Lam
       | `Lamadh
       | `Meem
       | `Mim
       | `No_Joining_Group
       | `Noon
       | `Nun
       | `Nya
       | `Pe
       | `Qaf
       | `Qaph
       | `Reh
       | `Reversed_Pe
       | `Rohingya_Yeh
       | `Sad
       | `Sadhe
       | `Seen
       | `Semkath
       | `Shin
       | `Swash_Kaf
       | `Syriac_Waw
       | `Tah
       | `Taw
       | `Teh_Marbuta
       | `Teh_Marbuta_Goal
       | `Teth
       | `Waw
       | `Yeh
       | `Yeh_Barree
       | `Yeh_With_Tail
       | `Yudh
       | `Yudh_He
       | `Zain
       | `Zhain ] prop
val joining_type : [ `C | `D | `L | `R | `T | `U ] prop
val line_break : [ `AI
       | `AL
       | `B2
       | `BA
       | `BB
       | `BK
       | `CB
       | `CJ
       | `CL
       | `CM
       | `CP
       | `CR
       | `EX
       | `GL
       | `H2
       | `H3
       | `HL
       | `HY
       | `ID
       | `IN
       | `IS
       | `JL
       | `JT
       | `JV
       | `LF
       | `NL
       | `NS
       | `NU
       | `OP
       | `PO
       | `PR
       | `QU
       | `RI
       | `SA
       | `SG
       | `SP
       | `SY
       | `WJ
       | `XX
       | `ZW ] prop
val logical_order_exception : bool prop
val lowercase : bool prop
val lowercase_mapping : [ `Cps of cp list | `Self ] prop
val math : bool prop
val name : [ `Name of string | `Pattern of string ] prop
In the `Pattern case occurences of the character '#' (U+0023) in the string must be replaced by the value of the code point as four to six uppercase hexadecimal digits (the minimal needed). E.g. the pattern "CJK UNIFIED IDEOGRAPH-#" associated to code point U+3400 gives the name "CJK UNIFIED IDEOGRAPH-3400".
val name_alias : (string * [ `Abbreviation | `Alternate | `Control | `Correction | `Figment ])
       list prop
val nfc_quick_check : [ `False | `Maybe | `True ] prop
val nfd_quick_check : [ `False | `Maybe | `True ] prop
val nfkc_quick_check : [ `False | `Maybe | `True ] prop
val nfkc_casefold : [ `Cps of cp list | `Self ] prop
val nfkd_quick_check : [ `False | `Maybe | `True ] prop
val noncharacter_code_point : bool prop
val numeric_type : [ `De | `Di | `None | `Nu ] prop
val numeric_value : [ `Frac of int * int | `NaN | `Num of int64 ] prop
val other_alphabetic : bool prop
val other_default_ignorable_code_point : bool prop
val other_grapheme_extend : bool prop
val other_id_continue : bool prop
val other_id_start : bool prop
val other_lowercase : bool prop
val other_math : bool prop
val other_uppercase : bool prop
val pattern_syntax : bool prop
val pattern_white_space : bool prop
val quotation_mark : bool prop
val radical : bool prop
type script = [ `Arab
       | `Armi
       | `Armn
       | `Avst
       | `Bali
       | `Bamu
       | `Batk
       | `Beng
       | `Bopo
       | `Brah
       | `Brai
       | `Bugi
       | `Buhd
       | `Cakm
       | `Cans
       | `Cari
       | `Cham
       | `Cher
       | `Copt
       | `Cprt
       | `Cyrl
       | `Deva
       | `Dsrt
       | `Egyp
       | `Ethi
       | `Geor
       | `Glag
       | `Goth
       | `Grek
       | `Gujr
       | `Guru
       | `Hang
       | `Hani
       | `Hano
       | `Hebr
       | `Hira
       | `Hrkt
       | `Ital
       | `Java
       | `Kali
       | `Kana
       | `Khar
       | `Khmr
       | `Knda
       | `Kthi
       | `Lana
       | `Laoo
       | `Latn
       | `Lepc
       | `Limb
       | `Linb
       | `Lisu
       | `Lyci
       | `Lydi
       | `Mand
       | `Merc
       | `Mero
       | `Mlym
       | `Mong
       | `Mtei
       | `Mymr
       | `Nkoo
       | `Ogam
       | `Olck
       | `Orkh
       | `Orya
       | `Osma
       | `Phag
       | `Phli
       | `Phnx
       | `Plrd
       | `Prti
       | `Qaai
       | `Rjng
       | `Runr
       | `Samr
       | `Sarb
       | `Saur
       | `Shaw
       | `Shrd
       | `Sinh
       | `Sora
       | `Sund
       | `Sylo
       | `Syrc
       | `Tagb
       | `Takr
       | `Tale
       | `Talu
       | `Taml
       | `Tavt
       | `Telu
       | `Tfng
       | `Tglg
       | `Thaa
       | `Thai
       | `Tibt
       | `Ugar
       | `Vaii
       | `Xpeo
       | `Xsux
       | `Yiii
       | `Zinh
       | `Zyyy
       | `Zzzz ] 
val script : script prop
val script_extensions : script list prop
val sentence_break : [ `AT
       | `CL
       | `CR
       | `EX
       | `FO
       | `LE
       | `LF
       | `LO
       | `NU
       | `SC
       | `SE
       | `SP
       | `ST
       | `UP
       | `XX ] prop
val simple_case_folding : [ `Cp of cp | `Self ] prop
val simple_lowercase_mapping : [ `Cp of cp | `Self ] prop
val simple_titlecase_mapping : [ `Cp of cp | `Self ] prop
val simple_uppercase_mapping : [ `Cp of cp | `Self ] prop
val soft_dotted : bool prop
val sterm : bool prop
val terminal_punctuation : bool prop
val titlecase_mapping : [ `Cps of cp list | `Self ] prop
val uax_42_element : [ `Char | `Noncharacter | `Reserved | `Surrogate ] prop
Not normative, artefact of Uucd. Corresponds to the XML element name that describes the code point.
val unicode_1_name : string prop
val unified_ideograph : bool prop
val uppercase : bool prop
val uppercase_mapping : [ `Cps of cp list | `Self ] prop
val variation_selector : bool prop
val white_space : bool prop
val word_break : [ `CR
       | `DQ
       | `EX
       | `Extend
       | `FO
       | `HL
       | `KA
       | `LE
       | `LF
       | `MB
       | `ML
       | `MN
       | `NL
       | `NU
       | `RI
       | `SQ
       | `XX ] prop
val xid_continue : bool prop
val xid_start : bool prop

Unihan properties

In alphabetic order. For now unihan properties are always represented as strings.

val kAccountingNumeric : string prop
val kAlternateHanYu : string prop
val kAlternateJEF : string prop
val kAlternateKangXi : string prop
val kAlternateMorohashi : string prop
val kBigFive : string prop
val kCCCII : string prop
val kCNS1986 : string prop
val kCNS1992 : string prop
val kCangjie : string prop
val kCantonese : string prop
val kCheungBauer : string prop
val kCheungBauerIndex : string prop
val kCihaiT : string prop
val kCompatibilityVariant : string prop
val kCowles : string prop
val kDaeJaweon : string prop
val kDefinition : string prop
val kEACC : string prop
val kFenn : string prop
val kFennIndex : string prop
val kFourCornerCode : string prop
val kFrequency : string prop
val kGB0 : string prop
val kGB1 : string prop
val kGB3 : string prop
val kGB5 : string prop
val kGB7 : string prop
val kGB8 : string prop
val kGradeLevel : string prop
val kGSR : string prop
val kHangul : string prop
val kHanYu : string prop
val kHanyuPinlu : string prop
val kHanyuPinyin : string prop
val kHDZRadBreak : string prop
val kHKGlyph : string prop
val kHKSCS : string prop
val kIBMJapan : string prop
val kIICore : string prop
val kIRGDaeJaweon : string prop
val kIRGDaiKanwaZiten : string prop
val kIRGHanyuDaZidian : string prop
val kIRGKangXi : string prop
val kIRG_GSource : string prop
val kIRG_HSource : string prop
val kIRG_JSource : string prop
val kIRG_KPSource : string prop
val kIRG_KSource : string prop
val kIRG_MSource : string prop
val kIRG_TSource : string prop
val kIRG_USource : string prop
val kIRG_VSource : string prop
val kJHJ : string prop
val kJIS0213 : string prop
val kJapaneseKun : string prop
val kJapaneseOn : string prop
val kJis0 : string prop
val kJis1 : string prop
val kKPS0 : string prop
val kKPS1 : string prop
val kKSC0 : string prop
val kKSC1 : string prop
val kKangXi : string prop
val kKarlgren : string prop
val kKorean : string prop
val kLau : string prop
val kMainlandTelegraph : string prop
val kMandarin : string prop
val kMatthews : string prop
val kMeyerWempe : string prop
val kMorohashi : string prop
val kNelson : string prop
val kOtherNumeric : string prop
val kPhonetic : string prop
val kPrimaryNumeric : string prop
val kPseudoGB1 : string prop
val kRSAdobe_Japan1_6 : string prop
val kRSJapanese : string prop
val kRSKanWa : string prop
val kRSKangXi : string prop
val kRSKorean : string prop
val kRSMerged : string prop
val kRSUnicode : string prop
val kSBGY : string prop
val kSemanticVariant : string prop
val kSimplifiedVariant : string prop
val kSpecializedSemanticVariant : string prop
val kTaiwanTelegraph : string prop
val kTang : string prop
val kTotalStrokes : string prop
val kTraditionalVariant : string prop
val kVietnamese : string prop
val kXHC1983 : string prop
val kWubi : string prop
val kXerox : string prop
val kZVariant : string prop

Unicode character databases


type block = (cp * cp) * string
The type for blocks. Code point range, name of the block.
type named_sequence = string * cp list
The type for named sequences. Sequence name, code point sequence.
type normalization_correction = cp * cp list * cp list * (int * int * int)
The type for normalization corrections. Code point, old normalization, new normalization, version
type standardized_variant = cp list * string * [ `Final | `Initial | `Isolate | `Medial ] list
The type for standarized variants. Code point sequence, description, when.
type cjk_radical = string * cp * cp
The type for CJK radicals. Radical number, CJK radical character, CJK unified ideograph.
type emoji_source = cp list * int option * int option * int option
The type for emoji sources. Unicode, docomo, kddi, softbank.
type t = {
   description :string;
   repertoire :props Cpmap.t;
   blocks :block list;
   named_sequences :named_sequence list;
   provisional_named_sequences :named_sequence list;
   normalization_corrections :normalization_correction list;
   standardized_variants :standardized_variant list;
   cjk_radicals :cjk_radical list;
   emoji_sources :emoji_source list;
}
The type for Unicode character databases.

Note. Absence of an optional top-level field in the database is denoted by the neutral element of its type (empty string, empty list, Cpmap.empty). This means that the module doesn't distinguish between absence of a field and presence of the field with empty data (but incurs no problems in this context).

val cp_prop : t -> cp -> 'a prop -> 'a option
cp_prop ucd cp p is the property p of the code point cp in db's repertoire, if p is in the repertoire and the property exists for p.

Decode


type src = [ `Channel of Pervasives.in_channel | `String of string ]
The type for input sources.
type decoder
The type for Unicode character database decoders.
val decoder : [< src ] -> decoder
decoder src is a decoder that inputs from src.
val decode : decoder -> [ `Error of string | `Ok of t ]
decode d decodes a database from d or returns an error.
val decoded_range : decoder -> (int * int) * (int * int)
decoded_range d is the range of characters spanning the `Error decoded by d. A pair of line and column numbers respectively one and zero based.

Basics

The database and subsets of it for Unicode 6.3.0 are available here. Databases with groups should be preferred, they maximize value sharing and improve parsing performance.

A database is decoded as follows: 

let ucd_or_die inf = try 

  let ic = if inf = "-" then stdin else open_in inf in

  let d = Uucd.decoder (`Channel ic) in 

  match Uucd.decode d with 

  | `Ok db -> db 

  | `Error e -> 

    let (l0, c0), (l1, c1) = Uucd.decoded_range d in

    Printf.eprintf "%s:%d.%d-%d.%d: %s\n%!" inf l0 c0 l1 c1 e; 

    exit 1

with Sys_error e -> Printf.eprintf "%s\n%!" e; exit 1



let ucd = ucd_or_die "/tmp/ucd.all.grouped.xml"
The convenience function Uucd.cp_prop can be used to query the property of a given code point. For example the general category of U+1F42B is given by: 
let u_1F42B_gc = Uucd.cp_prop ucd 0x1F42B Uucd.general_category