Summary

This web page explains the difference between Unicode normalization forms NFD, NFC, NFKD, and NFKC with Python code examples, focusing on their usage in Japanese text normalization.

Abstract

The web page discusses the need for normalizing special characters, particularly in Japanese, due to the presence of multiple forms of characters like Latin's full-width and half-width. It introduces four Unicode normalization forms (NFD, NFC, NFKD, and NFKC) and explains how they differ in terms of original form changes and character length changes. The web page provides Python code using the unicodedata library to demonstrate these forms and recommends using NFKC for text normalization tasks. It also includes references for further reading on the topic.

Bullet points

Normalizing special characters is crucial for NLP tasks, especially in Japanese, which has multiple forms of the same character.
Unicode provides four normalization forms: NFD, NFC, NFKD, and NFKC, which differ in original form changes and character length changes.
NFC and NFD preserve the original form but differ in character length, while NFKC and NFKD change the original form and character length.
NFD and NFKD decompose each Unicode character into two, while NFC and NFKC compose separated Unicode characters together.
The recommended normalization form for text normalization tasks is NFKC.
Python's unicodedata library provides functions to apply these normalization forms.
References are provided for further reading on the topic.

Difference Between NFD, NFC, NFKD, and NFKC Explained with Python Code

The difference between Unicode normalization forms

Recently I am working on an NLP task in Japanese, one problem is to convert special characters to a normalized form. So I have done a little research and write this post for anyone who has the same need.

Japanese contains different forms of the character, for example, Latin has two forms, full-width form, and half-width.

In the above example, we can see the full-width form is very ugly and is also hard to utilizing for the following processing. So we need to convert it to a normalized form.

TL;DR

Use NFKC method.

>>> from unicodedata import normalize
>>> s = "株式会社ＫＡＤＯＫＡＷＡ Ｆｕｔｕｒｅ Ｐｕｂｌｉｓｈｉｎｇ"
>>> normalize('NFKC', s)
株式会社KADOKAWA Future Publishing

Unicode normalization forms

There are 4 kinds of Unicode normalization forms. This article give a very detailed explanation. But I will explain the difference with a simple and easy understanding way.

First, we could see the below result for an intuitive understanding.

ｱｲｳｴｵ ==(NFC)==> ｱｲｳｴｵ
ｱｲｳｴｵ ==(NFD)==> ｱｲｳｴｵ
ｱｲｳｴｵ ==(NFKC)==> アイウエオ
ｱｲｳｴｵ ==(NFKD)==> アイウエオ
パピプペポ ==(NFC)==> パピプペポ
パピプペポ ==(NFD)==> パピプペポ
パピプペポ ==(NFKC)==> パピプペポ
パピプペポ ==(NFKD)==> パピプペポ
ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ ==(NFC)==> ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ
ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ ==(NFD)==> ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ
ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ ==(NFKC)==> パピプペポ
ﾊﾟﾋﾟﾌﾟﾍﾟﾎﾟ ==(NFKD)==> パピプペポ
ａｂｃＡＢＣ ==(NFC)==> ａｂｃＡＢＣ
ａｂｃＡＢＣ ==(NFD)==> ａｂｃＡＢＣ
ａｂｃＡＢＣ ==(NFKC)==> abcABC
ａｂｃＡＢＣ ==(NFKD)==> abcABC
１２３ ==(NFC)==> １２３
１２３ ==(NFD)==> １２３
１２３ ==(NFKC)==> 123
１２３ ==(NFKD)==> 123
＋－．～）｝ ==(NFC)==> ＋－．～）｝
＋－．～）｝ ==(NFD)==> ＋－．～）｝
＋－．～）｝ ==(NFKC)==> +-.~)}
＋－．～）｝ ==(NFKD)==> +-.~)}

There are two classification methods for these 4 forms.

# 1 original form changed or not
- A(not changed): NFC & NFD
- B(changed): NFKC & NFKD

# 2 the length of original length changed or not
- A(not changed): NFC & NFKC
- B(changed): NFD & NFKD

1 Whether the original form is changed or not

ａｂｃＡＢＣ ==(NFC)==> ａｂｃＡＢＣ
ａｂｃＡＢＣ ==(NFD)==> ａｂｃＡＢＣ
ａｂｃＡＢＣ ==(NFKC)==> abcABC
ａｂｃＡＢＣ ==(NFKD)==> abcABC

# 1 original form changed or not
- A(not changed): NFC & NFD
- B(changed): NFKC & NFKD

The first classification method is based on whether the original form is changed or not. More specifically, A group does not contain K but B group contains K. What does K means?

D = Decomposition 
C = Composition
K = Compatibility

K means compatibility, which is used to distinguish with the original form. Because K changes the original form, so the length is also changed.

>>> s= '…'
>>> normalize('NFKC', s)
'...'

>>> len(s)
1
>>> len(normalize('NFC', s))
1
>>> len(normalize('NFKC', s))
3
>>> len(normalize('NFD', s))
1
>>> len(normalize('NFKD', s))
3

2 Whether the length of original form is changed or not

パピプペポ ==(NFC)==> パピプペポ
パピプペポ ==(NFD)==> パピプペポ
パピプペポ ==(NFKC)==> パピプペポ
パピプペポ ==(NFKD)==> パピプペポ

# 2 the length of original length changed or not
- A(not changed): NFC & NFKC
- B(changed): NFD & NFKD

This second classification method is based on whether the length of the original form is changed or not. A group contains C(Composition), which won’t change the length. B group contains D(Decomposition), which will change the length.

You might be wondering why the length is change? Please see the test below.

>>> from unicodedata import normalize
>>> s = "パピプペポ"
>>> len(s)
5
>>> len(normalize('NFC', s))
5
>>> len(normalize('NFKC', s))
5
>>> len(normalize('NFD', s))
10
>>> len(normalize('NFKD', s))
10

We can find the “decomposition” method doubles the length.

This is because the NFD & NFKD decompose each Unicode character into two Unicode characters. For example, ポ(U+30DD) = ホ(U+30DB) + Dot(U+309A) . So the length change from 5 to 10. NFC & NFKC compose separated Unicode characters together, so the length is not changed.

Python Implementation

You can use the unicodedata library to get different forms.

>>> from unicodedata import normalize
>>> s = "パピプペポ"
>>> len(s)
5
>>> len(normalize('NFC', s))
5
>>> len(normalize('NFKC', s))
5
>>> len(normalize('NFD', s))
10
>>> len(normalize('NFKD', s))
10

Length

Take Away

Usually, we can use either of NFKC or NFKD to get the normalized form. The length won’t make trouble only if your NLP task is length sensitive. I usually use the NFKC method.

Check out my other posts on Medium with a categorized view! GitHub: BrambleXu LinkedIn: Xu Liang Blog: BrambleXu