9.2.5 Encoded String Comparison 'FieldComparatorEncodeString'

9.2.5 Encoded String Comparison '`FieldComparatorEncodeString`'

Phonetic name encoding is traditionally used to create blocking variables in the record linkage process, but it can also be used to compare strings. Several algorithms for phonetic encoding are implemented in the encode.py module.

The encoded string comparison function compares the two fields (or field lists) given to it as encoded strings, and returns the agreement weight if both strings are encoded the same way, otherwise the disagreement weight is returned.

The encoding method has to be selected with the encode_method argument. The following methods are currently implemented in Febrl:

soundex
The Soundex encoding keeps the original first character of a string and calculates a 3-digit code of the remainder of the string.
mod_soundex
Based on the original Soundex encoding this modified encoding algorithm uses a different mapping of letters into numbers.
nysiis
The NYSIIS (New York State Identification Intelligence System) encoding returns a code that contains only characters.
phonex
Phonex [21] is a variation of Soundex and NYSIIS which tries to improve the encoding quality by pre-processing names before they are encoded. Similar to Soundex the code consists of a leading letter followed by numbers.
dmetaphone
The more recently developed algorithm Double-Metaphone [28] attempts to better account for non-English words, like European and Asian names. Similar to NYSIIS, Double-Metaphone returns a code only consisting of letters. In general, Double-Metaphone seems to be closer to the correct pronunciation of names than NYSIIS.

All of these encodings are particularly sensitive to errors in the first letter of a string. Therefore, an additional argument to the encoded string comparator is reverse which can be either set to False or True. In the latter case, the strings are reversed first before they are encoded. The default value for reverse is False.

The maximum length of the codes calculated can be set with the argument max_code_length, which has a default value of 4.

If a frequency table is given for this field comparator, the agreement weight will be calculated using the frequency of the value in the input fields that are compared, as described in Section 9.2.1. Frequency dependent weights will also be used to calculate a partial agreement weight.