6.9 Record Cleaning and Standardisation

Cleaning and standardising records is done by defining components (like names, addresses and dates) made of one or more fields in the input data set and initialising corresponding component standardisers as explained in the previous sections, resulting in cleaned and standardised output fields. Examples of component and record standardisers were already presented in Chapter 5 and the previous sections. Here, all available arguments for the record standardiser are described and an example is given (see code below). The record standardiser is implemented in the module standardisation.py.

When a record standardiser is initialised, the following arguments must be given.

name
A name for the record standardiser. Most suitable is a short string.
description
A longer description of the record standardiser. Note that this argument is not mandatory, the standardisation process works fine without a description.
input_dataset
A reference to the input data set, from where the fields for the components are taken. Note that all input fields that are defined in component standardisers (as described in the previous sections) must be fields of this input data set.
output_dataset
A reference to the output data set, which will contain the cleaned and standardised output fields. Note that all output fields that are defined in component standardisers (as described in the previous sections) must be fields of this output data set.
component_standardisers
This must be a list of one or more component standardisers as presented in the previous sections. As shown in the examples in Chapter 5, these component standardisers must be initialised before they can be passed to a record standardiser using this argument.

Once one or more component standardisers and one record standardiser are initialised, the standardisation process can be started by loading records from the input data set, and passing them on to the record standardiser using the standardise (for one record) or standardise_block (for a list of records). The following example code shows how to initialise a record standardiser, assuming that the module standardisation.py has been imported, and both input and output data sets and several component standardisers have been initialised. Note that the standardisation is normally done within the deduplication and linkage routines as described in Section 9.6. Alternatively, if one is only interested in cleaning and standardising a data set a standardisation process can be started as shown in Section 6.10.

# ====================================================================
#
my_comp_standard = [name_rules_stand, address_hmm_stand, baby_std]

my_record_standardiser = RecordStandardiser(name = 'my-rec-stand',
                                   input_dataset = my_in_data,
                                  output_dataset = my_out_data,
                         component_standardisers = my_comp_standard)

# Load and standardise one record
#
one_record = my_in_data.read_record()

one_clean_record = my_record_standardiser.standardise(one_record)

# Load and standardise a block of records
#
records = my_in_data.read_records(0,10000)  # Load 10,000 records

clean_record_block = my_record_standardiser.standardise_block(records)