6.5. Run the anonymiser

Now you’ve created and edited your config file and data dictionary, you can run the anonymiser in one of the following ways:

crate_anonymise --full
crate_anonymise --incremental
crate_anonymise_multiprocess --full
crate_anonymise_multiprocess --incremental

The ‘multiprocess’ versions are faster (if you have a multi-core/-CPU computer). The ‘full’ option destroys the destination database and starts again. The ‘incremental’ one brings the destination database up to date (creating it if necessary). The default is ‘incremental’, for safety reasons.

Get more help with

crate_anonymise --help

6.5.1. crate_anonymise

This runs a single-process anonymiser.

Options:

usage: crate_anonymise [-h] [--config CONFIG] [--version] [--verbose]
                       [-i | -f] [--skipdelete] [--dropremake] [--drop_all]
                       [--optout] [--nonpatienttables] [--patienttables]
                       [--index] [--restrict RESTRICT]
                       [--limits LIMITS LIMITS] [--file FILE]
                       [--list LIST [LIST ...]]
                       [--free_text_limit FREE_TEXT_LIMIT] [--excludescrubbed]
                       [--process [PROCESS]] [--nprocesses [NPROCESSES]]
                       [--processcluster PROCESSCLUSTER] [--skip_dd_check]
                       [--seed SEED] [--chunksize [CHUNKSIZE]]
                       [--reportevery [REPORTEVERY]] [--debugscrubbers]
                       [--savescrubbers] [--echo]

Database anonymiser. (CRATE version 0.19.4, 2022-05-24. Created by Rudolf
Cardinal.)

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       Config file (overriding environment variable
                        CRATE_ANON_CONFIG). (default: None)
  --version             show program's version number and exit
  --verbose, -v         Be verbose (default: False)

Mode options:
  -i, --incremental     Process only new/changed information, where possible.
                        (default: True)
  -f, --full            Drop and remake everything. (default: False)
  --skipdelete          For incremental updates, skip deletion of rows present
                        in the destination but not the source. (default:
                        False)

Action options (default is to do all, but if any are specified, only those are done):
  --dropremake          Drop/remake destination tables, and admin tables
                        except opt-out tables. (default: False)
  --drop_all            Drop all destination tables known to the data
                        dictionary, and all admin tables, then stop. (May also
                        be helpful in revealing leftover tables in the
                        destination database, e.g. if the data dictionary has
                        changed.) (default: False)
  --optout              Update opt-out list in administrative database.
                        (default: False)
  --nonpatienttables    Process non-patient tables only. (default: False)
  --patienttables       Process patient tables only. (default: False)
  --index               Create indexes only. (default: False)

Restriction options:
  --restrict RESTRICT   Restrict which patients are processed. Specify which
                        field to base the restriction on or 'pid' for patient
                        ids. (default: None)
  --limits LIMITS LIMITS
                        Specify lower and upper limits of the field specified
                        in '--restrict'. (default: None)
  --file FILE           Specify a file with a list of values for the field
                        specified in '--restrict'. (default: None)
  --list LIST [LIST ...]
                        Specify a list of values for the field specified in '
                        --restrict'. (default: None)
  --free_text_limit FREE_TEXT_LIMIT
                        Filter out all free text fields over the specified
                        length. For example, if you specify 200, then
                        VARCHAR(200) fields will be permitted, but
                        VARCHAR(200), or VARCHAR(MAX), or TEXT (etc., etc.)
                        fields will be excluded. (default: None)
  --excludescrubbed     Exclude all text fields which are being scrubbed.
                        (default: False)

Processing options:
  --process [PROCESS]   For multiprocess mode: specify process number.
                        (default: 0)
  --nprocesses [NPROCESSES]
                        For multiprocess mode: specify total number of
                        processes (launched somehow, of which this is to be
                        one). (default: 1)
  --processcluster PROCESSCLUSTER
                        Process cluster name (used as part of log name).
                        (default: )
  --skip_dd_check       Skip data dictionary validity check. (default: False)
  --seed SEED           String to use as the basis of the seed for the random
                        number generator used for the transient integer RID
                        (TRID). Leave blank to use the default seed (system
                        time). (default: None)
  --chunksize [CHUNKSIZE]
                        Number of records copied in a chunk when copying PKs
                        from one database to another. (default: 100000)

Reporting and debugging:
  --reportevery [REPORTEVERY]
                        Report insert progress every n rows in verbose mode.
                        (default: 100000)
  --debugscrubbers      Report sensitive scrubbing information, for debugging.
                        (default: False)
  --savescrubbers       Saves sensitive scrubbing information in admin
                        database, for debugging. (default: False)
  --echo                Echo SQL. (default: False)

6.5.2. crate_anonymise_multiprocess

This runs multiple copies of crate_anonymise in parallel.

Options:

usage: crate_anonymise_multiprocess [-h] [--nproc [NPROC]] [--verbose]

Runs the CRATE anonymiser in parallel. Version 0.19.4 (2022-05-24). Note that
all arguments not specified here are passed to the underlying script (see
crate_anonymise --help).

optional arguments:
  -h, --help            show this help message and exit
  --nproc [NPROC], -n [NPROC]
                        Number of processes (default is the number of CPUs on
                        this machine)
  --verbose, -v         Be verbose

6.5.3. crate_anon_show_counts

This ancillary tool prints record counts from your source and destination databases.

usage: crate_anon_show_counts [-h] [--config CONFIG] [--verbose]

Print record counts from source/destination databases. (CRATE version 0.19.4,
2022-05-24. Created by Rudolf Cardinal.)

optional arguments:
  -h, --help       show this help message and exit
  --config CONFIG  Config file (overriding environment variable
                   CRATE_ANON_CONFIG). (default: None)
  --verbose, -v    Be verbose (default: False)

6.5.4. crate_anon_check_text_extractor

This ancillary tool checks that you have the text extraction software that you might want. See third-party text extractors.

usage: crate_anon_check_text_extractor [-h]
                                       [checkextractor [checkextractor ...]]

Check availability of tools to extract text from different document formats.
(CRATE version 0.19.4, 2022-05-24. Created by Rudolf Cardinal.)

positional arguments:
  checkextractor  File extensions to check for availability of a text
                  extractor. Try, for example, '.doc .docx .odt .pdf .rtf .txt
                  None' (use a '.' prefix for all extensions, and use the
                  special extension 'None' to check the fallback processor).
                  (default: None)

optional arguments:
  -h, --help      show this help message and exit

6.5.5. crate_anon_summarize_dd

This ancillary tool reads your data dictionary and summarizes facts about each table. It may be helpful to find problems with large data dictionaries.

usage: crate_anon_summarize_dd [-h] [--config CONFIG] [--verbose]
                               [--output OUTPUT]

Draft a data dictionary for the anonymiser. (CRATE version 0.19.4, 2022-05-24.
Created by Rudolf Cardinal.)

optional arguments:
  -h, --help       show this help message and exit
  --config CONFIG  Config file (overriding environment variable
                   CRATE_ANON_CONFIG). (default: None)
  --verbose, -v    Be verbose (default: False)
  --output OUTPUT  File for output; use '-' for stdout. (default: -)