7.1. Overview of NLP

The purpose of NLP is to start with free text that a human wrote and end up with structured data that a machine can deal with.

CRATE provides a high-level NLP management system. It churns through databases (typically, databases that have already been de-identified by CRATE’s anonymisation system) and sends text to one or more NLP processors. It takes the results and stashes them in an NLP output database.

NLP processors can be disparate. For example, CRATE has support for:

7.1.1. Standard NLP output columns

All CRATE NLP processors use the following output columns:

Column SQL type Description
_pk BIGINT Arbitrary PK of output record
_nlpdef VARCHAR(64) Name of the NLP definition producing this row
_srcdb VARCHAR(64) Source database name (from CRATE NLP config)
_srctable VARCHAR(64) Source table name
_srcpkfield VARCHAR(64) PK field (column) name in source table
_srcpkval BIGINT PK of source record (or integer hash of PK if the PK is a string)
_srcpkstr VARCHAR(64) NULL if the table has an integer PK, but the PK itself if the PK was a string, to deal with hash collisions.
_srcfield VARCHAR(64) Field (column) name of source text
_srcdatetimefield DATETIME Field (column) name containing the source date/time. (Added in v0.18.52.)
_srcdatetimeval DATETIME Date/time of the source field. (Added in v0.18.52.)
_crate_version VARCHAR(147) Version of CRATE that generated this NLP record, in semantic version form. (Added in v0.18.53.)
_when_fetched_utc DATETIME Date/time (in UTC) that the NLP processor fetched the record from the source database. (Added in v0.18.53.)

The length of the VARCHAR fields that refer to relational database entity names is set by the MAX_SQL_FIELD_LEN constant.

These default output columns are prefixed with an underscore to reduce the risk of name clashes (for example, with GATE NLP applications that can themselves generate arbitrary column names). Columns beginning with an underscore are a nuisance for R, though; one has to refer to them in data tables as e.g. dt$`_myfield` rather than dt$myfield.