Entity recognition identifies items such as people, organizations, locations, dates, and other business-relevant information within text.
The connector automatically selects the most appropriate Amazon Comprehend processing method based on the size of the submitted content:
| Document Type | Behavior Description |
|---|---|
| Small Documents |
For smaller documents, the connector performs synchronous entity extraction and returns the results immediately. The operation for this type of documents is DetectEntities. The DetectEntities operation will be called if the supplied text file is smaller than a configurable limit, by default 5000 bytes. |
| Medium Documents | For larger documents, the connector divides the content into smaller sections and
processes the sections in batches. The operation for this type of
documents is BatchDetectEntities. The BatchDetectEntities is used if the file is larger than the DetectEntities limit although it also has a configurable limit, by default 125000 bytes. When you use the Batch API call the input file is split into chunks of less than the configurable limit, by default 5000 bytes. |
| Large Documents | For documents that exceed synchronous processing limits, the connector uses Amazon
Comprehend asynchronous processing. The StartEntitiesDetectionJob and DescribeEntititesDetectionJob are used if the input file is larger than the BatchDetectEntities limit, by default 5000 bytes. Similar to the batch approach, the input file will be divided into a set of smaller files of a certain configured size. When dividing the original file, the engine ensures that it only includes full words and does not split on a non whitespace character. During asynchronous processing, the source document is divided into manageable sections that are uploaded to Amazon S3. Then the asynchronous entity recognition job is started and the connector monitors the job status until processing completes or a timeout occurs. The results are downloaded and combined into a single result set and the temporary processing files are removed. This approach allows large documents to be analyzed while remaining within Amazon Comprehend processing limits. |