The Data Curation subsystem performs the following:
-
Format Conversion - for example, convert PSD to JPEG
-
Text Extraction - convert files (MS Office, PDF, HTML) to text and generate image descriptions or audio transcripts
-
Text Splitting - split documents into semantically meaningful chunks and leverage Markdown structure or adjust chunking heuristics
-
Embedding Generation - use appropriate models to convert chunks into vectors
Data Curation uses DocumentFilter, open-source libraries, and services like AWS Textract.