Amazon Web Services recommend each shard should be not more than 50GB. For this example of one billion files the estimated total size of metadata to be indexed is 1.3TB. The shard size here can be set to 40GB which equals 32 shards.
On the same VPC set the number of shards:
curl -XPUT 'https://<Elasticsearch DNS>:443/alfresco?pretty' -H 'Content-Type: application/json' -d'
{
"settings" :{
"number_of_shards":32,
"number_of_replicas":0
}
}'
You can set other critical parameters such as the refresh interval and the translog flush threshold using curl. The refresh interval is the time in which indexed data is searchable and should be disabled. This is done by setting it to -1 or by setting it to a higher value during indexing to avoid the unnecessary usage of resources. The translog flush threshold is set to a higher size, for example 2GB, to avoid it periodically flushing during the indexing process.
To set the refresh interval to -1 to disable it:
curl -XPUT "https://<Elasticsearch DNS>:443/alfresco/_settings" -H 'Content-Type: application/json' -d '{ "index" : { "refresh_interval" : "-1" }}'
To set the translog flush threshold to 2GB:
curl -XPUT "https://<Elasticsearch DNS>:443/alfresco/_settings?pretty" -H 'Content-Type: application/json' -d '{"index":{"translog.flush_threshold_size" : "2GB"}}'
To verify the settings:
curl -XGET "https://<Elasticsearch DNS>:443/alfresco/_settings?pretty" -H 'Content-Type: application/json' -d '{ "index" : { "refresh_interval" }}'
To setup the Re-Indexing Instance:
Indexing Command:
nohup java -Xmx4G -jar alfresco-elasticsearch-reindexing-3.2.1-app.jar \ --server.port=<unique port> \ --alfresco.reindex.jobName=reindexByIds \ --spring.elasticsearch.rest.uris=https://<Elasticsearch DNS>:443 \ --spring.datasource.url=jdbc:postgresql://<DB Writer URL>:5432/alfresco \ --spring.datasource.username=**** \ --spring.datasource.password=**** \ --alfresco.accepted-content-media-types-cache.enabled=false \ --spring.activemq.broker-url=failover:\(ssl://<Broker 1>:61617,ssl://<Broker 2>:61617\) \ --spring.activemq.user=alfresco \ --spring.activemq.password=***** \ --alfresco.reindex.fromId=0 \ --alfresco.reindex.toId=50000000 \ --alfresco.reindex.multithreadedStepEnabled=true \ --alfresco.reindex.concurrentProcessors=30 \ --alfresco.reindex.metadataIndexingEnabled=true \ --alfresco.reindex.contentIndexingEnabled=false \ --alfresco.reindex.pathIndexingEnabled=true \ --alfresco.reindex.pagesize=10000 \ --alfresco.reindex.batchSize=1000 & Indexing Speed
The table summarizes different indexing capabilities obtained with different data volumes but with identical infrastructure and configuration as outlined above.