-
The recommended way to set up the
sync.cluster.interfaceproperty is in theconfig.ymlfile. This specifies a particular network interface that Hazelcast uses for clustering. It could be a wildcard value, such as10.256.*.*, which means an attempt is made to bind with an interface having an IP address that begins with10.256:sync: cluster: interface: 10.256.*.*By doing this, all the dynamically started Sync instances will be able to see each other and form a cluster successfully.
-
To run various operations on a Sync Service instance via JMX (for example, checking the number of instances in the cluster, running a cluster wide check to ensure that each member sees each other, shutting down the Sync Service etc.), you'll need to connect from a bastion host (i.e. a machine located in the same network as the instance). This is required because the
syncservice.shscript that's embedded in theAlfrescoSyncServer-3.0.x.zipdistribution zip doesn't specify a value forjava.rmi.server.hostname. By doing this, JMX will bind to the private IP, allowing Sync instances to be started dynamically by AWS Auto Scaling groups. - The last and most important aspect is to ensure that Sync Service is shut down gracefully, by running the
./syncservice.sh stopcommand. This will prevent hanging database connections. In addition, this will allow the current sync instance to deregister from the database.
When Sync Service starts, Hazelcast tries to find the other cluster members by performing connections with IPs of previously registered clustered members stored in the database. By deregistering a soon to be dead cluster member from the database means a smaller number of connections will be attempted. This should result in faster startup times for Hazelcast and inherently the Sync Service.
Follow the steps listed in the aws-lambda-lifecycle-hooks-function GitHub project to correctly run ./syncservice.sh stop when the instance is terminated, where you'll be using Auto Scaling lifecycle hooks, Lambda, and the EC2 Run Command.
Note
You'll need to modify some of the steps in aws-lambda-lifecycle-hooks-function:
-
Step 1 - Create an SNS topic to receive the result of the backup
Skip this step.
-
Step 3 - Create an Auto Scaling group and configure the lifecycle hook
-
Instead of using the User Data to specify the script to install the AWS Systems Manager (SSM) agent, you can manually install it, and then only afterwards, create the necessary AMI.
See Manually Install SSM Agent on Amazon EC2 Linux Instances to identify the steps depending on the Linux flavor.
-
Add the lifecycle hooks from the user interface.
-
Add a lifecycle hook from the Auto Scaling Groups EC2 console.
Complete the following fields:
- Lifecycle Transition: Instance Terminate
-
Heartbeat Timeout: 300 seconds
This defines the amount of time it takes from when the hook is invoked till the instance is terminated -
300sshould be more than enough for the./syncservice.sh stopcommand to run.
-
-
Step 4 - Create an S3 bucket for files
Skip this step.
-
Step 5 - Create the SSM document
The EC2 Systems Manager Shared Resources Document can be simplified to just execute
./syncservice.sh stop.There's no need to bother with adding code to complete the lifecycle hook, since this is done inside the Python lambda (see the
abandon_lifecyclemethod).{ "schemaVersion": "1.2", "description": "Shuts down Sync Service", "parameters": {}, "runtimeConfig": { "aws:runShellScript": { "properties": [ { "id": "0.aws:runShellScript", "runCommand": [ "#!/usr/bin/env bash", "sudo /home/ubuntu/service-sync/syncservice.sh stop" ] } ] } } } -
Step 6 - Create the Lambda function
Make one small change to the Python script.
-
In
def check_command(command_id, instance_id), before checking for the command execution status:response_iterator['CommandInvocations'][0]['Status'] -
Check if the list is empty. For example:
if not len(response_iterator['CommandInvocations']) == 0:
-