Using AWS ElasticSearch Service with Heartbeat (Uptime Monitor)

AWS ElasticSearch Service do not have the Uptime App, but it is possible to point a Heartbeat client and point it to AWS ES Service to monitor site uptime. Assuming you have AWS ES up and running, just do the following:

1. Install the OSS version of the Heartbeat Client.

   curl -L -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-oss-7.1.1-amd64.deb
   sudo dpkg -i heartbeat-7.1.1-amd64.deb
   

   Point to proper version as needed, the example above uses version 7.1.1. It is crucial that you install the OSS version, if you use the regular version, you will get the following errors from the Heartbeat Client:

   Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials
   

2. Configure the configuration as needed (/etc/heartbeat/heartbeat.yml), setup the endpoints that needs monitoring, point output.elasticsearch to the AWS ES VPC endpoint.

   Do not forget to add the following to heartbeat.yml

   setup.ilm.enabled: false
   

   Failing to do so will cause the following:

   Connection marked as failed because the onConnect callback failed: request checking for ILM availability failed: 401 Unauthorized: {"Message":"Your request: '/_xpack' is not allowed."}
   

   Once done, start the service.

   sudo heartbeat setup
   sudo service heartbeat-elastic start
   

3. Setup the Heartbeat index in Kibana.

   Go to your AWS ES Kibana Page -> Management -> Index Patterns and hit "Create Index Patterns". Type "heartbeat-*" in the index pattern and setup the index as needed. The output will be similar to the following:

4. Now you can setup Visualization from the Heartbeat index.