Provision new ES cluster

Create new ES domain: demo-es-domain

All the nodes contain the cluster state.

Node configuration

  • "It depends".

  • How important is CPU resources? When you're analyzing data, it depends on what the analyzers are doing...have been CPU-bound on data ingestion.

  • You can only run a maximum of ten instances per cluster. If your customer wants more, you’ll have to submit a limit increase request.

"Enable dedicated master"

  • Yes, just use small instances (3 x $15/month for 3 dedicated master)
  • The last thing you want is for a data node to be elected master and now you have master election.
  • "We don't really see heavy load on them."
  • Don't send index or search request to the dedicated master nodes
  • Not necessary for dev/test environments
  • From ES docs: "dedicated master nodes are nodes with the settings node.data: false and node.master: true. We actively promote the use of dedicated master nodes in critical clusters to make sure that there are 3 dedicated nodes whose only role is to be master, a lightweight operational (cluster management) responsibility. By reducing the amount of resource intensive work that these nodes do (in other words, do not send index or search requests to these dedicated master nodes), we greatly reduce the chance of cluster instability."

Dedicated master nodes perform the following cluster management tasks:

  • Tracks all nodes in the cluster
  • Tracks the number of indices in the cluster
  • Tracks the number of shards belonging to each index
  • Maintains routing information for nodes in the cluster
  • Updates the cluster state after state changes, such as index creation and the addition and removal of nodes in the cluster
  • Replicates changes to the cluster state across all nodes in the cluster
  • Monitors the health of all cluster nodes by sending heartbeat signals, or periodic signals that monitor the availability of the data nodes in the cluster

Because a dedicated master node does not process search and query requests, the instance type chosen for this role typically does not require a large CPU and memory

Zone awareness

  • Recommended for production clusters
  • Introduces additional network latency for the cross-AZ replication
  • Even number of instances

Storage configuration

  • Disk is a big one...especially if you do a lot of analytics. If you do ingest, disk is important, but for analytics, disk is very important.
  • After testing, Elastic would recommend using local ephemeral storage. We have a lot of customers running ES on local ephemeral.
  • Local SSD has size limits...if you start putting large amounts of data.
  • IOPS isn't the issue, it's latency that was the issue.
  • You can back up an index to HDFS (just using HDFS as a filesystem)...you could restore the backups. MapR actually integrates with ES (their version of HBase will stream their writes into ES).
  • ES have started implementing more auto-tuning.

Mention storage limitations

  • EBS volume limits

Snapshot configuration

  • Today, you need to contact the AWS Support to restore from an automated snapshot.
  • You can create a manual snapshot anytime, it will be stored in S3, and you can restore that yourself.

Access policy

  • Configuration service requests must always be signed.
  • Upload and search requests must be signed unless you configure anonymous access for those services.

Discuss storage configuration

  • rest.action.multi.allow_explicit_index
  • indices.fielddata.cache.size

Need to research what this means

Launch cluster

  • It takes about 10 minutes before the cluster is ready, and you can't run load data or run queries until it's ready.
  • "Since we don't want to wait 10 minutes, I've already provisioned another ES cluster, so let's connect to that and start adding documents."