Install Filebeat agent on App server

The Filebeat agent is implemented in Go, and is easy to install and configure. It uses the lumberjack protocol to communicate with the Logstash server.

More details from elastic.co's blog:

  • "Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash Forwarder, Filebeat tails logs and quickly sends this information to Logstash for further parsing and enrichment or to Elasticsearch for centralized storage and analysis."

  • "Our goal was to build a platform that makes it easy for our community to create new Beats. For this we developed libbeat, the Go library that contains the common parts of all Beats for dealing with common tasks like inserting in bulk into Elasticsearch, securely sending events to Logstash, load-balancing the events to multiple Logstash and Elasticsearch nodes, and sending events in synchronous and asynchronous modes. The libbeat platform also includes mechanisms for detecting when downstream servers are getting overloaded or the network in between is getting congested, so it can reduce the sending rate."

  • "Filebeat is the successor of the Logstash Forwarder, a lightweight log shipper that has been used in production by many companies for years. Logstash-Forwarder is a simple lightweight Go application that forwards all the logs of your servers to a central location for further processing.

  • Logstash-Forwarder was started by the creator of Logstash, Jordan Sissel and maintained by the Logstash developers. Unfortunately it tended to lag behind in terms of improvements and bug fixes when compared to Logstash itself. Because of the clear similarities with the Beats, we decided the best path forward was to transform the Logstash Forwarder into a Beat. So we took the Forwarder code, we split it into pieces, replaced the rusty parts, added unit tests, and then put it all back together into Filebeat."

  • "Configuring Filebeat is simple. After installing it on your servers, just configure the paths for Filebeat to crawl and it will start sending your logs to Elasticsearch via Logstash for further processing. It is intelligent enough to deal with log rotation, file renames, and the temporary unavailability of the downstream server, so you never lose a log line."

Copy the Logstash server's public key, from the local machine to each app server that will need to send logs to the Logstash server

Earlier, we had copied this public key from the Logstash server where the keypair was originally generated. Now, we need to copy this public key to each of the servers running the filebeat agent, in order for the servers to send the log data securely to the Logstash server.

From local machine: 

# create the /etc/pki/tls/certs/ directory on the remote machine if doesn't exist
ssh -i $ESTEST_INSTANCE_1_KEYPAIR ubuntu@$ESTEST_INSTANCE_1_DNS 'sudo mkdir -p /etc/pki/tls/certs/'

# scp the file from local machine to the remote machine, and rename it to the desired filename
scp -i $ESTEST_INSTANCE_1_KEYPAIR /tmp/logstash-forwarder.crt  ubuntu@$ESTEST_INSTANCE_1_DNS:/tmp/
ssh -i $ESTEST_INSTANCE_1_KEYPAIR ubuntu@$ESTEST_INSTANCE_1_DNS 'sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/'

Login to Instance 1 (Application server with App and Syslogs, and log delivery agents)

ssh -i $ESTEST_INSTANCE_1_KEYPAIR ubuntu@$ESTEST_INSTANCE_1_DNS

Install the Filebeat agent

echo "deb https://packages.elastic.co/beats/apt stable main" |  sudo tee -a /etc/apt/sources.list.d/beats.list
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get update
sudo apt-get install -y filebeat

Filebeat YAML configuration file

We will configure to send syslogs and auth.log to the Logstash server on port 5044.

# !! replace with your Logstash DNS
# example: LOGSTASH_DNS=ip-10-231-159-134.us-west-2.compute.internal
LOGSTASH_DNS={your Logstash DNS here}

# backup the default config
sudo cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak

cat << EOF > /tmp/filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/syslog
      #  - /var/log/*.log

      input_type: log

      document_type: syslog

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["$LOGSTASH_DNS:5044"]

    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
  files:
    rotateeverybytes: 10485760 # = 10MB
EOF
sudo cp /tmp/filebeat.yml /etc/filebeat/filebeat.yml

Restart the agent to pick up changes

sudo service filebeat restart

# add system startup
sudo update-rc.d filebeat defaults 95 10