Author Archive

Querying Logstash Data in Elasticsearch Using Curl and JQ

Thursday, August 8th, 2019
Today I came across the need to query logstash logs directly from elasticsearch. Here is a quick example of how to do so using curl and jq.

logstash-server:~$ cat
curl -XGET 'localhost:9200/_search?pretty&size=10000' -d '
    "query": {
        "query_string" : {
            "query" : "facility:19,local3 AND @timestamp:[2019-08-04T03:00 TO 2019-08-04T03:15]"
    "sort": ["@timestamp"]
} '

logstash-server:~$ bash | jq '.hits.hits[]._source | {timestamp,host,level,message}' | head -20

Elasticsearch Storage Tiering – Index to SSD then Archive to Spinning Disk

Friday, August 5th, 2016

Elasticsearch Storage Tiering – Overview

In this article I will explain how to configure Elasticsearch storage tiering. For the purposes of this example let’s say that you need to provide an ELK (Elasticsearch, Logstash, Kibana) service with logs online for 90 days. The majority of our queries are for events within the last 5 days, so we will keep 1 week on expensive fast servers with SSD storage and age older events off to cheap and slow near-line spinning disk systems.

Let’s get started!

Set Elasticsearch node metadata

First we will set a disktype metadata attribute on each of our Elasticsearch nodes. We will tag systems with SSD disk as disktype:ssd and systems with spinning disk as disktype:hdd.

SSD disk “ssd” system configuration example:

node.disktype: ssd

Spinning disk “hdd” system configuration example:

node.disktype: hdd

Once these values have been set perform a rolling restart of your cluster to apply the disktype metadata attribute.

Note: You may want to look into delayed shard allocation to speed up and reduce resource utilization during rolling maintenance

Configure Logstash index template to require SSD nodes.

Now that our Elasticsearch nodes are tagged according to their storage type we can configure Logstash to write new indicies to SSD. We’ll accomplish this by adding an index shard allocation filter to the Elasticsearch template used by Logstash when creating a new logstash-YYYY.MM.DD index.

First, we need to ensure that our template changes will not be overwritten by Logstash. Since the template could overwritten by Logstash we need to set template_overwrite to false in the Elasticsearch output section of our Logstash configuration.

# logstash.conf
output {
  elasticsearch {
    # your Elasticsearch output section here
    # add the following line
    template_overwrite => false

Note: if you have a more complicated Logstash with multiple output sections ensure that you apply this configuration to all relevant elasticsearch output sections.

Now we’re ready to proceed with a Logstash template update. The Logstash Elasticsearch index template is used each day when Logstash creates a new index (logstash-YYYY.MM.DD). We are going to add the Elasticsearch index shard allocation filter setting “index.routing.allocation.require.disktype” to the template. This will require new Logstash indicies to reside on only nodes with “ssd” disktype.

"index.routing.allocation.require.disktype" : "ssd"

To add this setting this you can either pull your running template and merge the above shard allocation filter setting, or use the example supplied below which was created using a vanilla logstash-2.3 template.

To merge, you can pull your active Logstash template like this:

curl -XGET localhost:9200/_template/logstash?pretty

Here is a full example Logstash template with ssd shard allocation setting included. This was created using a vanilla Logstash 2.3 template and includes the “routing.allocation.require.disktype” : “ssd” setting.

curl -XPUT http://localhost:9200/_template/logstash -d'
    "template" : "logstash-*",
    "settings" : {
      "index" : {
        "refresh_interval" : "5s",
        "routing.allocation.require.disktype" : "ssd"
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [ {
          "message_field" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "fielddata" : {
                "format" : "disabled"
              "type" : "string"
            "match_mapping_type" : "string",
            "match" : "message"
        }, {
          "string_fields" : {
            "mapping" : {
              "index" : "analyzed",
              "omit_norms" : true,
              "fielddata" : {
                "format" : "disabled"
              "type" : "string",
              "fields" : {
                "raw" : {
                  "index" : "not_analyzed",
                  "ignore_above" : 256,
                  "type" : "string"
            "match_mapping_type" : "string",
            "match" : "*"
        } ],
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "location" : {
                "type" : "geo_point"
              "longitude" : {
                "type" : "float"
              "latitude" : {
                "type" : "float"
              "ip" : {
                "type" : "ip"
          "@version" : {
            "index" : "not_analyzed",
            "type" : "string"
        "_all" : {
          "enabled" : true,
          "omit_norms" : true
    "aliases" : { }

You can read more about Elasticsearch shard allocation filters at

Configure Curator to move indicies to spinning disk over time.

To move indicies from SSD to spinning disk as they age we will use Curator. We also will use curator to remove indicies that are older than 90 days. Curator will run out of cron on a daily basis.

#Move indicies older than 7 days to spinning disk
curator --host localhost allocation --rule disktype=hdd indices --older-than 7 --time-unit days --timestring '%Y.%m.%d' --prefix logstash
#Remove indicies older than 90 days
curator --host localhost delete indices --older-than 90 --time-unit days --timestring '%Y.%m.%d' --prefix logstash

Note: this is curator v3.5 syntax. Full Curator documentation is available from Elastic at


You should start seeing new indices created on SSD nodes and moved off to spinning HDD after 7 days. You can use a tool like Marvel to monitor your Elasticsearch cluster and visualize shard and index allocation.

RHEL Interface Route – Interface Routing on CentOS / RHEL

Wednesday, October 29th, 2014


In my environment I have several RHEL systems with multiple network interfaces. Each interface is connected to a separate network. To avoid asynchronous routing and ensure that traffic to each network is sent via the correct interface (instead of to the default gateway) I have defined an interface route. Here’s how it works.

In this example we will configure an interface route to send traffic to out the eth1 interface.

These instructions are valid for RHEL, CentOS and Oracle Linux systems.

Adding the interface route

First, add a route statement to the /etc/sysconfig/network-scripts/route-eth1 file. If this file doesn’t exist you can safely create it.

The below statement will send traffic to the subnet via the eth1 interface. A null gateway ( is used because this traffic will stay local to the LAN eth1 is connected to.

#/etc/sysconfig/network-scripts/route-eth1 via dev eth1

Activating the interface route

In order to apply the route statement added above the interface needs to be brought down and back up.

# Bring down eth1 using network scripts
[root@host network-scripts]# ifdown eth1
# Bring up eth1 using network scripts
[root@host network-scripts]# ifup eth1

Verifying the rhel interface route

Now that the new route configuration has been applied we can verify that the configuration is active.

First, check the routing table for the interface route. We want to make sure that a route statement for via eth1 is present.

[root@host network-scripts]# netstat -nr | grep
# Check routing table for subnet that is interface routed.
# Added heading for readability
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface   U         0 0          0 eth1

Here you can see that the network ( and /24 mean the same thing) is now routed via interface eth1 with no gateway specified.

Then, confirm that traffic is flowing correctly using tcpdump. You should see traffic from only leaving via the eth1 interface.

# Listen on eth1 for traffic in subnet
[root@host ~]# tcpdump -n -i eth1 net

Using the above command you should only see outbound traffic to You may need to generate some test traffic. Using the same tcpdump filter on other interfaces should show no outbound traffic to this subnet.

Docker CIFS – How to Mount CIFS as a Docker Volume

Friday, October 24th, 2014

Docker CIFS Samba


In my environment I’m using a drobo CIFS NAS storage device. Data on the NAS is exported over CIFS to various servers. I have a few docker containers that need read/write access to the data stored on CIFS. However, when attempting to use the docker -v option to bind mount the cifs share into the docker container I ran into “permission denied” issues caused by SELinux and the CIFS mount UID/GID mapping. Here’s how I solved them.

In this article we will

  • Mount a CIFS share on the host OS with the appropriate SELinux context and UID/GID mapping for Docker.
  • Create a Docker container that uses our CIFS share as a data volume with read/write access.
  • SELinux Configuration

    The docker container will not be permitted to access your CIFS mount point unless you or mount the share with the appropriate SELinux security context.

    The context used to make this work on my CentOS 7 host is:


    Here’s an example of the fstab entry to mount the CIFS share on /mnt/cifs_share:

    //        /mnt/cifs_share cifs    defaults,username=USER,password=PASS,context=system_u:object_r:svirt_sandbox_file_t:s0      0 0

    Mapping Local User Permissions to CIFS Share Permissions

    Unlike NFS, a CIFS mount handles user authentication at mount time. This means that all files and directories at our CIFS mount point will be owned by the same user/group. This may be a problem if the user of the process running in your container and the user of the CIFS mount are not the same. I worked around this by mounting the same CIFS share multiple times with different uid and gid arguments.

    To mount a CIFS share as a particular user use the uid, gid, forceuid and forcegid mount.cifs options.

    # from 'man mount.cifs'
    uid=arg - sets the uid that will own all files or directories on the mounted filesystem when the server does not provide ownership information. It may be specified as either a username or a numeric uid. When not specified, the default is uid 0. The mount.cifs helper must be at version 1.10 or higher to support specifying the uid in non-numeric form. See the section on FILE AND DIRECTORY OWNERSHIP AND PERMISSIONS below for more information.
    forceuid -instructs the client to ignore any uid provided by the server for files and directories and to always assign the owner to be the value of the uid= option.
    gid=arg - sets the gid that will own all files or directories on the mounted filesystem when the server does not provide ownership information. It may be specified as either a groupname or a numeric gid. When not specified, the default is gid 0. The mount.cifs helper must be at version 1.10 or higher to support specifying the gid in non-numeric form.
    forcegid - instructs the client to ignore any gid provided by the server for files and directories and to always assign the owner to be the value of the gid= option.


    Here’s an example FSTAB entry to mount CIFS share as local user “nobody” at path /mnt/cifs_share-nobody

    #mount CIFS share as local user "nobody" at path /mnt/cifs_share-nobody
    //        /mnt/cifs_share-nobody cifs  username=USER,password=PASS,uid=nobody,forceuid,gid=nobody,forcegid,context=system_u:object_r:svirt_sandbox_file_t:s0

    Creating the Docker CIFS enabled container

    Now that we have the CIFS share mounted as the appropriate user with the necessary SELinux context on the host OS we can use it in our Docker container as a data volume with the “docker run -v” option.

    #from 'man docker-run'
                  -v, --volume=volume[:ro|:rw]
                     Bind mount a volume to the container.
                  The -v option can be used one or more times to add one or more mounts to a container. These mounts can then be used in other containers using the --volumes-from option.
                  The  volume  may be optionally suffixed with :ro or :rw to mount the volumes in read-only or read-write mode, respectively. By default, the volumes are mounted read-write.
                  See examples.


    Now, let’s mount our “nobody” CIFS share as /data in an example ubuntu Docker container that runs bash.

    docker run -it -v /mnt/cifs_share-nobody:/data 
    # Create a test file from the Host OS
    [root@dockerbox ~]# touch /mnt/cifs_share-nobody/testfile
    # Run an ubuntu:trusty container mounting our CIFS volume as /data inside
    [root@dockerbox ~]# docker run -it -v /mnt/cifs_share-nobody:/data ubuntu:trusty /bin/bash
    # Access our CIFS share from inside the container
    root@c90e78c4f38e:/# ls /data

    And there you have it. Access to a CIFS share from inside a Docker container.

    More about CIFS Permissions from the mount.cifs man page:

    File And Directory Ownership And Permissions
    The core CIFS protocol does not provide unix ownership information or mode for files and directories. Because of this, files and directories will generally appear to be owned by whatever values the uid= or gid= options are set, and will have permissions set to the default file_mode and dir_mode for the mount. Attempting to change these values via chmod/chown will return success but have no effect.
    When the client and server negotiate unix extensions, files and directories will be assigned the uid, gid, and mode provided by the server. Because CIFS mounts are generally single-user, and the same credentials are used no matter what user accesses the mount, newly created files and directories will generally be given ownership corresponding to whatever credentials were used to mount the share.
    If the uid's and gid's being used do not match on the client and server, the forceuid and forcegid options may be helpful. Note however, that there is no corresponding option to override the mode. Permissions assigned to a file when forceuid or forcegid are in effect may not reflect the the real permissions.
    When unix extensions are not negotiated, it's also possible to emulate them locally on the server using the "dynperm" mount option. When this mount option is in effect, newly created files and directories will receive what appear to be proper permissions. These permissions are not stored on the server however and can disappear at any time in the future (subject to the whims of the kernel flushing out the inode cache). In general, this mount option is discouraged.
    It's also possible to override permission checking on the client altogether via the noperm option. Server-side permission checks cannot be overriden. The permission checks done by the server will always correspond to the credentials used to mount the share, and not necessarily to the user who is accessing the share.

    X Forwarding Sudo SSH Session

    Tuesday, July 29th, 2014

    Many environments make use of sudo to delegate access to administrative or application user accounts. This can make ssh X forwarding tricky because your environment changes as you sudo to another user. This is a workaround that will allow you to continue your ssh X forwarding session while running sudo as another user.

    How to SSH X Forward Sudo

    First you will need a working X11 environment on your local system. There are a number of different options available depending on your OS.

    Once you have your local X11 environment working and tested you’re ready to proceed x forwarding sudo.

    you@local$ ssh -XC server
    you@server$ xauth list
    you@server$ sudo su - otheruser
    otheruser@server$ xauth add [paste output from "xauth list"]
    otheruser@server$ xterm (or other X application)

    For more information please refer to the xauth, sudo and ssh man pages.

    For x11 on a non-linux OS:

    Xming for windows
    Xquartz for OSX

    Fixing rpc.statd is not running but is required for remote locking

    Thursday, July 24th, 2014

    If you come across this error while attempting to mount an NFS filesystem it means that the statd process is not running.

    # mount -a -t nfs
    mount.nfs: rpc.statd is not running but is required for remote locking.
    mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
    mount.nfs: an incorrect mount option was specified

    Here’s how to fix the rpc.statd is not running error on el6

    First, ensure that rpcbind is running and that it is set to start on boot

    # /etc/init.d/rpcbind start
    Starting rpcbind:                                          [  OK  ]
    # chkconfig rpcbind on

    Then, start the nfslock service

    # /etc/init.d/nfslock start
    Starting NFS statd:                                        [  OK  ]
    # chkconfig nfslock on

    That’s it. You should now be able to mount your nfs filesystem.

    How to fix vyatta protocol daemon is not running

    Monday, July 21st, 2014

    The problem: protocol daemon is not running

    While attempting to commit a change on a new vyatta router install I ran into this error:

    vyatta@host# commit
    [[interfaces bonding bond0]] failed
    [ interfaces ethernet eth1 ]
    % Protocol daemon is not running
    [[interfaces ethernet eth1]] failed
    [ interfaces ethernet eth0 ]
    % Protocol daemon is not running
    [[interfaces ethernet eth0]] failed
    [[interfaces ethernet eth0 smp_affinity]] failed
    [[interfaces ethernet eth0 bond-group]] failed
    [[interfaces ethernet eth1 bond-group]] failed
    [[interfaces ethernet eth1 smp_affinity]] failed
    [[interfaces bonding bond0 address]] failed
    [ protocols static route next-hop ]
    % Protocol daemon is not running
    [[protocols static]] failed
    Commit failed

    The solution

    It turns out that the system had not fully initialized itself on boot. Lucikly the fix is easy. A simple restart of the vyatta-unicast service resolved the issue.

    sudo /etc/init.d/vyatta-unicast restart

    How to manage linux bonding without ifenslave using sysfs

    Wednesday, July 16th, 2014

    While working with a vyatta linux system I encountered a situation where bonded network interfaces were configured but the server didn’t have the ifenslave tool installed. As it turns out, linux bonded interfaces are easily managed directly through a simple sysfs interface. Here’s a quick run down of how to manage linux bonding without ifenslave using sysfs.

    Create a new bond interface called bond0 using eth0 and eth1

    #modprobe bond
    echo "+bond0" >  /sys/class/net/bonding_masters
    echo "+eth0" > /sys/class/net/bond0/bonding/slaves
    echo "+eth1" > /sys/class/net/bond0/bonding/slaves

    Remove a slave interface from bond0

    echo "-eth0" > /sys/class/net/bond0/bonding/slaves

    Delete a bond interface

    echo "-bond0" >  /sys/class/net/bonding_masters

    Additional documentation

    If you’re looking for more info here is the complete sysfs interface documentation from the linux kernel bonding driver

    3.4 Configuring Bonding Manually via Sysfs
    	Starting with version 3.0.0, Channel Bonding may be configured
    via the sysfs interface.  This interface allows dynamic configuration
    of all bonds in the system without unloading the module.  It also
    allows for adding and removing bonds at runtime.  Ifenslave is no
    longer required, though it is still supported.
    	Use of the sysfs interface allows you to use multiple bonds
    with different configurations without having to reload the module.
    It also allows you to use multiple, differently configured bonds when
    bonding is compiled into the kernel.
    	You must have the sysfs filesystem mounted to configure
    bonding this way.  The examples in this document assume that you
    are using the standard mount point for sysfs, e.g. /sys.  If your
    sysfs filesystem is mounted elsewhere, you will need to adjust the
    example paths accordingly.
    Creating and Destroying Bonds
    To add a new bond foo:
    # echo +foo > /sys/class/net/bonding_masters
    To remove an existing bond bar:
    # echo -bar > /sys/class/net/bonding_masters
    To show all existing bonds:
    # cat /sys/class/net/bonding_masters
    NOTE: due to 4K size limitation of sysfs files, this list may be
    truncated if you have more than a few hundred bonds.  This is unlikely
    to occur under normal operating conditions.
    Adding and Removing Slaves
    	Interfaces may be enslaved to a bond using the file
    /sys/class/net/<bond>/bonding/slaves.  The semantics for this file
    are the same as for the bonding_masters file.
    To enslave interface eth0 to bond bond0:
    # ifconfig bond0 up
    # echo +eth0 > /sys/class/net/bond0/bonding/slaves
    To free slave eth0 from bond bond0:
    # echo -eth0 > /sys/class/net/bond0/bonding/slaves
    	When an interface is enslaved to a bond, symlinks between the
    two are created in the sysfs filesystem.  In this case, you would get
    /sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and
    /sys/class/net/eth0/master pointing to /sys/class/net/bond0.
    	This means that you can tell quickly whether or not an
    interface is enslaved by looking for the master symlink.  Thus:
    # echo -eth0 > /sys/class/net/eth0/master/bonding/slaves
    will free eth0 from whatever bond it is enslaved to, regardless of
    the name of the bond interface.
    Changing a Bond's Configuration
    	Each bond may be configured individually by manipulating the
    files located in /sys/class/net/<bond name>/bonding
    	The names of these files correspond directly with the command-
    line parameters described elsewhere in this file, and, with the
    exception of arp_ip_target, they accept the same values.  To see the
    current setting, simply cat the appropriate file.
    	A few examples will be given here; for specific usage
    guidelines for each parameter, see the appropriate section in this
    To configure bond0 for balance-alb mode:
    # ifconfig bond0 down
    # echo 6 > /sys/class/net/bond0/bonding/mode
     - or -
    # echo balance-alb > /sys/class/net/bond0/bonding/mode
    	NOTE: The bond interface must be down before the mode can be
    To enable MII monitoring on bond0 with a 1 second interval:
    # echo 1000 > /sys/class/net/bond0/bonding/miimon
    	NOTE: If ARP monitoring is enabled, it will disabled when MII
    monitoring is enabled, and vice-versa.
    To add ARP targets:
    # echo + > /sys/class/net/bond0/bonding/arp_ip_target
    # echo + > /sys/class/net/bond0/bonding/arp_ip_target
    	NOTE:  up to 16 target addresses may be specified.
    To remove an ARP target:
    # echo - > /sys/class/net/bond0/bonding/arp_ip_target
    To configure the interval between learning packet transmits:
    # echo 12 > /sys/class/net/bond0/bonding/lp_interval
    	NOTE: the lp_inteval is the number of seconds between instances where
    the bonding driver sends learning packets to each slaves peer switch.  The
    default interval is 1 second.
    Example Configuration
    	We begin with the same example that is shown in section 3.3,
    executed with sysfs, and without using ifenslave.
    	To make a simple bond of two e100 devices (presumed to be eth0
    and eth1), and have it persist across reboots, edit the appropriate
    file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the
    modprobe bonding
    modprobe e100
    echo balance-alb > /sys/class/net/bond0/bonding/mode
    ifconfig bond0 netmask up
    echo 100 > /sys/class/net/bond0/bonding/miimon
    echo +eth0 > /sys/class/net/bond0/bonding/slaves
    echo +eth1 > /sys/class/net/bond0/bonding/slaves
    	To add a second bond, with two e1000 interfaces in
    active-backup mode, using ARP monitoring, add the following lines to
    your init script:
    modprobe e1000
    echo +bond1 > /sys/class/net/bonding_masters
    echo active-backup > /sys/class/net/bond1/bonding/mode
    ifconfig bond1 netmask up
    echo + /sys/class/net/bond1/bonding/arp_ip_target
    echo 2000 > /sys/class/net/bond1/bonding/arp_interval
    echo +eth2 > /sys/class/net/bond1/bonding/slaves
    echo +eth3 > /sys/class/net/bond1/bonding/slaves

    How to fix yum [Errno -1] Metadata file does not match checksum

    Friday, June 20th, 2014

    I recently upgraded my local yum repository server to RHEL6. After upgrading I noticed the below error when my el5 clients would attempting to install packages from the local repo with yum.

    [Errno -1] Metadata file does not match checksum Trying other mirror

    As it turns out the default checksum type was changed when upgrading to the newer version of createrepo available on my el6 system. The default was changed from sha to sha256 which causes an incompatability with the older rhel5 clients. Here’s an excerpt from the createrepo man page:

    -s --checksum
    Choose the checksum type used in repomd.xml and for packages in the metadata. The default is now "sha256" (if python has hashlib). The older default was "sha", which is
    actually "sha1", however explicitly using "sha1" doesn’t work on older (3.0.x) versions of yum, you need to specify "sha".

    The fix lucikly was simple, I re-ran createrepo with the ‘–checksum sha’ option.

    #Createrepo command compatible with older el5 systems

    createrepo --checksum sha /path/to/repo

    Then I ran a yum clean all on the client, just to be sure.

    #clear cache on client system

    yum clean all

    And that’s it! My el6 yum repository is now happily serving both el5 and el6 clients.

    PuppetDB API Cheat Sheet and Useful One Liners

    Thursday, June 5th, 2014

    This is my puppetdb api cheat sheet of useful one liners for querying against the puppetdb api. These are geared for a command line environment, so you’ll notice that they depend on the curl jq commands. These available through the package manager on most linux distributions.

    This It’s a work in progress.

    Generate a host list from the puppetdb api.

    $ curl -s 'http://localhost:8080/v3/nodes' | jq -r '.[].name'

    PuppetDB API documentation:”

    jq documentation: