Remote Tail

This article is just a quick code snipped for bash shell to allow us to tail log files from a remote endpoint.

Nowadays, we are building plenty of microservices and the common pattern is to aggregate them using tools like Kibana to store and search them with the help of some correlation ids. Despite, this is a very good solution, sometimes we do not need anything that fancy.

In Spring Actuator, we can find the endpoint “/logfile” that it has proven a lot of times to be pretty useful. This endpoint allows us to recover the log file from the server. We can download it or just check it on the browser. The problem is when this logfile reach a size that our browser can not manage properly. We can use tools like “wget” to download the log file and analyse it locally but it seems absurd to download the whole file every time we want an update.

I have a different proposal. With a few lines of shell scripting, we can write a snipped to tail the log file into a local file, and we can use “less” to monitor it or perform searches on the file.

#!/bin/bash

#
# Check if the given server support HTTP range header
# param 1: url
#
function check_ranges_support() {
  ret=`curl -s -I -X HEAD $1 | grep "Accept-Ranges: bytes"`

  if [ -z "$ret" ]; then
    echo "Ranges are nor supported by the server"
    exit 1
  fi
}

#
# Recovers the total length of the given file
# param 1: url
#
function get_length() {
  ret=`curl -s -I -X HEAD $1 | awk '/Content-Length:/ {print $2}'`
  echo $ret | sed 's/[^0-9]*//g'
}

#
# Print the requested part of the remote file
# param 1: url
# param 2: off
# param 3: len
# param 4: output file
#
function print_to_logfile() {
  curl --header "Range: bytes=$2-$3" -s $1 >> $4
}

#
# Clean the previous log file
#
function clean_logfile() {
  rm -f $1
}

# call validation
if [ $# -lt 1 ]; then
  echo "Syntax: remote-tail.sh <URL> [<logfile name>]"
  exit 1
fi

url=$1
offset=0
logfile=tmplog

if [ $# -eq 2 ]; then
  logfile=$2
fi

check_ranges_support $url
clean_logfile $logfile

len=`get_length $url`
off=$((len - offset))

until [ "$off" -gt "$len" ]; do
  len=`get_length $url`

  if [ "$off" -eq "$len" ]; then
    sleep 5 # we refresh every 5 seconds if no changes
  else
    sleep 1 # we refresh every second to not hammer too much the server
    print_to_logfile $url $off $len $logfile
  fi

  off=$len
done

We can use it with:

./remote-tail.sh https://server/logfile

I hope it helps.

Remote Tail

Random enum

As a software engineers or software developers we need to test what we are implementing (you have a description of types of tests here). When implementing these tests, we can hardcode the data we are using or, we can randomly generate it what, despite we can think it is going to make our life more difficult when debugging errors, it is going to make it easier in the long run. Our code should not be linked to the data it is processing, it should be generic for the type of data it is expecting.

To do this, the easiest way is to implement or use libraries that implement random generators. One of the most interesting and one it is always forgotten is the random generator for our enums.

In Java, there is an easy way to implement it.

It can be just for one enum class:

private static CarBrand randomCarBrand() {
    return CarBrand.class.getEnumConstants()[new Random().nextInt(CarBrand.class.getEnumConstants().length)];
}

Or, even more interesting, it can be a generic random generator that it receives as a parameter an enum class and return the random value:

public static <T extends Enum<?>> T randomEnum(Class<T> clazz) {
    int x = random.nextInt(clazz.getEnumConstants().length);
    return clazz.getEnumConstants()[x];
}

We can see an example here:

import java.util.Random;

public class RandomEnum {

    public static void main(String[] args) {

        for (int i = 0; i < 10; i++) {
            System.out.println(randomCarBrand());
        }
    }

    private static CarBrand randomCarBrand() {
        return CarBrand.class.getEnumConstants()[new   Random().nextInt(CarBrand.class.getEnumConstants().length)];
    }

    enum CarBrand {
        ARUTI_SUZUKI,
        TATA,
        HONDA,
        HYUNDAI,
        FORD,
        MAHINDRA,
        SKODA,
        ARIEL,
        ASHOK_LEYLAND,
        ASTON_MARTIN,
        AUDI,
        BAJAJ,
        BENTLEY,
        BMW,
    }
}

Do not forget, from know on, if you are not doing it, try to make all your tests, except if you want to test an edge case, random and see how it goes.

Random enum

Java reflection: Accessing a private field

Sometimes, when we import 3rd-party libraries, we can find cases where the information we want is has been populated in an object but, the property is a private one and there are no methods to recover it. Here, it is where the Java reflection capabilities can help us.

Note: Use reflection carefully and, usually, as a last resource.

Let’s say we have a simple class with a private property:

class A {
    private String value = "value";
}

We want to access the property “value” but, for obvious reasons, we cannot do it.

We can implement our reflection method to access the property value:

import java.lang.reflect.Field;

public class Reflection {

   public static void main(String[] args) throws NoSuchFieldException, IllegalAccessException {
      final A a = new A();
      final Field secureRenegotiationField = a.getClass().getDeclaredField("value");

      secureRenegotiationField.setAccessible(true);

      System.out.println((String) secureRenegotiationField.get(a));
   }
}

With these few lines of code, we will be able to access the value.

Java reflection: Accessing a private field

Groups in Regular Expressions

We cannot discuss the power of regular expression, an amazing tool with unlimited (usually our imagination) capabilities to progress strings. Every developer should, at least, have a basic understanding of them. But, lately, I have realized not a lot of people knows the possibility of creating and labeling “groups”. Groups allow us to access in a very  simple and clear way to the expressions matching our regular expression.

Regular expressions allow us to not just match text but also to extract information for further processing. This is done by defining groups of characters and capturing them using the special parentheses “(” and “)” metacharacters. Any subpattern inside a pair of parentheses will be captured as a group. In practice, this can be used to extract information like phone numbers or emails from all sorts of data.

Here, I am just going to write a little example to show the basic behavior and, I leave to all of you to find the appropriate use cases. In the example, I am going to extract some different hashes for further processing.

public static void main(String[] args) {
    final Pattern HASH_PATTERN = Pattern.compile("^(?<md5>[0-9a-f]{32})(?:/)(?<sha1>[0-9a-f]{40})(?:/)(?<sha256>[0-9a-f]{64})$");
    final Matcher matcher = HASH_PATTERN.matcher("ce114e4501d2f4e2dcea3e17b546f339/a54d88e06612d820bc3be72877c74f257b561b19/c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e");

    if (matcher.matches()) {         
        final String md5 = matcher.group("md5");         
        final String sha1 = matcher.group("sha1");         
        final String sha256 = matcher.group("sha256");         
        ...
    }     
    ... 
}

As you can see, the example is pretty simple, it takes one line that contains a string and extracts the MD5, SHA1 and SHA256 hashes. We can see the code is easy to read and understand because everything is using human readable labels not just numbers to access the groups and there is no need to process the string with split operations or similars.

The syntax for the groups is:

(?<name>X)– X, as a named-capturing group

(?:X) – X, as a non-capturing group

With this, we can easily make our code easier to read and maintain when we are extracting information or doing some text processing.

For further information, check the Java documentation: Pattern (Java SE 10 & JDK 10)

Groups in Regular Expressions

Exploring the logs

As a developers an important part of our job sometimes is to fix problems in the different environments where our applications are deployed. Usually, this means to deal with huge log files to find where errors occur, and their stacktraces to add some context to the problem. The problem is that usually log files are verbose and contain a lot of information.

A couple of useful command to deal with this can be:

  • grep
  • zgrep

Both have the same purpose the only difference it that “grep” works with normal files and “zgrep” works with compressed (.gz) files. Usualy files are compressed due to the logs rotation scheduled in the servers. Both commands have multiple options and flags but, I am going to expose here two flags that have been useful multiple times:

  • -E expr: Allow as to supply a pattern for the search.
  • -C num: Print num lines of leading and trailing output context.
  • –color: Shows the matched information in color in the terminal.

As an example we have:

zgrep --color -E '(Sending email)' myLog.log-20170621.gz
grep --color -E '(Sending email)' myLog.log
grep --color -C 25 -E '(Sending email)' myLog.log

As we can see, obviously, they can be combined.

Exploring the logs

Simplifiying SSH

Nowadays we are use to deploy code in the cloud and to have all our machines and servers in cloud environments. All of this, it has even made more important the use of ssh to connect remotely to our servers allocated in the cloud.

I have written multiple times in my console the commands to connect to one server or another but, as every developer, I am lazy and I try to simplify my life. In this case, we can do this with a simple lines in a couple of files:

  • ~/.ssh/config: We are going to configure the machines we want to connect or tunnels we wnat to create.
  • ~/.bashrc or ~/.bash_profile: Create some alias to easily connect to our servers

SSH config file

Server to connect

# MyServer-1
Host                    myServer1
HostName                myserver1.myorg.com
User                    username
IdentityFile            ~/.ssh/myCertificate.pem PasswordAuthentication  no
StrictHostKeyChecking   no

Create tunnel

# MyServer-1 - myDb
Host                    myServer1Db
HostName                myserver1.myorg.com
User                    username
IdentityFile            ~/.ssh/myCertificate.pem PasswordAuthentication  no
StrictHostKeyChecking   no
LocalForward            3307 myserver1.myorg.com:3306

Bash Config file

alias myserver1="ssh myServer1" alias myserver1db="ssh myServer1Db"

Conclusion

After this, it will be enough to connect to our remote servers with executing our aliases in our console. No more remember commands.

Simplifiying SSH