remote

Remote Tail

fjavierm, , Leave a comment

This article is just a quick code snipped for bash shell to allow us to tail log files from a remote endpoint.

Nowadays, we are building plenty of microservices and the common pattern is to aggregate them using tools like Kibana to store and search them with the help of some correlation ids. Despite, this is a very good solution, sometimes we do not need anything that fancy.

In Spring Actuator, we can find the endpoint “/logfile” that it has proven a lot of times to be pretty useful. This endpoint allows us to recover the log file from the server. We can download it or just check it on the browser. The problem is when this logfile reach a size that our browser can not manage properly. We can use tools like “wget” to download the log file and analyse it locally but it seems absurd to download the whole file every time we want an update.

I have a different proposal. With a few lines of shell scripting, we can write a snipped to tail the log file into a local file, and we can use “less” to monitor it or perform searches on the file.

#!/bin/bash

#
# Check if the given server support HTTP range header
# param 1: url
#
function check_ranges_support() {
  ret=`curl -s -I -X HEAD $1 | grep "Accept-Ranges: bytes"`

  if [ -z "$ret" ]; then
    echo "Ranges are nor supported by the server"
    exit 1
  fi
}

#
# Recovers the total length of the given file
# param 1: url
#
function get_length() {
  ret=`curl -s -I -X HEAD $1 | awk '/Content-Length:/ {print $2}'`
  echo $ret | sed 's/[^0-9]*//g'
}

#
# Print the requested part of the remote file
# param 1: url
# param 2: off
# param 3: len
# param 4: output file
#
function print_to_logfile() {
  curl --header "Range: bytes=$2-$3" -s $1 >> $4
}

#
# Clean the previous log file
#
function clean_logfile() {
  rm -f $1
}

# call validation
if [ $# -lt 1 ]; then
  echo "Syntax: remote-tail.sh <URL> [<logfile name>]"
  exit 1
fi

url=$1
offset=0
logfile=tmplog

if [ $# -eq 2 ]; then
  logfile=$2
fi

check_ranges_support $url
clean_logfile $logfile

len=`get_length $url`
off=$((len - offset))

until [ "$off" -gt "$len" ]; do
  len=`get_length $url`

  if [ "$off" -eq "$len" ]; then
    sleep 5 # we refresh every 5 seconds if no changes
  else
    sleep 1 # we refresh every second to not hammer too much the server
    print_to_logfile $url $off $len $logfile
  fi

  off=$len
done

We can use it with:

./remote-tail.sh https://server/logfile

I hope it helps.

Remote Tail

Simplifiying SSH

fjavierm, , , Leave a comment

Nowadays we are use to deploy code in the cloud and to have all our machines and servers in cloud environments. All of this, it has even made more important the use of ssh to connect remotely to our servers allocated in the cloud.

I have written multiple times in my console the commands to connect to one server or another but, as every developer, I am lazy and I try to simplify my life. In this case, we can do this with a simple lines in a couple of files:

  • ~/.ssh/config: We are going to configure the machines we want to connect or tunnels we wnat to create.
  • ~/.bashrc or ~/.bash_profile: Create some alias to easily connect to our servers

SSH config file

Server to connect

# MyServer-1
Host                    myServer1
HostName                myserver1.myorg.com
User                    username
IdentityFile            ~/.ssh/myCertificate.pem
PasswordAuthentication  no
StrictHostKeyChecking   no

Create tunnel

# MyServer-1 - myDb
Host                    myServer1Db
HostName                myserver1.myorg.com
User                    username
IdentityFile            ~/.ssh/myCertificate.pem
PasswordAuthentication  no
StrictHostKeyChecking   no
LocalForward            3307 myserver1.myorg.com:3306

Bash Config file

alias myserver1="ssh myServer1" alias myserver1db="ssh myServer1Db"

Conclusion

After this, it will be enough to connect to our remote servers with executing our aliases in our console. No more remember commands.

Simplifiying SSH

Publishing to GitHub

fjavierm, , , Leave a comment

GitHub is a web-based Git or version control repository and Internet hosting service. As a developers, I am quite sure that all of you know the platform.

Every-time we start a new project, even if it is just something for us, it is a good idea to use a version control system. Here is where Git and GitHub can help us.

There are two easy ways to create and upload our project to a repository.

The first way, it is to create the repository in GitHub, and after that clone in our machine the created repository and start writing our code. This is the simplest way. Doing it in this way, our local repository is already connected to the remote repository and we just need to start committing things. The clone command, assuming the account name is “fjavierm” and the repository name is “myproject”,  is:

git clone https://github.com/fjavierm/myproject.git

Te second way, it is when we already have a project in our local machine and we want to add the project to a repository. In this case, we need to perform a few more steps.

  1. We need to create the repository under our GitHub account. In this case, my account is “fjavierm” and my repository is going to be “myproject”.
  2. In our local machine, in our local project folder, we need to initiate the git repository. This will add a folder “.git”: init
  3. We can check the status of the available files: status. This instruction will show us the files that we can add to the repository to commit. In this step, it deserves to pay special attention to the files that we do not want to add and, maybe, it is a good idea to create the files “.gitignore” and “README.md”
  4. Add the files to the repository.
  5. Commit the added files adding a descriptive message. Usually, issue number and description, or something meaningful.
  6. Connect our local repository with our remote repository.
  7. Check we have perform the previous step properly.
  8. Push our changes to the remote repository.
git init
git status
git add .
git commit -m "Starting the project."
git remote add origin https://github.com/fjavierm/myproject.git
git remote -v
git push -u origin master

With these few instructions we should have available all our code in the GitHub repository.

Publishing to GitHub