code-snippet

PostgreSQL: Advisory Locks

fjavierm, , Leave a comment

Today, we are going to talk about PostgreSQL Advisory Locks. This kind of locks are created by the application and developers and, they have meaning inside the application, PostgreSQL does not enforce their use and they are there to fulfil a business or coding specific case. I was going to try to explain and to add some literature around them but, after reading PostgreSQL documentation (can be found here) I do not think it is necessary because the definition it is easy to understand and, besides, on the same page we can find the other types of locks available giving us some extra context. Instead, we are going to see some real-world code as an example.

Let’s say we have our shiny service that runs multiple instances at the same time on our production environment and, on that services, we run a scheduled task that updates one of our database tables adding a different sequence number to the existing rows (buildings) for all the existing cities. Something like:

id (uuid)city (text)building. (text)registered (timestamp)occurrence (bigint)
e6448a82LondonBritish Museum2021/02/01 13:00:00.000null
97347903LondonTower of London2021/02/01 12:59:59.999null
7befe492ParisEiffel Tower2021/01/31 07:23:34.294null
b426681aParisLouvre Museum2021/02/01 12:59:59.999null
156e1f89LondonBig Ben2021/02/01 12:59:59.999null
Table ‘buildings’

For some curious minds about the reason why we need this ‘occurrence‘ sequence, one of the cases can be to create an endpoint to allow other systems to synchronise these buildings. We could sort using the ‘registered‘ field but, it can happen that two buildings in the same city can be registered at the same time making it impossible to warrantee the information is going to be returned always on the same order, and this can cause synchronisation problems or even missing a building due to paginated requests. We want to be able to sort them in an immutable way.

Going back to the multiple services running the tasks, we can have some ugly situations were one of the tasks running is working already and, in the middle of updating a city, when another task in a different service start processing the same city, especially if we do this on batches due to the huge amount of data we store.

One simple solution of this is to use Advisory Locks allowing us, developers, to lock a city when the task is updating it. For this purpose, PostgreSQL offers us two nice functions to work with:

  • pg_advisory_lock: Obtains an exclusive session-level advisory lock, waiting if necessary.
  • pg_try_advisory_lock: Obtains an exclusive session-level advisory lock if available. This will either obtain the lock immediately and return ‘true‘, or return ‘false‘ without waiting if the lock cannot be acquired immediately.

The full list of system administration functions related with advisory locks can be found here.

For the purposes of the example code, we are going to implement, we will be using the second one because it makes sense if one city it is been processed, we do not want to process it again till the next scheduled time.

public void assignOccurrenceSequences() {
    final List<String> cities = buildingDao.retrievePendingCities();

    for (final String city : cities) {
        final int lockId = Math.abs(Hashing.sha256().newHasher()
            .putString(city, StandardCharsets.UTF_8)
            .hash().asInt());

        logger.info("Taking advisory_lock {} for city {} ", lockId, city);
        try (Connection connection = dataSource.getConnection()) {
            connection.setAutoCommit(true);

            final boolean lockObtained;
            try (Statement statement = connection.createStatement()) {
                lockObtained = statement.execute(format("select pg_try_advisory_lock(%d)", lockId));
            }

            if (lockObtained) {
                try {
                    final int updates = buildingDao.populateOccurrenceSequences(city);
                    logger.info("Assigning {} sequences for city {}", updates, city);
                } finally {
                    try (Statement statement = connection.createStatement()) {
                        statement.execute(format("select pg_advisory_unlock(%d)", lockId));
                    }

                    logger.info("Released advisory_lock {} for city {}", lockId, city);
                }
            } else {
                logger.info("advisory_lock {} for city {} already taken", lockId, city);
            }
        } catch (SQLException e) {
            throw new IllegalStateException(e);
        }
    }
}

On lines 5, 6 and 7 we create a unique lock id we will be using to establish the lock and make sure all the tasks running calculate the same id. And yes, before someone points it, we are assuming that ‘city‘ is unique. With this generated lock id, we can try to acquire the lock. In case of success, we proceed with the update. In case of fail, we skip that city and proceed with the rest of the cities.

PostgreSQL: Advisory Locks

Add a header to Spring RestTemplate

fjavierm, Leave a comment

Today, just a short code snippet. How to add a header to the ‘RestTemplate’ on Spring.

public class HeaderRequestInterceptor implements ClientHttpRequestInterceptor {

    private final String headerName;
    private final String headerValue;

    public HeaderRequestInterceptor(String headerName, String headerValue) {
        this.headerName = headerName;
        this.headerValue = headerValue;
    }

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException {
        request.getHeaders().set(headerName, headerValue);
        return execution.execute(request, body);
    }
}

Now, we add it to our ‘RestTemplate’:

List<ClientHttpRequestInterceptor> interceptors = new ArrayList<ClientHttpRequestInterceptor>();
interceptors.add(new HeaderRequestInterceptor("X-Custom-Header", "<custom_value>"));

RestTemplate restTemplate = new RestTemplate();
restTemplate.setInterceptors(interceptors);

And, that’s all.

Just an extra side note. As of Spring Framework 5, a new HTTP client called ‘WebClient’ has been added. It is assumed that ‘RestTemplate’ will be deprecated at some point. If we are starting a new application, specially if you are using the ‘WebFlux’ stack, it will be a better choice to use the new version.

Add a header to Spring RestTemplate

Spring Application Events

fjavierm, Leave a comment

Today, we are going to implement a simple example using spring application events.

Spring application events allow us to throw and listen to specific application events that we can process as we wish. Events are meant for exchanging information between loosely coupled components. As there is no direct coupling between publishers and subscribers, it enables us to modify subscribers without affecting the publishers and vice-versa.

To build our PoC and to execute it, we are going to need just a few classes. We will start with a basic Spring Boot project with the ‘web’ starter. And, once we have that in place (you can use the Spring Initializr) we can start adding our classes.

Let’s start with a very basic ‘User’ model

public class User {

    private String firstname;
    private String lastname;

    public String getFirstname() {
        return firstname;
    }

    public User setFirstname(String firstname) {
        this.firstname = firstname;
        return this;
    }

    public String getLastname() {
        return lastname;
    }

    public User setLastname(String lastname) {
        this.lastname = lastname;
        return this;
    }

    @Override
    public String toString() {
        return "User{" +
                "firstname='" + firstname + '\'' +
                ", lastname='" + lastname + '\'' +
                '}';
    }
}

Nothing out of the ordinary here. Just a couple of properties and some getter and setter methods.

Now, let’s build a basic service that is going to simulate a ‘register’ operation:

...
import org.springframework.context.ApplicationEventPublisher;
...

@Service
public class UserService {

    private static final Logger logger = LoggerFactory.getLogger(UserService.class);

    private final ApplicationEventPublisher publisher;

    public UserService(ApplicationEventPublisher publisher) {
        this.publisher = publisher;
    }

    public void register(final User user) {
        logger.info("Registering {}", user);

        publisher.publishEvent(new UserRegistered(user));
    }
}

Here we have the first references to the event classes the Spring Framework offers us. The ‘ApplicationEventPublished’ that it will allow us to publish the desired event to be consumer by listeners.

The second reference we are going to have to the events framework is when we create and event class we are going to send. In this case, the class ‘UserRegistered’ we can see on the publishing line above.

...
import org.springframework.context.ApplicationEvent;

public class UserRegistered extends ApplicationEvent {

    public UserRegistered(User user) {
        super(user);
    }
}

As we can see, extending the class ‘ApplicationEvent’ we have very easily something we can publish and listen to it.

Now. let’s implements some listeners. The first of them is going to be one implementing the class ‘ApplicationListener’ and, the second one, it is going to be annotation based. Two simple options offered by Spring to build our listeners.

...
import org.springframework.context.ApplicationListener;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

public class UserListeners {

    // Technical note: By default listener events return 'void'. If an object is returned, it will be published as an event

    /**
     * Example of event listener using the implementation of {@link ApplicationListener}
     */
    static class RegisteredListener implements ApplicationListener<UserRegistered> {

        private static final Logger logger = LoggerFactory.getLogger(RegisteredListener.class);

        @Override
        public void onApplicationEvent(UserRegistered event) {
            logger.info("Registration event received for {}", event);
        }
    }

    /**
     * Example of annotation based event listener
     */
    @Component
    static class RegisteredAnnotatedListener {

        private static final Logger logger = LoggerFactory.getLogger(RegisteredAnnotatedListener.class);

        @EventListener
        void on(final UserRegistered event) {
            logger.info("Annotated registration event received for {}", event);
        }
    }
}

As we can see, very basic stuff. It is worth it to mention the ‘Technical note’. By default, the listener methods return ‘void’, they are initially design to received an event, do some stuff and finish. But, obviously, they can at the same time publish some messages, we can achieve this easily, returning an object. The returned object will be published as any other event.

Once we have all of this, let’s build a simple controller to run the process:

@RestController
@RequestMapping("/api/users")
public class UserController {

    private final UserService userService;

    public UserController(UserService userService) {
        this.userService = userService;
    }

    @GetMapping
    @ResponseStatus(HttpStatus.CREATED)
    public void register(@RequestParam("firstname") final String firstname,
                         @RequestParam("lastname") final String lastname) {
        Objects.requireNonNull(firstname);
        Objects.requireNonNull(lastname);

        userService.register(new User().setFirstname(firstname).setLastname(lastname));
    }
}

Nothing out of the ordinary, simple stuff.

We can invoke the controller with any tools we want but, a simple way, it is using cURL.

curl -X GET "http://localhost:8080/api/users?firstname=john&lastname=doe"

Once we call the endpoint, we can see the log messages generated by the publisher and the listeners:

Registering User{firstname='john', lastname='doe'}
Annotated registration event received for dev.binarycoders.spring.event.UserRegistered[source=User{firstname='john', lastname='doe'}]
Registration event received for dev.binarycoders.spring.event.UserRegistered[source=User{firstname='john', lastname='doe'}]

As we can see, the ‘register’ action is executed and it publishes the event and, both listeners, the annotated and the implemented, receive and process the message.

As usual you can find the source for this example here, in the ‘spring-events’ module.

For some extra information, you can take a look at one of the videos of the last SpringOne.

Spring Application Events

Builder pattern + inheritance

fjavierm, Leave a comment

In general, it is very simple to implement in Java the builder pattern, a few lines of code and the problem is solved but, when we are using inheritance, it is not as intuitive apparently as it should be. I have lately seen poor attempts of doing it and not achieving the desired result.

In this article, we are going to build a very simple example of that.

public class Parent {

    private final String a;

    protected Parent(final Builder<?> builder) {
        this.a = builder.a;
    }

    public String getA() { return a; }

    public static class Builder<T extends Builder<T>> {

        private String a;

        public T a(final String a) {
            this.a = a;
            return (T) this;
        }

        public Parent build() {
            return new Parent(this);
        }
    }
}

In this first class, the parent class, we can see we are using generics to allow child classes to pass their builders.

public class Children extends Parent {

    private final String b;

    protected Children(final Builder builder) {
        super(builder);
        this.b = builder.b;
    }

    public String getB() {
        return b;
    }

    public static Builder builder() {
        return new Builder();
    }

    public static class Builder extends Parent.Builder<Builder> {
        private String b;

        public Builder b(final String b) {
            this.b = b;
            return this;
        }

        public Children build() {
            return new Children(this);
        }
    }
}

Here, we can see how we pass the child builder in the diamond operator, this will allow us to add values for the properties to the patent and to the child using the builder.

public class Main {

    public static void main(String[] args) {
        final Children children = Children.builder()
            .a("Hi")
            .b("Bye")
            .build();

        System.out.println(children.toString());
    }
}

Here, we can see how to use the builder. Thanks to the generics, the call to .a(“Hi”) returns a child builder and not a parent builder what it would make impossible to call .b(“Bye”).

I hope it is useful.

Builder pattern + inheritance

Remote Tail

fjavierm, , Leave a comment

This article is just a quick code snipped for bash shell to allow us to tail log files from a remote endpoint.

Nowadays, we are building plenty of microservices and the common pattern is to aggregate them using tools like Kibana to store and search them with the help of some correlation ids. Despite, this is a very good solution, sometimes we do not need anything that fancy.

In Spring Actuator, we can find the endpoint “/logfile” that it has proven a lot of times to be pretty useful. This endpoint allows us to recover the log file from the server. We can download it or just check it on the browser. The problem is when this logfile reach a size that our browser can not manage properly. We can use tools like “wget” to download the log file and analyse it locally but it seems absurd to download the whole file every time we want an update.

I have a different proposal. With a few lines of shell scripting, we can write a snipped to tail the log file into a local file, and we can use “less” to monitor it or perform searches on the file.

#!/bin/bash

#
# Check if the given server support HTTP range header
# param 1: url
#
function check_ranges_support() {
  ret=`curl -s -I -X HEAD $1 | grep "Accept-Ranges: bytes"`

  if [ -z "$ret" ]; then
    echo "Ranges are nor supported by the server"
    exit 1
  fi
}

#
# Recovers the total length of the given file
# param 1: url
#
function get_length() {
  ret=`curl -s -I -X HEAD $1 | awk '/Content-Length:/ {print $2}'`
  echo $ret | sed 's/[^0-9]*//g'
}

#
# Print the requested part of the remote file
# param 1: url
# param 2: off
# param 3: len
# param 4: output file
#
function print_to_logfile() {
  curl --header "Range: bytes=$2-$3" -s $1 >> $4
}

#
# Clean the previous log file
#
function clean_logfile() {
  rm -f $1
}

# call validation
if [ $# -lt 1 ]; then
  echo "Syntax: remote-tail.sh <URL> [<logfile name>]"
  exit 1
fi

url=$1
offset=0
logfile=tmplog

if [ $# -eq 2 ]; then
  logfile=$2
fi

check_ranges_support $url
clean_logfile $logfile

len=`get_length $url`
off=$((len - offset))

until [ "$off" -gt "$len" ]; do
  len=`get_length $url`

  if [ "$off" -eq "$len" ]; then
    sleep 5 # we refresh every 5 seconds if no changes
  else
    sleep 1 # we refresh every second to not hammer too much the server
    print_to_logfile $url $off $len $logfile
  fi

  off=$len
done

We can use it with:

./remote-tail.sh https://server/logfile

I hope it helps.

Remote Tail

Random enum

fjavierm, , Leave a comment

As a software engineers or software developers we need to test what we are implementing (you have a description of types of tests here). When implementing these tests, we can hardcode the data we are using or, we can randomly generate it what, despite we can think it is going to make our life more difficult when debugging errors, it is going to make it easier in the long run. Our code should not be linked to the data it is processing, it should be generic for the type of data it is expecting.

To do this, the easiest way is to implement or use libraries that implement random generators. One of the most interesting and one it is always forgotten is the random generator for our enums.

In Java, there is an easy way to implement it.

It can be just for one enum class:

private static CarBrand randomCarBrand() {
    return CarBrand.class.getEnumConstants()[new Random().nextInt(CarBrand.class.getEnumConstants().length)];
}

Or, even more interesting, it can be a generic random generator that it receives as a parameter an enum class and return the random value:

public static <T extends Enum<?>> T randomEnum(Class<T> clazz) {
    int x = random.nextInt(clazz.getEnumConstants().length);
    return clazz.getEnumConstants()[x];
}

We can see an example here:

import java.util.Random;

public class RandomEnum {

    public static void main(String[] args) {

        for (int i = 0; i < 10; i++) {
            System.out.println(randomCarBrand());
        }
    }

    private static CarBrand randomCarBrand() {
        return CarBrand.class.getEnumConstants()[new   Random().nextInt(CarBrand.class.getEnumConstants().length)];
    }

    enum CarBrand {
        ARUTI_SUZUKI,
        TATA,
        HONDA,
        HYUNDAI,
        FORD,
        MAHINDRA,
        SKODA,
        ARIEL,
        ASHOK_LEYLAND,
        ASTON_MARTIN,
        AUDI,
        BAJAJ,
        BENTLEY,
        BMW,
    }
}

Do not forget, from know on, if you are not doing it, try to make all your tests, except if you want to test an edge case, random and see how it goes.

Random enum

Java reflection: Accessing a private field

fjavierm, Leave a comment

Sometimes, when we import 3rd-party libraries, we can find cases where the information we want is has been populated in an object but, the property is a private one and there are no methods to recover it. Here, it is where the Java reflection capabilities can help us.

Note: Use reflection carefully and, usually, as a last resource.

Let’s say we have a simple class with a private property:

class A {
    private String value = "value";
}

We want to access the property “value” but, for obvious reasons, we cannot do it.

We can implement our reflection method to access the property value:

import java.lang.reflect.Field;

public class Reflection {

   public static void main(String[] args) throws NoSuchFieldException, IllegalAccessException {
      final A a = new A();
      final Field secureRenegotiationField = a.getClass().getDeclaredField("value");

      secureRenegotiationField.setAccessible(true);

      System.out.println((String) secureRenegotiationField.get(a));
   }
}

With these few lines of code, we will be able to access the value.

Java reflection: Accessing a private field

Groups in Regular Expressions

fjavierm, , Leave a comment

We cannot discuss the power of regular expression, an amazing tool with unlimited (usually our imagination) capabilities to progress strings. Every developer should, at least, have a basic understanding of them. But, lately, I have realized not a lot of people knows the possibility of creating and labeling “groups”. Groups allow us to access in a very  simple and clear way to the expressions matching our regular expression.

Regular expressions allow us to not just match text but also to extract information for further processing. This is done by defining groups of characters and capturing them using the special parentheses “(” and “)” metacharacters. Any subpattern inside a pair of parentheses will be captured as a group. In practice, this can be used to extract information like phone numbers or emails from all sorts of data.

Here, I am just going to write a little example to show the basic behavior and, I leave to all of you to find the appropriate use cases. In the example, I am going to extract some different hashes for further processing.

public static void main(String[] args) {
    final Pattern HASH_PATTERN = Pattern.compile("^(?<md5>[0-9a-f]{32})(?:/)(?<sha1>[0-9a-f]{40})(?:/)(?<sha256>[0-9a-f]{64})$");
    final Matcher matcher = HASH_PATTERN.matcher("ce114e4501d2f4e2dcea3e17b546f339/a54d88e06612d820bc3be72877c74f257b561b19/c7be1ed902fb8dd4d48997c6452f5d7e509fbcdbe2808b16bcf4edce4c07d14e");

    if (matcher.matches()) {         
        final String md5 = matcher.group("md5");         
        final String sha1 = matcher.group("sha1");         
        final String sha256 = matcher.group("sha256");         
        ...
    }     
    ... 
}

As you can see, the example is pretty simple, it takes one line that contains a string and extracts the MD5, SHA1 and SHA256 hashes. We can see the code is easy to read and understand because everything is using human readable labels not just numbers to access the groups and there is no need to process the string with split operations or similars.

The syntax for the groups is:

(?<name>X)– X, as a named-capturing group

(?:X) – X, as a non-capturing group

With this, we can easily make our code easier to read and maintain when we are extracting information or doing some text processing.

For further information, check the Java documentation: Pattern (Java SE 10 & JDK 10)

Groups in Regular Expressions

Exploring the logs

fjavierm, , Leave a comment

As a developers an important part of our job sometimes is to fix problems in the different environments where our applications are deployed. Usually, this means to deal with huge log files to find where errors occur, and their stacktraces to add some context to the problem. The problem is that usually log files are verbose and contain a lot of information.

A couple of useful command to deal with this can be:

  • grep
  • zgrep

Both have the same purpose the only difference it that “grep” works with normal files and “zgrep” works with compressed (.gz) files. Usualy files are compressed due to the logs rotation scheduled in the servers. Both commands have multiple options and flags but, I am going to expose here two flags that have been useful multiple times:

  • -E expr: Allow as to supply a pattern for the search.
  • -C num: Print num lines of leading and trailing output context.
  • –color: Shows the matched information in color in the terminal.

As an example we have:

zgrep --color -E '(Sending email)' myLog.log-20170621.gz

grep --color -E '(Sending email)' myLog.log

grep --color -C 25 -E '(Sending email)' myLog.log

As we can see, obviously, they can be combined.

Exploring the logs