A few days ago I was talking with one of my acquaintances and he told me he has recently been the victim of a phishing attack. Luckily, he realised just after introduce his data in a bogus form something was not right, he could contact his service provider company and fix the problem without too many headaches.
During the conversation he showed me the message he received with the link he clicked on and, I was quick to notice it was a fake message. It was just an SMS message, not enough text to check as it can be done in an email but, the link was obviously (to me) a fake one.
From other conversations I have had with him in the past, I know he has some basic knowledge and security awareness, especially because his company make them do some basic training about it, which is very good, but when discussing with him, I realised he had never heard about the term domain impersonation and, other than a basic comment about checking links before you click them he was never given any examples of what to look for.
Trying to put my grain of salt out there to raise awareness, we are going to review quickly a few of the most common techniques and try to learn a bit more by example.
Let’s say we have a service provider that offers its services through a web page hosted on telecomexample.org. This will be our starting address, the real and original one. Now, let’s see what kind of techniques we can apply to mislead and trick unfortunate users.
Omission
This technique consists into skip one character on the original address. For example, telecmexample.org. As we can see, the example is omitting an “o“. The longer the address is, the easier is to miss that.
Typosquad: Substitution
This technique consists of replacing a character with a similar one. For example, telecomexemple.org. As we can see, we are replacing the “a” with an “e“. Other common replacement are: “i -> 1” or “i -> l“.
Homoglyph
This technique consists of replacing a character with another similar-looking character from a different alphabet. Usually from Latin, Greek or Cyrillic. For example, teʟecomexample.org. In this case, we have replaced the “l“.
Addition
This technique consists of adding an extra character to the address. For example, tellecomexample.org. Reading it carefully, we can see an extra “l” has been added.
Transposition
This technique just alters the order of one or more characters on the address. For example, telecomxeample.org. In this case, we have swapped the “e” and the “x“.
Homophone
This technique uses similar-sounding words such as “narrows” and “narroughs“. Like, telecomsample.org. Where the word “example” has been replaced by the similar-sounding word “sample“. Note: Probably there are better examples, but given the address domain, and not being a native speaker, it is hard, feel free to comment with better suggestions.
Subdomain
The service provider domain is used as a subdomain for a domain owned by the attackers. Such as telecomexample.accounts.org. Where the attacker owns the domain “accounts.org“.
TLD swap
Using the service provider domain but with a different top-level domain. For example, telecomexample.com. Where the top-level domain COM is used instead of the real one ORG.
Hyphenation
Adding an intermediate hyphen to the domain. Like, telecom-example.org. Where a hyphen has been added between the words “telecom” and “example“.
Added keywords
In this case, an extra word is added to the original domain trogon to mislead users. For example, telecomexamplelogin.org. Where the word “login” has been added to the original domain.
Today, just a short article but, I hope it helps to raise some awareness about very common impersonation domain techniques used by attackers to deceive users.se
The Java Virtual Machine (JVM) is a specification that provides a runtime environment in which Java code can be executed. It is the component of the technology responsible for its hardware and operating system independence. The JVM has two primary functions:
To allow Java programs to run on any device or operating system (“Write once, run anywhere” principle).
To manage and optimize program memory.
There are three important distinctions that need to be made when talking about the JMV:
JVM specification: The specification document describes an abstract machine, formally describes what is required in a JVM implementation, it does not describe any particular implementation of the Java Virtual Machine. Implementation details are left to the creativity of implementors. The specification document for version 16 can be found here.
JVM implementations: They are concrete implementations of the JVM specification. As it has been said before, it is up to the implementors how to develop and materialise the specification. This allows different implementations to focus on the improvement of different areas, prioritise different parts of the specification, or build non-standard implementations. Some reasons why to develop and implementation are:
Platform support: Run Java on a platform for which Oracle does not provide a JVM.
Resource usage: Tun Java on a device that does not have enough resources to run Oracle’s implementation.
Performance: Oracle’s implementation is not fast, scalable or predictable enough.
Licensing: Disagreement with Oracle’s licensing policy.
Competition: Offering an alternative.
Research or fun: Because, why not?
Some examples of implementations are Azu Zulu, Eclipse OpenJ9, Graals VM or, Hotspot.
JVM instances: It is a running implementation of the JVM.
JVM Architecture
The JVM consists of three distinct components:
Class Loader
Runtime Memory/Data Area
Execution Engine
Class Loader
Class loaders are responsible for loading dynamically Java classes into the data areas during runtime to the JVM. There are three phases in the class loading process: loading, linking, and initialization.
Loading
Loading involves taking the binary representation (bytecode) of a class or interface with a particular name, and generating the original class or interface from that. There are three built-in class loaders available in Java:
Bootstrap Class Loader: It loads the standard Java packages like java.lang, java.net, java.util, and so on. These packages are present inside the rt.jar file and other core libraries present in the $JAVA_HOME/jre/lib directory.
Extension Class Loader: The extension class loader is a child of the bootstrap class loader and takes care of loading the extensions of the standard core Java classes so that it is available to all applications running on the platform. Extensions are present in the $JAVA_HOME/jre/lib/ext directory.
Application Class Loader: It loads the files present on the classpath. By default, the classpath is set to the current directory of the application but, it can be modified.
Loading classes follow a hierarchical pattern, if a parent class loader (Bootstrap -> Extension -> Application) is unable to find a class, it delegates the work to a child class loader. If the last child class loader is not able to load the class either, it throws NoClassDefFoundError or ClassNotFoundException.
Linking
After a class is loaded into memory, it undergoes the linking process. Linking a class or interface involves combining the different elements and dependencies of the program together. Linking includes the following steps:
Verification: This phase checks the structural correctness of the .class file by checking it against a set of constraints or rules. If verification fails for some reason, we get a VerifyException. For example, if it has been compiled for a different version of Java.
Preparation: In this phase, the JVM allocates memory for the static fields of a class or interface, and initializes them with default values.
Resolution: In this phase, symbolic references are replaced with direct references present in the runtime constant pool.
Initialisation
Initialisation involves executing the initialisation method of the class or interface. This can include calling the class’s constructor, executing the static block, and assigning values to all the static variables. This is the final stage of class loading.
Runtime Data Area
Method Area
The Runtime Data Area is divided into five major components:
All the class level data such as the run-time constant pool, field, and method data, and the code for methods and constructors, are stored here. If the memory available in the method area is not sufficient for the program startup, the JVM throws an OutOfMemoryError.
Important to point the method area is created on the virtual machine start-up, and there is only one method area per JVM.
Heap Area
All the objects and their corresponding instance variables are stored here. This is the run-time data area from which memory for all class instances and arrays is allocated.
Again, important to point the heap is created on the virtual machine start-up, and there is only one heap area per JVM.
Stack Area
Whenever a new thread is created in the JVM, a separate runtime stack is also created at the same time. All local variables, method calls, and partial results are stored in the stack area. If the processing been done in a thread requires a larger stack size than what’s available, the JVM throws a StackOverflowError.
For every method call, one entry is made in the stack memory which is called the Stack Frame. When the method call is complete, the Stack Frame is destroyed.
The Stack Frame is divided into three sub-parts:
Local Variables: Each frame contains an array of variables known as its local variables. All local variables and their values are stored here. The length of this array is determined at compile-time.
Operand Stack: Each frame contains a last-in-first-out (LIFO) stack known as its operand stack. This acts as a runtime workspace to perform any intermediate operations. The maximum depth of this stack is determined at compile-time.
Frame Data: All symbols corresponding to the method are stored here. This also stores the catch block information in case of exceptions.
Program Counter (PC) Registers
The JVM supports multiple threads at the same time. Each thread has its own PC Register to hold the address of the currently executing JVM instruction. Once the instruction is executed, the PC register is updated with the next instruction.
Native Method Stacks
The JVM contains stacks that support native methods. These methods are written in a language other than Java, such as C and C++. For every new thread, a separate native method stack is also allocated.
Execution Engine
Once the bytecode has been loaded into the main memory, and details are available in the runtime data area, the next step is to run the program. The Execution Engine handles this by executing the code present in each class.
However, before executing the program, the bytecode needs to be converted into machine language instructions. The JVM can use an interpreter or a JIT compiler for the execution engine.
The Execution Engine has three main components:
Interpreter
The interpreter reads and executes the bytecode instructions line by line, due to this line by line execution, the interpreter is comparatively slower. In addition, if a method is called multiple times, every time a new interpretation is required.
JIT Compiler
The JIT Compiler neutralizes the disadvantage of the interpreter. The Execution Engine will be using the help of the interpreter in converting byte code, but when it finds repeated code it uses the JIT compiler, which compiles the entire bytecode and changes it to native code. This native code will be used directly for repeated method calls, which improve the performance of the system. The JIT Compiler has the following components:
Code Optimizer: Optimizes the intermediate code for better performance.
Target Code Generator: Converts intermediate code to native machine code.
Profiler: Finds the hotspots (code that is executed repeatedly)
Garbage Collector
The Garbage Collector (GC) collects and removes unreferenced objects from the heap area. It is the process of reclaiming the runtime unused memory automatically by destroying them. Garbage collection makes Java memory-efficient because it frees space for new objects.
Garbage Collections is done automatically by the JVM at regular intervals and does not need to be handled separately but, It can also be triggered by calling System.gc() with the execution not guaranteed.
It involves two phases:
Mark: In this step, the GC identifies the unused objects in memory.
Sweep: In this step, the GC removes the objects identified during the previous phase.
The JVM contains 3 different types of garbage collectors:
Serial GC: This is the simplest implementation of GC, and is designed for small applications running on single-threaded environments. It uses a single thread for garbage collection. When it runs, it leads to a stop-the-world event where the entire application is paused.
Parallel GC: This is the default implementation of GC in the JVM, and is also known as Throughput Collector. It uses multiple threads for garbage collection but still pauses the application when running.
Garbage First (G1) GC: G1GC was designed for multi-threaded applications that have a large heap size available (more than 4GB). It partitions the heap into a set of equal size regions and uses multiple threads to scan them. G1GC identifies the regions with the most garbage and performs garbage collection on that region first.
Concurrent Mark Sweep (CMS) GC: Deprecated on Java 9 and removed on Java 14.
Java Native Interface (JNI)
JNI acts as a bridge for permitting the supporting packages for other programming languages such as C, C++, and so on. This is especially helpful in cases where you need to write code that is not entirely supported by Java, like some platform-specific features that can only be written in C.
Native Method Libraries
Native Method Libraries are libraries that are written in other programming languages, such as C, C++, and assembly. These libraries are usually present in the form of .dll or .so files. These native libraries can be loaded through JNI.
JVM memory structure
As exposed earlier, JVM manages the memory automatically with the help of the garbage collector. Memory management is the process of the allocation & de-allocation of the objects from a memory.
We have already described the main memory areas on the “Runtime Data Area” but, let’s explore a bit more how they work.
Heap Area
Heap space in Java is used for dynamic memory allocation for Java objects and JRE classes at the runtime. New objects are always created in heap space and the references to these objects are stored in stack memory. These objects have global access and can be accessed from anywhere in the application. This memory model is further broken into smaller parts called generations, these are:
Young Generation: This is where all new objects are allocated and aged. A minor Garbage collection occurs when this fills up.
Eden space: It is a part of the Young Generation space. When we create an object, the JVM allocates memory from this space.
Survivor space: It is also a part of the Young Generation space. Survivor space contains existing objects which have survived the minor GC phases of GC. There a Survivor Space 0 and Survivor Space 1.
Old or Tenured Generation: This is where long surviving objects are stored. When objects are stored in the Young Generation, a threshold for the object’s age is set and when that threshold is reached, the object is moved to the old generation.
Permanent Generation: This consists of JVM metadata for the runtime classes and application methods.
This area works as follows:
When an object is created, it first allocated to Eden space because this is not that big and gets full quite fast. The garbage collector runs on the Eden space and clears all non-reference object.
When the GC runs, it moves all objects surviving the garbage collecting process into the Survivor space 0. And, if they still survive, object in Survivor Space 0 into Survivor space 1.
If an object survives for X rounds of the garbage collector (X depends on the JVM implementation), it is most likely that it will survive forever, and it gets moved into the Old space.
Metaspace (PermGen)
Metaspace is a new memory space starting from the Java 8 version; it has replaced the older PermGen memory space.
Metaspace is a special heap space separated from the main memory heap. The JVM keeps track of loaded class metadata in the Metaspace. Additionally, the JVM stores all the static content in this memory section. This includes all the static methods, primitive variables, and references to the static objects. It also contains data about bytecode, names, and JIT information.
Class metadata are the runtime representation of java classes within a JVM process, basically any information the JVM needs to work with a Java class. That includes, but is not limited to, runtime representation of data from the JVM class file format.
Metaspace is only released when a GC did run and unload class loaders.
Performance Enhancements
On the Oracle documentation, some performance enhancements can be found. These enhancements are:
Compact Strings
Tiered Compilation
Compressed Ordinary Object Pointer
Zero-Based Compressed Ordinary Object Pointers
Escape Analysis
For details on the enhancements better check the documentation where there is a solid explanation of these topics.
Tuning the Garbage Collector
Tuning should be the last option we use for increasing the throughput of the application and only when we see a drop in performance because of longer GC causing application timeouts.
Java provides a lot of memory switches that we can use to set the memory sizes and their ratios. Some of the commonly used memory switches are:
-Xms For setting the initial heap size when JVM starts
-Xmx For setting the maximum heap size
-Xmn For setting the size of the young generation (rest is old generation)
-XX:PermGen For setting the initial size of the Permanent Generation Memory
-XX:MaxPermGen For setting the maximum size of Perm Gen
-XX:SurvivorRatio For providing a ratio of Eden space
-XX:NewRatio For providing a ratio of old/new generation sizes. The default value is 2
-XX:+UserSerialGC For enable Serial garbage collector
-XX:+UseParallelGC For enable Parallel garbage collector
-XX:+UseConcmarkSweepGC For enable CMS garbage collector
-XX:+ParallelCMSThreads For enabling CMS Collector as number of threads to use
-XX:+UseG1GC For enable G1 garbage collector
-XX:HeapDumpOnOutOfMemory Pass a parameter to create a heap dump file when this error happens next time.
Tools
After all this explanation and, in addition to all the configurations, a very interesting point is the monitorization of our Java memory. To do this we have multiple tools we can use:
jstat
It is a utility that provides information about the performance and resource consumption of running java applications. We can use the command with the garbage collection option to obtain various information from a java process.
S0C – Current survivor space 0 capacity (KB)
S1C – Current survivor space 1 capacity (KB)
S0U – Survivor space 0 utilization (KB)
S1U – Survivor space 1 utilization (KB)
EC – Current eden space capacity (KB)
EU – Eden space utilization (KB)
OC – Current old space capacity (KB)
OU – Old space utilization (KB)
MC – Metasapce capacity (KB)
MU – Metaspace utilization (KB)
CCSC – Compressed class space capacity (KB)
CCSU – Compressed class space used (KB)
YGC – Number of young generation garbage collector events
YGCT – Young generation garbage collector time
FGC – Number of full GC events
FGCT – Full garbage collector time
GCT – Total garbage collector time
jmap
It is a utility to print the memory-related statistics for a running VM or core file. It is a utility for enhanced diagnostics and reduced performance overhead.
jcmd
It is a utility used to send diagnostic command requests to the JVM, where these requests are useful for control, troubleshoot, and diagnose JVM and Java applications. It must be used on the same machine where the JVM is running, and have the same effective user and group identifiers that were used to launch the JVM.
jhat
It is a utility that provides a convenient browser base easy to use Heap Analysis Tool (HAT). The tool parses a heap dump in binary format (e.g., a heap dump produced by jcmd).
This is slightly different from other tools because when we execute it, it creates a webserver we can access from our browser to read the results.
VisualVM
VisualVM allows us to get detailed information about Java applications while they are running on a JVM and it can be in a local or a remote system also possible to save and capture the data about the JVM software and save data to the local system. VisualVM can do CPU sampling, memory sampling, run garbage collectors, analyze heap errors, take snapshots, and more.
JConsole
It is a graphical monitoring tool to monitor JVM and Java applications both on a local or remote machine. JConsole uses the underlying features of JVM to provide information on the performance and resource consumption of applications running on the Java platform using Java Management Extension (JMX) technology.
Memory Analyzer (MAT)
The Eclipse Memory Analyzer is a fast and feature-rich graphical Java heap analyzer that helps you find memory leaks and reduce memory consumption.
I imagine that everyone reading this blog should be, by now, familiar with the term Infrastructure as Code (IaC). If not because we have written a few articles on this blog, probably because it is a widely extended-term nowadays.
At the same time I assume familiarity with the IaC term, I have not heard a lot of people talking about a similar concept called Diagram as Code (DaC) but focus on diagrams. I am someone that, when I arrive at a new environment, finds diagrams very useful to have a general view of how a new system works or, when given explanations to a new joiner. But, at the same time, I must recognise, sometimes, I am not diligent enough to update them or, even worst, I arrived at projects where, or they are too old to make sense or there are none.
The reasons for that can be various, it can be hard for developers to maintain diagrams, lack of time, lack of knowledge on the system, not obvious location of the editable file, only obsolete diagrams available and so on.
I have been playing lately with the DaC concept and, with a little bit of effort, all new practices require some level of it till they are part of the workflow, it can fill this gap and help developers and, other profiles in general, to keep diagrams up to date.
In addition, there are some extra benefits of using these tools such as easy version control, the need for only a text editor to modify the diagram and, the ability to generate the diagrams everywhere, even, as part of our pipelines.
This article is going to cover some of the tools I have found and play with it lately, the goal is to have a brief introduction to the tools, be familiar with their capabilities and restrictions and, try to figure out if they can be introduced on our production environments as a long term practise. And, why not, compare which tool offers the most eye-catching ones, in the end, people are going to pay more attention to these ones.
Just a quick note before we start, all the tools we are going to see are open source tools and freely available. I am not going to explore any payment tool and, I am not involved in any way in the tools we are going to be using.
Graphviz
The first tool, actually a library, we are going to see is Graphviz. In advance, I am going to say, we are not exploring this option in depth because it is too low level for my taste or the purpose we are trying to achieve. It is a great solution if you want to generate diagrams as part of your applications. In fact, some of the higher-level solution we are going to explore use this library to be able to generate the diagrams. With that said, they define themselves as:
Graphviz is open source graph visualization software. Graph visualization is a way of representing structural information as diagrams of abstract graphs and networks. It has important applications in networking, bioinformatics, software engineering, database and web design, machine learning, and in visual interfaces for other technical domains.
The Graphviz layout programs take descriptions of graphs in a simple text language, and make diagrams in useful formats, such as images and SVG for web pages; PDF or Postscript for inclusion in other documents; or display in an interactive graph browser. Graphviz has many useful features for concrete diagrams, such as options for colors, fonts, tabular node layouts, line styles, hyperlinks, and custom shapes.
The installation process in an Ubuntu machine is quite simple using the package manager:
sudo apt install graphviz
As I have said before, we are not going to explore deeper this library, we can check on the documentation for some examples built in C and, the use of the command line tools it offers but, for my taste, it is a too low-level solution to directly use it when implementing DaC practises.
PlantUML
The next tool is PlantUML. Regardless of what the name seems to imply, the tool is able to generate multiple types of diagrams, all of them listed on their web page. Some examples are:
The diagrams are defined using a simple and intuitive language and, images can be generated in PNG, SVG or, LaTeX format.
They provide an online tool it can be used for evaluation and testing purposes but, in this case, we are going to be installing it locally, especially to test how hard it is and how integrable is in our CI/CD pipelines as part of our evaluation. By the way, this is one of the tools it uses Graphviz behind the scenes.
There is no installation process, the tool just needs the download of the corresponding JAR file from their download page.
Now, let’s use it. We are going to generate a State Diagram. It is just one of the examples we can find on the tool’s page. The code used to generate the example diagram is the one that follows and is going to be stored in a TXT file:
@startuml
scale 600 width
[*] -> State1
State1 --> State2 : Succeeded
State1 --> [*] : Aborted
State2 --> State3 : Succeeded
State2 --> [*] : Aborted
state State3 {
state "Accumulate Enough Data\nLong State Name" as long1
long1 : Just a test
[*] --> long1
long1 --> long1 : New Data
long1 --> ProcessData : Enough Data
}
State3 --> State3 : Failed
State3 --> [*] : Succeeded / Save Result
State3 --> [*] : Aborted
@enduml
Now, let’s generate the diagram:
java -jar ~/tools/plantuml.jar plantuml-state.txt
The result is going to be something like:
State Diagram generated with PlantUML
As we can see, the result is pretty good, especially if we consider that it has taken us around five minutes to write the code without knowing the syntax and we have not had to deal with positioning the element on a drag and drop screen.
Taking a look at the examples, the diagrams are not particularly beautiful but, the easy use of the tool and the variety of diagrams supported makes this tool a good candidate for further exploration.
WebSequenceDiagrams
WebSequenceDiagrams is just a web page that allows us to create in a quick and simple way Sequence Diagrams. It has some advantages such as offering multiple colours, there is no need to install anything and, having only one purpose, it covers it quite well in a simple way.
We are not going to explore this option further because it does not cover our needs, we want more variety of diagrams and, it does not seem integrable on our daily routines and CI/CD pipelines.
Asciidoctor Diagram
I assume everyone is more or less aware of the existence of the Asciidoctor project. The project is a fast, open-source text processor and publishing toolchain for converting AsciiDoc content to HTML5, DocBook, PDF, and other formats.
Asccidoctor Diagram is a set of Asciidoctor extensions that enable you to add diagrams, which you describe using plain text, to your AsciiDoc document.
The installation of the extension is quite simple, just a basic RubyGem that can be installed following the standard way.
gem install asciidoctor-diagram
There are other options of usage but, we are going to do an example using the terminal and, using the PlantUML syntax we have already seen.
[plantuml, Asciidoctor-classes, png]
....
class BlockProcessor
class DiagramBlock
class DitaaBlock
class PlantUmlBlock
BlockProcessor <|-- DiagramBlock
DiagramBlock <|-- DitaaBlock
DiagramBlock <|-- PlantUmlBlock
....
The result been something like:
Generated with Asciidoctor Diagrams extension
One of the advantages of this tool is that it supports multiple diagram types. As we can see, we have used the PlantUML syntax but, there are many more available. Check the documentation.
Another of the advantages is it is based on Asciidoctor that is a very well known tool and, in addition to the image it generates an HTML page with extra content if desired. Seems worth it for further exploration.
Structurizr
I was going to skip this one because, despite having a free option, requires some subscription for determinate features and, besides, it does not seem as easy to integrate and use as other tools we are seeing.
Despite all of this, I thought it was worth it to mention it due to the demo page they offer where, with just some clicking, you can see the diagram expressed on different syntaxes such as PlantUML or WebSequenceDiagrams.
Diagrams
Diagrams is a tool that seems to have been implemented explicitly to follow the Diagram as Code practice focus on infrastructure. It allows you to write diagrams using Python and, in addition to support and having nice images for the main cloud providers, it allows you to fetch non-available images to use them in your diagrams.
Installation can be done using any of the available common mechanism in Python, in our case, pip3.
pip3 install diagrams
This is another one of the tools that, behind the scenes, uses Graphviz to do its job.
Let’s create our diagram now:
from diagrams import Cluster, Diagram
from diagrams.onprem.analytics import Spark
from diagrams.onprem.compute import Server
from diagrams.onprem.database import PostgreSQL
from diagrams.onprem.inmemory import Redis
from diagrams.onprem.aggregator import Fluentd
from diagrams.onprem.monitoring import Grafana, Prometheus
from diagrams.onprem.network import Nginx
from diagrams.onprem.queue import Kafka
with Diagram("Advanced Web Service with On-Premise", show=False):
ingress = Nginx("ingress")
metrics = Prometheus("metric")
metrics << Grafana("monitoring")
with Cluster("Service Cluster"):
grpcsvc = [
Server("grpc1"),
Server("grpc2"),
Server("grpc3")]
with Cluster("Sessions HA"):
master = Redis("session")
master - Redis("replica") << metrics
grpcsvc >> master
with Cluster("Database HA"):
master = PostgreSQL("users")
master - PostgreSQL("slave") << metrics
grpcsvc >> master
aggregator = Fluentd("logging")
aggregator >> Kafka("stream") >> Spark("analytics")
ingress >> grpcsvc >> aggregator
And, let’s generate the diagram:
python3 diagrams-web-service.py
With that, the result is something like:
Diagram generated with Diagrams
As we can see, it is easy to understand and, the best part, it is quite eye-catching. And, everything looks in place without the need to mess with a drag and drop tool to position our elements.
Conclusion
As always, we need to evaluate which tool is the one that best fit our use case but, after seeing a few of them, my conclusions are:
If I need to generate infrastructure diagrams I will go with the Diagrams tools. Seems very easy to use been based on Python and, the results are very visually appealing.
For any other type of diagram, I will be inclined to use PlantUML. It seems to support a big deal of diagram types and, despite not being the most beautiful ones, it seems the results can be clear and useful enough.
Asciidoctor Diagrams seems a good option if your team or organisation is already using Asciidoctor and, it seems a good option if we want something else than just a diagram generated.
The ambit of IT, software development, operations or similar tends to be full of people that likes to try new trends or tools related directly with their day to day tasks or just out of curiosity. One quick way of doing this, it is to install all the tools and libraries in our machines and, after we have finished, try to clean everything or, at least, revert all the mistakes or, not very good practices we did when learning. Despite this been a valid way, overtime, our machines get polluted will lost dependencies, configuration files or libraries.
To avoid that, it seems a better way to try all the new stuff on an isolated environment and, if we like it and we decide do use it in our daily environments, to install it from scratch again probably correcting some initial mistakes or avoiding some bad practices.
There are plenty of solutions out there to achieve this and, to have an easy to set up throw-away environment. Most of them based on virtual machines or some kind of virtualisation. More traditional ones such as VirtualBox or VMWare or, some based on management solutions for virtual machines such as Vagrant.
Today, I just want to bring to the table a different one I have been playing with lately and, I did not know a few months ago. I do not know how popular is it or how extended it is but, I think that knowing different options it is always a plus. The tool is called Multipass. And, as Ubuntu describes it, it is “Ubuntu VMs on demand for any workstation. Multipass can launch and run virtual machines and configure them with cloud-init like a public cloud. Prototype your cloud launches locally for free.”
I have found it very easy to use and, for the purposes of having trow-away isolated environments laying around, quite useful.
We are going to see the install process and, the basic execution of a few commands related with an instance.
Installation
Before we start applying the steps to install Multipass on our machines, there are a couple of requirement we need to consider. They are related with the platform is going to be used to virtualise the images. In new operative systems, no extra requirements are needed but, some old ones have them. Check on the official documentation.
For Linux:
sudo snap install multipass
For Windows:
Just download the installer from the web page and proceed with the suggested steps.
For MacOS:
MacOS offers us two different alternatives. One based on an installation file similar to Windows and, one based on a package manager solution like Homebrew. If installing using the installation file, just execute it and follow the suggested steps and, if installing using Homebrew just execute the appropriate command (here):
brew install --cask multipass
Once the installation is done, any other command executed should be the same in all three environments.
Just as a side note, there is the possibility of using VirtualBox as a virtualisation platform if we desire it but, this is completely optional and depends only on our preferences. I am not using it. The command to install it can be found below but, I encourage you to go to the official documentation on this specific point.
Now we have finished the installation, let’s create our first instance.
Creating and using an instance
Let’s check what images are available:
‘find’ execution – List of available images
We can see there are multiple images available but, in this case, we are going to create an instance using the latest version (20.10). By default, if not image is specified, multipass uses the las LTS version.
It is worth it to mention that, by default, multipass assign some values to our instance in terms of CPU, disk, memory and, others.
default values – multipass launch -h‘launch’ execution – Creates a new instance
As we can see it is quite fast and, if we create a second image, it will be even faster.
We can execute a command inside the instance:
‘exec’ execution – Executes a command inside the instance
Or, we can just login into the instance:
‘shell’ execution – Login into the instance
From now on, we can just work with this instance as we please.
There are a few commands we can use to manage our instances such as instances running, available instances or information about running instances. All these command are available on the help menu.
‘help’ execution – List available commands
Mounting a shared folder
Multipass offers the possibility of mounting folders to share information between the host and the instance using the command mount.
Sharing a folder between host and instance
Cleaning (Deleting an instance)
Finally, as we do not want to leave throw-away instances laying around, after we have finished working, we can remove it.
‘delete’ execution – Removes an instance
This is just a brief introduction to multipass. More complex scenarios can be found on the official documentation.
We live in a containerised world. Container solutions like Docker are now so extended that they are not a niche thing any more or a buzzword, they are mainstream. Multiple companies use it and, the ones that do not are dreaming with it probably.
The only problems are that they are still something new. The adoption of them has been fast and, it has arrived like a storm to all kind of industries that use technology. The problem is that from a security point of view we, as an industry, do not have all the awareness we should have. Containers and, especially, containers running on cloud environments are hidden partially the fact that they exist and they need to be part of our security considerations. Some companies use them thinking they are completely secure, trusting the cloud providers or the companies that generate the containers take care of everything and, even, for less technology focus business, they are an abstraction and not real and tangible thing. They are not the old bare metal servers, the desktop machines or the virtual machines they were used to it, and till a certain point, they worried because they were things that could be touched.
All of that has made that while security concerns for web applications are first-level citizens, not as much as it should but the situation has improved a lot on the last few years, security concerns about containers seem to be the black sheep of the family, no one talks about it. And, this is not right. It should have the same level of concern and the same attention should be paid to it and, be part of the development life cycle.
In the same way that web applications can be attacked in multiple ways, containers have their own attack vectors, some of which we are going to see here. We will see that some of the attack vectors can be easily compared with known attack vectors on spaces we are more aware like web applications.
Vulnerable application code
Containers package applications and third-party dependencies that can contain known flaws or vulnerabilities. There are thousands of published vulnerabilities that attackers can take advantage to exploit our systems if found on the applications running inside the containers.
The best to try to avoid running container with known vulnerabilities is to scan the images we are going to deploy and, not just as a one-time thing. This should be part of our delivery pipelines and, the scans should apply all the time. In addition to known vulnerabilities, scanners should try to find out-of-date packages that need an update. Even, some available scanners try to find some possible malware on the images.
Badly configured container images
When configuring how a container is going to be built some vulnerabilities can be introduced by mistake or if not the proper attention is paid to the building process that can be later exploited by attackers. A very common example is to configure the container to run with unnecessary root permissions giving it more privileges on the host than it really needs.
Build machine attacks
As any piece of software, the one we use to run CI/CD pipelines and build container images can be attacked successfully and, attackers can add malicious code to our containers during the build phase obtaining access to our production environment once the containers have been deploy and, even, utilising these compromised containers to pivot to other parts of our systems or networks.
Supply chain attacks
Once containers have been built they are stored in registries and retrieved or “pulled” when they are going to be run. Unfortunately, no one can guarantee the security of this registries and, an attacker can compromise the registry an replace the original image with a modified one including a few surprises.
Badly configured containers
When creating configuration files for our containers, i.e. a YAML file, we can make some mistakes and add configurations to the containers we did not need. Some possible examples are unnecessary access privileges or unnecessary open ports.
Vulnerable host
Containers run on host machines and, in the same way, we try to ensure containers are secure host should be too. Some times they run old versions of orchestration component with known vulnerabilities or other components for monitorisation. A good idea is to minimise the number of components installed on the host, configure them correctly and apply security best practices.
Exposed secrets
Credentials, tokens or passwords are all of them necessary if we want our system to be able to communicate with other parts of the system. One risk is the way we supply the container and the applications running in it these secret values. There are different approaches with varying levels of security that can be used to prevent any leakage.
Insecure networking
The same than non containerised applications, containers need to communicate using networks. some level of attention will be necessary to set up secure connections among components.
Container escape vulnerabilities
Containers are prepared to run on isolation from the hosts were they are running, in general, all container runtimes like “containerd” or “CRI-O” have been heavily tested and are quite reliable but, as always, there are vulnerabilities to be discovered. Some of these vulnerabilities can let malicious code running inside a container escape out into the host. Due to the severity of this, some stronger isolation mechanisms can be worth to consider.
Some other risks related to containers but not directly been containers can be:
Attacks to code repositories of application deployed on the containers poisoning them with malicious code.
Hosts accessible from the Internet should be protected as expected with other tools like firewalls, identity and access management systems, secure network configurations and others.
When container run under an orchestrator, i.e. Kubernetes, a door to new attack vectors is open. Configurations, permission or access not controlled properly can give attackers access to our systems.
As we can see some of the attack vectors are similar to the one existing in more mature areas like networking or web application but, due to the abstraction and the easy-to-use approach, the security on containers, unfortunately, is left out the considerations.
Today, we are going to talk about PostgreSQLAdvisory Locks. This kind of locks are created by the application and developers and, they have meaning inside the application, PostgreSQL does not enforce their use and they are there to fulfil a business or coding specific case. I was going to try to explain and to add some literature around them but, after reading PostgreSQL documentation (can be found here) I do not think it is necessary because the definition it is easy to understand and, besides, on the same page we can find the other types of locks available giving us some extra context. Instead, we are going to see some real-world code as an example.
Let’s say we have our shiny service that runs multiple instances at the same time on our production environment and, on that services, we run a scheduled task that updates one of our database tables adding a different sequence number to the existing rows (buildings) for all the existing cities. Something like:
id (uuid)
city (text)
building. (text)
registered (timestamp)
occurrence (bigint)
e6448a82
London
British Museum
2021/02/01 13:00:00.000
null
97347903
London
Tower of London
2021/02/01 12:59:59.999
null
7befe492
Paris
Eiffel Tower
2021/01/31 07:23:34.294
null
b426681a
Paris
Louvre Museum
2021/02/01 12:59:59.999
null
156e1f89
London
Big Ben
2021/02/01 12:59:59.999
null
Table ‘buildings’
For some curious minds about the reason why we need this ‘occurrence‘ sequence, one of the cases can be to create an endpoint to allow other systems to synchronise these buildings. We could sort using the ‘registered‘ field but, it can happen that two buildings in the same city can be registered at the same time making it impossible to warrantee the information is going to be returned always on the same order, and this can cause synchronisation problems or even missing a building due to paginated requests. We want to be able to sort them in an immutable way.
Going back to the multiple services running the tasks, we can have some ugly situations were one of the tasks running is working already and, in the middle of updating a city, when another task in a different service start processing the same city, especially if we do this on batches due to the huge amount of data we store.
One simple solution of this is to use Advisory Locks allowing us, developers, to lock a city when the task is updating it. For this purpose, PostgreSQL offers us two nice functions to work with:
pg_advisory_lock: Obtains an exclusive session-level advisory lock, waiting if necessary.
pg_try_advisory_lock: Obtains an exclusive session-level advisory lock if available. This will either obtain the lock immediately and return ‘true‘, or return ‘false‘ without waiting if the lock cannot be acquired immediately.
The full list of system administration functions related with advisory locks can be found here.
For the purposes of the example code, we are going to implement, we will be using the second one because it makes sense if one city it is been processed, we do not want to process it again till the next scheduled time.
public void assignOccurrenceSequences() {
final List<String> cities = buildingDao.retrievePendingCities();
for (final String city : cities) {
final int lockId = Math.abs(Hashing.sha256().newHasher()
.putString(city, StandardCharsets.UTF_8)
.hash().asInt());
logger.info("Taking advisory_lock {} for city {} ", lockId, city);
try (Connection connection = dataSource.getConnection()) {
connection.setAutoCommit(true);
final boolean lockObtained;
try (Statement statement = connection.createStatement()) {
lockObtained = statement.execute(format("select pg_try_advisory_lock(%d)", lockId));
}
if (lockObtained) {
try {
final int updates = buildingDao.populateOccurrenceSequences(city);
logger.info("Assigning {} sequences for city {}", updates, city);
} finally {
try (Statement statement = connection.createStatement()) {
statement.execute(format("select pg_advisory_unlock(%d)", lockId));
}
logger.info("Released advisory_lock {} for city {}", lockId, city);
}
} else {
logger.info("advisory_lock {} for city {} already taken", lockId, city);
}
} catch (SQLException e) {
throw new IllegalStateException(e);
}
}
}
On lines 5, 6 and 7 we create a unique lock id we will be using to establish the lock and make sure all the tasks running calculate the same id. And yes, before someone points it, we are assuming that ‘city‘ is unique. With this generated lock id, we can try to acquire the lock. In case of success, we proceed with the update. In case of fail, we skip that city and proceed with the rest of the cities.
Distributed architecture styles, while much more powerful in terms of performance, scalability and availability than monolithic architecture styles, have significant trade-offs. One of the groups of issues is known as the fallacies of distributed computing.
A fallacy is something that is believed or assumed to be true but is not. All fallacies when analysed are common sense, even they are things we experiment every day but, for some reason, they are some times forgotten when designing new distributed systems.
Fallacy #1: The Network is Reliable
Ok, with the cloud environments we do not trip over network cables anymore but, while networks have become more reliable over time, the fact is that networks still remain generally unreliable, this being a factor that influences distributed system due to its reliance on the network for communication.
Fallacy #2: Latency is Zero
Let’s face it, the network sometimes goes faster and sometimes slower, is there someone out there it has never seen a streaming film suddenly freezing for a few seconds when the killer is behind the main character? Communication latency on a network is never zero and, it is always bigger than local latencies. When working with distributed systems we need to consider the latency average. And, not only that, we should consider the 95th to 99th percentile too because the values can differ.
Fallacy #3: Bandwidth is Infinite
We are sure about that, right? Explain that to your three flatmates when you are trying to call home and they are, independently, watching their favourite film on a streaming platform. The fact of the system been distributed increases the amount of information that travels through the network and every byte matters. Maybe, a simple request of 200 kilobytes seems small but multiply it for the number of requests made per second and, include all the request among services performed at the same time. This number can grow easily.
Fallacy #4: The Network is Secure
Just two words “cybercriminals everywhere” (no reason to be scared). The surface area for threats and attacks increases by magnitudes when moving from a monolithic to a distributed architecture. We know the need to secure all endpoints, even when communicating among internal services.
Fallacy #5: The Topology Never Changes
Raise your hand if you think IT members never change anything on your network over time. No hands. Good! Routers, hubs, switches, firewalls, networks and appliances used, even cloud networks can suffer changes or need updates and modifications that can affect services communications or latencies on the network.
Fallacy #6: There is Only One Administrator
Have you ask someone from IT to do something and asked later about the progress to a different person in the department to just need to explain your request again because the request was never logged? That happens, a lot, sometimes things get lost, the coordination is not good enough, the communication is not good enough or … you see where I am going.
Fallacy #7: Transport Cost is Zero
If you have an internet connection at home, probably your internet provider sends you a bill from time to time, if it does not, please stop using your neighbours’ Wi-Fi. It is not exactly the same but exemplifies that to be able to communicate certain infrastructure and network topology are necessary. The needs of monolithic applications are substantially different from the needs of distributed systems. servers, firewalls, multiple load balancers, proxies…
Fallacy #8: The Network is Homogeneous
I do not have a more mundane example for this one to make it easy to remember but a real one should be simple enough. A company using multiple cloud services at the same time. All of them are going to work well initially but, not all of them have been exactly built and tested in the same way. It can be differences in the services like latency or reliability, basically, everything named on the previous fallacies.
Reference: “Fundamentals of Software Architecture by Mark Richards and Neal Ford (O’Reilly). Copyright 2020 Mark Richards, Neal Ford, 978-1-492-04345-4″
During the last few years, we have been hearing about a lot of new practices applying to Software Development and DevOps. One of these topics is Infrastructure as Code. Probably, in this space, two of the most well-known solutions are Terraform and CloudFormation.
We have already discussed Terraform on this blog previously. If we take a look to the basic code on the example on this blog or you are already using it, probably, you are aware that when the infrastructure grows, the Terraform code, been quite verbose, grows fast too and, unless we have a very good code structure it can get very messy. The same can be said about CloudFomation. In addition, they are not as developer-friendly as common programming languages are and, they need to be learned as a new language.
To solve the first problem, code turning messing over time, there are some measures to split the Terraform code like creating modules but, for much of the projects I have been taken a look and articles I have read about it, it seems there is no agreement about the best way to split the code and, usually, if you do not work regularly with this kind of projects, it is very hard to find things and move around.
Trying to solve this problem and the second one, using languages well know by developers, there are projects like Pulumi trying to bring infrastructure code to use familiar programming languages and tools, including packaging and APIs.
Investigating a little bit about this, I have found another one, AWS Cloud Development Kit (AWS CDK), which is a software development framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. As said, initially AWS CDK is designed for CloudFormation but, there is an implementation to generate Terraform code.
Today’s purpose is just to play a little bit with this technology, just the one that generates CloudFormation code and not to compare them (I do not know enough about it to compare them…yet).
I was going to write a long post with examples but, I have found a great introduction workshop offered by AWS that it does the job nicely. For this reason, I am just leaving here my project on Github.
One of the nice things I have found about AWS CDK is that support multiple languages:
Typescript
Javascript
Python
Java
C#
And, after writing a few lines, it feels very comfortable to be writing code using Java APIs and using all the power of the multiple tools that exist on the Java ecosystem. Ond, of course, be using Java packages to sort the infrastructure code.
Just another alternative in our toolbox that seems worth it to explore.
Binary Coders 