Ok, so we all probably know why keeping all Infinity Stones in one place is a bad idea, right? You must now be wondering what the relationship between Infinity Stones and Talend could be. Worry not, Thanos isn’t coming and there is a reasonable explanation behind the MCU fandom references, I promise.
I like to use this analogy to emphasize how distributing your Talend services among different servers according to Talend recommended architecture guidelines is as important as keeping the Infinity stones scattered across the universe and away from the clutches of evil. Are you following me?
In this blog, we will explore some common customer questions which come up during Talend architecture discussions and our answers to those queries.
Most Common query during planning stage
One of the most common queries I used to get while interacting with customers:
“What should be the right methodology to allocate various Talend services in our ecosystem? Shall I give you one gigantic server which can handle all our computational requirements?”
After going through the blog, I am sure that you will find the answer to this query yourself. Before going to the details, we need to agree that the Cloud technology has given power to the customers to select any type of servers quickly and they can set the server up and running in a matter of minutes. This means, theoretically, you can put all the Talend services in single server for demonstrational purposes. But is it a right approach? Before making the final call, we will discuss about various factors surrounding this scenario.
Going back to Monolithic server approach
The world had already moved out of monolithic systems when I started my IT career (well, it doesn’t mean I am too old or too young ?). One of the tendencies we are seeing is that some users are trying to go back to same monolithic environment patterns by taking a different route. This time, the monolithic server concept is wrapped carefully with gift paper in the form of a single high computational server available in various Cloud environments. Customers often overlook the fact that those servers are available in Cloud for specific high computational use cases like graphics processing, scientific modeling, dedicated gaming servers, machine learning etc.
From the Talend perspective, it’s always ideal to distribute various services (remember the Infinity Stones) to various servers as per recommended architecture.
Keeping all eggs in same basket problem
For argument’s sake, let’s put all the services of Talend for a Production environment in a single server. When a server failure occurs, this approach will bring down the entire Talend ecosystem.
This might be a costly affair especially if the enterprise is having specific data transfer SLAs between various systems. But if you are distributing the Talend environments in recommended architecture style, you can manage these types of failures in a graceful manner.
The battle for system resources
Still not convinced? Then I will take you to the next phase where the battle to grab system resources is happening. The basic scenario remains the same where you have installed all the services in single server.
Imagine you are going to pump a lot of data from source to target in batch format. We must also consider that multiple transformation rules need to be applied to the incoming data. This means your job server will be ready to grab a lot of available “treasure” (which is nothing more than system resources like CPU, memory, disk space etc). At the same time, you need to make sure that system resources are available for other services like TAC, Continuous Integration systems, Run time servers.
The tug-of-war for system resources will eventually lead to a big battle among various services. For example, let’s assume a case where TAC fails due to lack of memory. This means that you have lost all the control over the ecosystem and there is nobody to manage the services. If the victim is the run time server, it means your data flow through various web services and routes will start failing.
At the same time, if you start using a gigantic single server, you may not use the entire computational capacity all the time. It will be like gold plating all of your weapons and the result will be too much cost to maintain the underlying infrastructure.
Refreshing our minds with Talend Architecture
I hope, by now, all of you are convinced by the rationale of not keeping all the Infinity Stones Talend services in a single server. Before going further into detail on recommended architecture, let us quickly refresh our minds about various services involved in Talend Architecture. I will start with On-premise version. Below diagram will help you to understand the various services involved to handle both batch and streaming data. If you would like to understand more about each service, you can get the details from this link.
The Talend Cloud Architecture simplifies the overall landscape. You need to remember that you may still have to manage Remote Engines (either for Studio jobs or Pipeline Designer), Runtime Servers, Continuous Integration related activities etc.
If you would like to know more about Talend Cloud Architecture, I would recommend you have a look at the Cloud Data Integration User Guide.
Talend Recommended Architecture
The detailed description of Talend Recommended Architecture (including server sizing) for On-premise products can be referred from this link. I am not going to repeat the content, but I would like to show a high-level view of Talend recommended server layout for your quick reference.
The Cloud recommended architecture layout is much simpler since the Talend services are managed from Cloud environment. You can refer to the recommended Talend Cloud Architecture at this link. A quick peek of the server layout in the case of Talend Cloud is as shown below.
I hope this discussion on the “Infinity Stones” of Talend was as interesting for you as it was for me ?. Until I come up with another clever analogy to write a blog around, enjoy your time using Talend and keep those “stones” safe!