Achieving Unlimited Scale Using Talend ESB & Auto-scaling on AWS
Can you believe it? The holidays are upon us again and many online shopping retailers are getting ready to scale up for the season. Online shopping companies always need to design a stable and scalable environment for their web services applications – especially with the need to be 100% available during sales events (Christmas sales, Black-Friday sales, etc).
It’s becoming very challenging for them if they need to have heavy capacity preparation activities before each event, such as bringing extra server infrastructure, deploying web service code into those new servers, manual configurations on load balancer, monitoring and administration tools. This list goes on and on …
However, we think that Talend and Amazon Web Services has a solution, an auto-scaling one at that. Below, the diagram shows how you can design an auto-scaling online shopping system by using Talend and AWS technologies, to fit with stable demand patterns or usage variabilities (such as sales events).
Let’s run through some of the main technologies and components used here:
- AWS Auto Scaling - This helps to maintain application availability and allows you to scale your Amazon EC2 capacity up or down automatically according to conditions defined. You can use Auto Scaling to help ensure that you are running your desired number of Amazon EC2 instances and enable your environment to automatically increase the number of Amazon EC2 instances during demand spikes to maintain performance and decrease capacity during lulls to reduce costs.
- Talend ESB Runtime – Talend ESB runtime provides an Apache Karaf-based ESB container pre-configured to support Talend Mediation route (Apache Camel routing) and Talend Web Services Jobs (Apache CXF-based services - both REST and SOAP-based).
- Talend Administration Center (TAC)- Talend administration interfaces to manage user/project settings, authorization/authentication, code deployment, monitoring, server registrations, etc. It also provides MetaServlet REST service APIs capacities to allow interactions with TAC without human interventions for most activities such as registering a new Runtime/ETL job servers, deploy job/service code, etc.
In this architecture, we could build a Talend Data Services platform hosted on Amazon EC2 instances, and setup AWS Auto Scaling capacities to increase/decrease Talend ESB Runtime servers based on demand.
Let’s run through this architecture process step-by-step:
1. Talend TAC has one or multiple pre-configured Runtime servers (from the same Auto Scaling group) and web services applications (Talend Jobs or Routes) have been deployed from Nexus to each Runtime container. Existing Runtime(s) have AWS Elastic Load Balancer in front to distribute incoming traffic.
2. When one or more runtime(s) EC2 instances have passed resource capacity limitation pre-defined in AWS CloudWatch event (E.g one of runtime EC2 instance’s CPU is running above 80% during last 30 minutes - with increased Web Service requests during sales event).
3. AWS CloudWatch notifies Talend Auto Scaling Group (AWS Launch Configuration) to spin up one or more runtime server EC2 instances and use the same Load Balancer in front, so that increased incoming traffic could be redirected to new Runtime instances.
4. Once new Runtime Instance(s) are initialized by Auto Scaling group, it will use Talend MetaServlet as Linux service scripts to:
- Register itself in Talend TAC as a new Runtime server.
- Create a new task in TAC/ESB Conductor and deploy the same service from Nexus repository to new Runtime container(s)
5. Once incoming traffic is back to “normal” – E.g. All runtime server EC2 instance’s CPU is running under 20% for last hour, AWS CloudWatch event will notify Auto Scaling group to stop/determine Runtime server instances, some Talend MetaServlet scripts can be used here:
- Undeploy service from Runtime server.
- Remove the service from TAC/ESB conductor,
- Deregister Talend Runtime in TAC/Servers configuration page before shutdown.
For client requests traffic, all of the above steps will be keeping transparent during the entire process, since the AWS Load Balancer will be providing a unique entry service endpoint.
As you can see no human interventions will be needed during this Auto-Scaling process. Your Talend ESB infrastructure could be scaled in/out in an elegant manner. This enables online retailers who have varying demand on their infrastructure to rest easy during the holiday months ahead!
If you are interested in how this works into more detail, you can find this complete technical article on help.talend.com: https://help.talend.com/reader/er2QTM0FZy4M4dhswTYQsw/tAFwuwYVCnVJZtacJHPWoQ
Find us at AWS re:Invent 2017!
Want to see more Talend and Amazon Web Services solutions? Swing by booth #607 to connect with peers and cloud experts, collaborate and learn how AWS and Talend can improve productivity and security.