How to Architect, Engineer and Manage Performance (Part 1)
This is the first of a series of blogs on how to architect, engineer and manage performance. In it, I’d like to attempt to demystify performance by defining it clearly as well as describing methods and techniques to achieve performance requirements. I'll also cover how to make sure the requirements are potentially achievable.
Why is Performance Important?
To start, let's look at a few real-world scenarios to illustrate why performance is critical in today's business:
- It is the January sales and you are in the queue at your favorite clothing stores. There are ten people in front of you, all with 10-20 items to scan and scanning is slow. Are you going to wait or give up? Are you happy? What would a 30% increase in scan rates bring to the business? Would it mean fewer tills and staff at quiet times? Money to save, more money to make and better customer satisfaction.
- Let's say you work at a large bank and you are struggling to meet the payment deadline for inter-bank transfers on peak days. If you fail to meet these, you get fined 1% of the value of all transfers waiting. You need to make the process faster, increase capacity, improve parallelism. It could lose you money, worse damage your reputation with other banks and customers.
- You’ve needed a mortgage. XYZ Bank offers the best interest rate and they can give you a decision within a month. ABC Bank cost 0.1% more, but guarantee a decision in 72 hours. The vendors of the house you want to buy need a decision within the week.
- Traders in New York and London can make millions by simply being 1ms faster than others. This is one of the rare performance goals, where there is no limit to how fast, except the cost versus the return.
Why is performance important? Because it means happy customers, cost savings, avoiding lost sales opportunities, differentiating your services, protecting your reputation and much more.
The Outline for Better Performance Processes
Let’s start with performance testing. While this is part of the performance process on it is own, it is usually a case of “too little, too late” and the costs of late change are often severe.
So, let me outline a better process for achieving performance and then I’ll talk a little about each part now, and in more detail in the next few blogs:
- Someone should be ultimately responsible for performance – the buck stops with someone, not some team.
- This performance owner leads the process – they help others achieve the goals.
- Performance goals need to be measurable and realistic, and potentially achievable - they are the subject of discussions and agreement between parties.
- Goals will be broken down amongst the team delivering the project – performance is a team game.
- Performance will be achieved in stages using tools and techniques – there will no miracles, magic or silver bullets.
- Performance must be monitored, measured and managed – how do we deal with the problems which will occur.
Responsibility for Performance
Earlier, I stated that one person should be solely responsible for performance. Let me expand upon that. When you have more than one person in charge, you can no longer be sure how the responsibility is owned. Even with two people, there will be things which you haven’t considered and do not fit neatly in one or other roles. If possible, do not combine roles like Application and Performance Architect (Leader) as this may lead to poor compromises. In my opinion, it is far better for each person to fight for one thing and have the PM or Chief Architect to judge the arguments rather than one person trying to weigh the benefits on their own. Clearly, in small projects, this is not possible, and care is necessary to ensure anyone carrying multiple roles can balance the two or bring out the multiple roles without bias. It is very easy to favour the role we know best or enjoy most.
I probably won’t return to the topic until, towards the end of the series, it will be easier to understand after looking at the other parts in more detail.
The Performance Architect – Performance Expert, Mentor, Leader and Manager
In an ideal scenario, a Performance Architect’s role is to guide others through the performance improvement process, not to do it all themselves. No one is an enough of an expert in all aspects of performance to do this. Performance Architects should:
- Manage and orchestrate the process on behalf of the project.
- Lead by taking responsibility for the division of a requirement(s) and alignment of the performance requirements across the project.
- Provide expertise in performance, giving more specialised roles ways to solve the challenges of performance – estimation, measurement, tuning, etc.
This needs a bit more explanation in a future blog in this series, but I will cover some of the other parts first.
Setting Measurable Goals – Clear Goals that Reflect Reality
Setting measurable performance goals requires more work than you first think. Consider this example, “95% of transactions must complete in < 5 secs”. On the face of it this seems ok, but consider:
- Are all transactions equal – logon, account update, new account opening?
- What do the five seconds mean – elapsed time, processing time, what about the thinking time of the user?
- What if the transaction fails?
- Data variations – all customers named “Peabody” versus “Mohammed” or “Smith”.
That one requirement will need to be expanded and broken down into much more detail. The whole process of getting the performance requirements “right” is a major task and the requirements will continue to be refined during the project as requirements change.
It is worth stating that trying to tune something to go as fast as possible is not the aim of performance. It isn’t a measurable goal, you don’t know when you’ve achieved the goal, and it would be very expensive.
This area can be involved, it takes time, effort and practice, and this is the basis for the rest of the work, so a topic for more discussion in the next blog.
Breaking the Goals Down – Dividing the Cake
If we have five seconds to perform the login transaction, we need to divide that time between the components and systems that will deliver the transaction. The Performance Architect will do this with the teams involved, but I've found that they’ll get it wrong at the start. They don’t have all the facts, it doesn’t matter if it is right at the finish. You are, probably struggling to see where to start, don’t worry about that for a bit, the next section will help.
I’ll look at this in a future blog, but it is probably best discussed after looking at estimation techniques.
Achieving Performance – The Tools and Techniques
When most people drop a ball, they know it falls downwards due to gravity. But not many of us are physicists and four-year-old children haven’t studied any physics, but they don’t seem to struggle to use the experience (knowledge).
How long will an ice cube take to melt? An immediate reaction goes something like this, “I don’t know it depends on the temperature of the water/air around the ice cube”. So make some assumptions, provide an estimate, then start asking questions and doing research to confirm the assumptions or produce a new estimate.
How long will the process X take? How long did it take last time? What is different this time? What is similar? What does the internet suggest? Could we do an experiment? Could we build a model? Has it been done before?
Start with an estimate (guess, educated guess, the rule of thumb – use the best method available or methods) and then as the project progresses we can use other techniques to model, measure and refine.
You might be saying, "But I still don’t know"! That’s correct, and you probably never will at the outset.
Statistically, it is almost certain you won’t die by being struck by lightning (Google estimates 24,000 or 6,000 people per year on Earth pass away this way – no one knows the real figure – think about that, we don’t know the real answer). Most (or all, I hope) assume we won’t be sick tomorrow, but some people will. Nothing is certain, you don’t the answer to nearly everything in the future with accuracy, but it doesn’t stop you making reasonable assumptions and dealing with the surprises, both good and bad.
This is a huge topic and I need to spend some time on this in the series to build your confidence in your skills by showing you just some of the options you have now, you can learn and develop.
“Data Science” and “Statistics” are whole areas of academic study interested in prediction, so this is more than a topic. There are probably more ways to produce estimates than there are to solve the IT problem you are estimating.
Keeping on track – Monitoring, measuring and managing
Donald Rumsfeld made the point there are things we know, we know we don’t know and things that surprise us (unknown unknowns). Actually, it is worse, people often think they know and are wrong, or assume they don’t know and then realise their estimate was better than the one the project used.
Risk management is how we deal with the whole performance process. Risk management, just like the Project Manager uses will help us manage the process. As we progress through the project we will build up our knowledge and confidence in the estimates and reduce the risk and use the information to focus our effort where the greatest risk is.
A Project Manager measures the likelihood of the risk occurring and the impact. For performance, we measure the chance of us achieving the performance goal and how confident we are of the current estimate.
This will be easier to understand as we look at other parts in more depth, we’ll revisit this in a future blog.
In future blogs in the series I will cover:
- Setting goals – Refining the performance requirements.
- Tools and techniques - Where to start – Estimation, Research,
- Monitoring, measuring and managing – risk and confidence.
- Breaking the goals down across the team.
- Tools and techniques – Next Steps – Volumetrics, Model and Statistics
- Tools and techniques – Later Stages – Some testing (and monitoring) options.
- Responsibility and Leadership
Chris first became interested in computers aged 9. His formal IT education was from ages 13 to 21, informally it has never stopped. He joined the British Computer Society (BCS) at 17 and is a Chartered Engineer, Chartered IT Professional and Fellow of the BCS. He is proud and honoured to have held several senior positions in the BCS including Trustee, Chair of Council and Chair of Membership Committee, and remains committed to IT professionalism and the development of IT professionals.
He has worked for two world leading companies before joining his third, Talend as a Customer Success Architect. He has over 30 years of professional working experience, with data and information systems. He is passionate about customer success, understanding the requirements and the engineering of IT solutions.
The Talend Customer Success Architecture (CSA) team is a growing worldwide team of 20+ highly experienced technical information professionals. It is part of Talend’s Customer Success organisation dedicated to the mission of making all Talend’s customers successful and encompasses Customer Success Management, Professional Services, Education and Enablement, Support and the CSA teams.
Built for cloud data integration, Talend Cloud allows you to liberate your data from everything holding it back. Data integration is a critical step in digital transformation. Talend makes it easy.