Gorillas, be aware!

 During the last 2 weeks I attended the Guerrilla Capacity and Performance online class.
It was somehow different from my normal technical activities as the content was totally agnostic of any specific technology and solely focussed on proper thinking and using of methodologies.
We learned sufficient about queueing theory, Littles Law, Amdahl and USL, PDQ and Statistical Forecasting like Multivariate Regression.
 Beside these technical skills the most important aspect was the tactical one: As modern timeframes and project plans does not allow a dedicated, full blown capacity planning step, the required approach is more guerilla like: Use oportunities, don't waste (your) resources, make achievements visible communicate properly.
 I have to thank Dr. Neil J. Gunther for giving this class. Beside the raw facts and figures he shared a great set of wisdom and anectotes.
 It's now on me to practice - the only way for becoming a practicioner. - Let's hope for many interesting projects ahead where I can apply my new skills!

shared resources

Some days ago I had one of these calls from an application colleague:

The database was slow yesterday, can you please check what's the problem?

Of course, I had some short discussion if he really means the DB or should it be called the application is slow. Also some other questions needed to be asked first, e.g. if it's a response time or throughput issue, when it was "good" last time, what "bad" and "good" means in numbers (seconds/ experience, requests / second), if it's affecting all the experiences on this DB or only a subset. Can it be reproduced or at least does it occur on a known pattern. Also the DB name and users affected I had to ask.
Some time I should create a comprehensive checklist for these questions as I'm missing some of them regularly. But that not today's topic.

As the answers are very vague (at least I got a DB & username with some rough timestamp when it might has started) I checked the DB. I'm it would be more precise to focus on user experience. But I know the developers quite well; it would cost a lot of effort & management pressure to convince them to implement proper instrumentation.
So I was going for low hanging fruits first. ASH/AWR showed this picture:
There is time of more activities in the Database, and it's quite visible where it started and ended. With some more analysis it was obvious the number of calls didn't change dramatically. Getting data from the application (yes, there was at least an activity log) showed in fact the number of requests followed a daily pattern, but this didn't look like the picture above. The pattern of demand was similar to other days; but on any other day AWR showed no such picture, only a "breathe" similar to the applications demand.
By this conversation we implicit generated a (still very blurry) "good" pattern to compare the "bad" one.

---

These bits of information were not perfect, but at least something I could use for further investigations.
AWR shows much higher IO times than normal. I crosschecked the most active SQL_IDs from a "good" pattern against the "bad" one and could confirm they are IO-bound in "good" as well, so they might be affected by this unusual IO.
On the DB-cluster TFA is installed, and so is oswatcher. (Again, aggregates only. But still much better information than nothing). A small awk script analysed the data there (I just don't know why oswatcher doesn't has a SQL interface similar to osquery ) and there were huge response time, but no queueing on the Linux hosts.
This would now require digging deeper through the storage layers (FC SAN and several big central storage boxes). Unfortunately I does not have access to them and their performance data is not externalized right now and the SAN admin was to busy with other things, like compiling reports for senior managers 🤦.
But at least I'm somehow lucky: At this company a nice schema is implemented which gives ASM-disks much more information than only /dev/mapper/anything. As the name of the storage box is part of the ASM-disk path, by some knowledge about the possible servers involved (e.g. only servers in this datacenter, and with some other attributes could be involved at all) and a centralized repository I got a list of all DB-servers and DBs potentially involved on that layer.
With this list and another part of our repository (in that case periodic v$sysstat snapshots [aggregates of aggregates - guess the accuracy there]) a list of other DBs with high IO numbers during that time could be compiled.
Checking these databases there was a top producer of IO:

Some additional analysis showed this load on this particular DB created sufficient IO to saturate one subsystem in the storage box. By doing so all other system which use the same subsystem also suffered. To crosscheck that, the Storage admin (who completed the report in the meantime) provided other DBs utilizing that subsystem - and they showed similar patterns at exactly that time.

Even with all the data collected, this accumulation of coincidents is still no prove what was going on. But it was sufficient likely for the management to think about remediation activities.

---

Here the question might show up why only the admins of this one application showed up and complained about their system? During the investigation it was visible that other DBs showed similar patterns and so other applications were likely to suffer also.
I invested some time to  get answers to these questions. This is not related to the initial goal of the activities, only to satisfy my curiosity and improve my knowledge about the whole landscape.
It showed these other applications also suffered, but the impact was much smaller for 2 reasons:

1. The portion of DB IO time to their user experience was relatively small. Even these high response time didn't cause a lot of harm. 
2. The requirements of the most suffering application were more strict. It can be explained by the time a person accepts to wait after a click: If this click is on a webpage, most readers of this blog might understand what's going on behind the scene and some tenth of a second up to some seconds are acceptable for an answer. But if you flip a switch to turn on the light, they expect the light to shine immediately (aka. within less time than the human senses can grasp)

---

After all the analysis was done and roughly explained to the manager of the most suffering application, he invited for a meeting.
A manager, some people with different roles in development and ops, a representative for the "culprit application which caused the high load" and I were there. He opened the meeting with a summary (condensed by me to):

We and they share the same storage, they did some heavy query and we suffered. 

Technically this is right, but the way it's provided to those not experienced in IT leads to the managers request:

We must ensure they do not cause such IO load anymore. 

This might seem legit, but it might not lead to the managers real goal: Make his application reasonable fast in as many situations as possible. So I had to kill this branch of the discussion for a moment and explained, why they are not the real problem:
As the whole IT is a shared system on various layers (different hosts share the same storage system, different virtual hosts share the same VM layer, different instances share the same DB-cluster, different sessions share the same instance, ...) it is an unreasonable big effort to define resource limits, quotas and similar for all components involved. Especially in a highly dynamic environment with hundreds of DBs, thousands of servers and zillions of processes.
A majority in this room showed some shock & horror and claimed they were not aware of the high level of complexity in basic infrastructure.

With some effort we can limit the IO consumption of this particular DB which caused the issue this time. But next time (and there will be a next time) we only can be sure this particular DB is not involved.
As that's not sufficient, the next request was:

Let's then monitor the storage box and alarm every time a component is saturated.

A nice idea, isn't it?
But whom to alarm? -

The SAN admin of course. 

What should he do then? He can't reduce the load and can't increase the storage capacity either (at least not short term without budget, vendor involved, ...). 

So let's inform those which are connected to this resource! 

In our example it would have been two dozen applications - most of them didn't suffer or care.

Then only the top 5.

In our case you wouldn't have been alarmed.

…
This also doesn't lead to anything useful.

About at that time I raised a simple question:

Who in this room is responsible for the (suffering part of the) application?

---

<silence>

(the managers face went pale for a second).

After some moments, an external (not employed) developer raised his hand and muttered something like "If no one wants to, I can take that". 

I was slightly bored of this meeting, so to speed up things my next question was:

Is every IO your application doing [during the critical activity] necessary?

The argument in short goes like that: If you are less dependent on IO timing, even bad IOs can't harm that much. I call this

make it robust against impacts out of your control

 Beside the obvious effect to improve the application, a nice side effect is a less busy total environment. This again pays back as better service for the whole company.

The meeting stopped close to that point. The new found architect & developers promised to see if they can make their application more robust, the manager wants to raise awareness in all IT departments what a shared environment is and how it can affect business processes within the company.
I have a mixed feeling: the analysis seems valid, but it was full of educated guess & shortcuts - this doesn't feel right as it contains "experience" and "luck"; there is a method which describes a more structured way, I just felt unable to follow these steps . I hope I could transfer the initial reaction (finger-pointing to another team) to something more valuable for the company: use less resources, make your application robust.
But I doubt my effort was lasting.

The Slow Tires

Once there was a man with a car. On this car he had 4 tires.
As the car is a modern one it has a nice board computer which collects many measurements. One is the rotation per minute of the tires.
One day it took more time for the man to get home than usually. So he decided to check his car's computer for any values which could lead to that delay.
After some minutes searching he made an alarming observation: This days average rotations per minute of the tires were much lower than the values of the day before. So he did a quick check if he found some errors in the computers log, but as everything seemed ok he was puzzled.
The shop where he bought the tires was still open and so he went there and ask a technician to check if the tires were ok, as they were slower today than yesterday. A technician checked correct pressure and even made sure the tire balancing was ok.
But there still was the lower rotation per minute for this day, and no explanation.
Next to the tire retailer there was a garage, which owner was friends with the technician of the tire shop. He agreed to check the car, if there was anything he could find about this days slow tires. But there wasn't anything. No errors, motor control all ok, the shock absorbers were even less utilised this day than the previous day.
During the investigation our man tried to remember if anything has been different to the previous day. He remembered some things: the day before he had lunch, which he skipped on this day to get home early. And even the coke in the cup holder was empty. So in theory the car should even be lighter than the day before. He told the two engineers about his memories, but they didn't care about some decagram changes in weight. At least the mechanic confirmed the average rotation was slower on this day. It matched the slower rotation in the gearbox, something which was not available for a customer in the cars memory. So the measurement was correct.
All three men were puzzled. But the mechanic had a friend in highway administration who had access to the highways measurements. Maybe there wa a technical issue there?
But the highway was in good shape, quite similar as the day before. Nothing was broken, all signs working ok. No accident reported. Asphalt temperature was slightly higher than the day before, but perfect within the tires comfort zone. The mechanic got one more information from his friend: Highway administration was more pleased about this day, as the utilisation was higher when the man went home on this day compared to his yesterdays commute, but still within acceptable boundaries to avoid heavy complains.
Now the men had checked virtually everything which came to their mind: tires, many components of the car, highway. Everything fine, no errors or accidents, totally within acceptable boundaries.
Big data, big confusion.
The tire retailer grabbed his smartphone in frustration to check new gossip in the app of his preferred local radio station. By accident he hit the traffic tab with his calloused fingers and saw a nice histogram of the traffic in highway segments around the city. At the travel time of our man on this day there were much more cars in his highway segment than at the day before. He showed this nice graph to the others and slowly it all made sense:
They all know from own experience, more cars on the highway will lead to slower average speed at some filling level. Traffic is still smooth but the individual just can't do high speed anymore, regardless of legal limitations or police.
Of course at this point the utilization increases, as from the highway administration perspective, there are more tires on the road now. Even when each one rotates slightly slower, the total sum of all of them is higher than a single car at highest speed ever could create.
Our man just could not drive as fast as yesterday as many other cars were in his way.
And the tires could not rotate faster for exactly that reason.
With all this insight our mechanic once again called his friend and asked him if he know a way to improve the situation. The answer was simple:
Build a dedicated highway for you. Never share it with anyone. Not even with your best friend. IT will go perfect most of the time. But the one day you want to do the surprise birthday party, exactly this friend will be on the highway ahead of you and block all your best intended efforts.

---

I'm sorry for all who love the english language - if you want to translate this to proper english, please do so and inform me, I'll remove this content and link to yours!