Comments on: Are we missing the point with SSD/Flash?

By: Mike Workman

Mike Workman — Tue, 02 Jun 2009 18:27:36 +0000

Steve,
I think the guys above said it very well, except I think they were too nice
Until we come up with breakthrough architecture that solve ALL the problems, we make incremental improvements. Always have, always will. Faster processors, more cache, higher RPM’s and the like.
Adding SSD to the storage pool just gives us one more weapon in the war on storage *subsystem* performance. It is probably, for now, not as great a weapon as the hype would suggest, but it is certainly a good one, and it will get better over time. Someday, it will be stupid not to have it in your array. In the meantime people will argue against it only if their subsystem design cannot make effective use of it, or if their management system turns it into a huge kludge to manage. We all know who these folks are by the structure of their arguments for or against SSD.
In my first Blog on this topic I made exactly your first point – put the SSD closest to the processors doing the work – Fusion IO for example connected with PCIe busses – far better than accessing a shared network with SSD’s hanging off the back of it. That is all fine if we are willing to give up the shared storage architecture. It some cases, it is the best weapon for performance.
Finally, I am curious as to how all of a sudden people think FAST is so cool, when Pillar did this Tiering in the box for the last last 3+ years? Our QoS does exactly that, automated movement of data between different Tiers of non-volatile storage components. Out management structure makes addition of SSD into the storage pool trivial. Before people beat me up over this – I didn’t say using SSD effectively is trivial, the technology certainly presents it’s challenges.
Now for the Silver Oak: I made mine Shafer One Point Five. If you haven’t tried it, you should. At least you should if you like big wines. If you are partial to things closer to standard Pinot’s then stay away from any Shafer – it will cause you to convulse….if you have a big budget, go for Hillside Select.
Have a great weekend.
Mike

By: the storage anarchist

the storage anarchist — Fri, 29 May 2009 22:03:39 +0000

A very thought provoking conversation – thank you sir.
(shrugging off my storage-centric blinders)…and yes, I have to agree, in the big picture, all we can struggle to accomplish is relocate the bottleneck somewhere else.
And we (by definition) will NEVER succeed in eliminating all the bottlenecks, unless we stop “improving” software, that is.
Heady stuff – I think there’s a bottle of Silver Oak waiting at home that needs my attention.
Thanks for making Friday Fun!

By: the storage anarchist

the storage anarchist — Fri, 29 May 2009 20:44:21 +0000

Pragmatically speaking, I agree with you that centralizing storage resources is more efficient than distributing them out into the servers. This is the foundation of the external storage value proposition.
And I believe that centralizing expensive resources designed to improve I/O performance is also most prudent – we’ve done that for years with large DRAM caches and the compute power to fuel intelligent pre-fetch to increase cache hit rates, for example.
But I take exception with the assertion that adding Flash to the centralized resource pool in an external array necessarily increases management complexity – in fact, I believe that the opposite is true.
I contend that with the appropriate intelligent algorithms and architecture, the external array can leverage Flash to improve performance in much the same way that it leverages DRAM cache today.
Furthermore, extensive installed-base analysis shows that the vast majority of workloads are far more static than people imagine. This is especially true the larger your granularity of analysis: DRAM cache is appropriate for block-level optimization because the “busy” block set will always be larger than DRAM. But when you move up in granularity, you find that the vast majority of systems service 80-90% of all IO cache-miss requests from only 5-10% of the LUNs.
And perhaps surprisingly, the set of “workhorse” LUNs don’t really change all that much over time.
This fact is indeed the foundational reason why tiering across 15K, 10K and 7200rpm SATA drive works for so many customers. And it is ALSO the foundational reason why wide-striping can work – each drive has to support a small fraction of the workhorse LUNs, and the rest of the LUNs don’t see much I/O anyway so their impact on the workhorses is rare.
What most people haven’t considered is that taking the HEAVIEST workhorse LUNs – the ones that service the MOST IOs per day/week/month – and moving just these to Flash has two immediate benefits: 1) these busy LUNs get the benefit of the drastically reduced response times afforded by Flash, and 2) the rest of the workload left on spinning rust ALSO gets better response time BECAUSE THE DISK DRIVES DON’T HAVE TO WORK AS HARD! Less head contention = better performance!
In fact, customers often find that they can wide-stripe the remaining workload on SATA drives, using a combination of Flash (lowest $/IOP) and SATA (lowest $/GB) to reduce their overall purchase cost and environmental footprint – without sacrificing performance OR increasing operational overhead!
And given that the workload distribution of most arrays is typically very stable, you usually won’t have to add any overhead to constantly rebalance or redistribute the data in/out of Flash.
And for those environments where the workloads ARE more dynamic, EMC has announced FAST – Fully Automated Storage Tiering. FAST will dynamically analyze running workloads and adjust the distribution of LUNs and/or sub-LUNs across the available storage tiers.
FAST is probably the “that” you’re looking for, but it’s not necessarily the case that everyone needs FAST to benefit from centralized Flash.
—-Well at least pragmatically you agree with me! A few points:
1. I am NOT suggesting that improving ANY end node (array, server, filer, etc.) element is bad – I like “better” given the alternative. If the problem can be contained to a single system and isolated to a single tier, then go nuts. If you can fix any problem by adding some memory, why wouldn’t you?
2. Your argument is completely valid (albeit only for the storage end of the equation) PRESUMING that all my data is in your stuff – and the magic that your stuff contains makes sure that the right data is in the right tier at all times. If all I had were super-mega Symms and FAST, then yes, I could theoretically solve all of my optimization/automation issues at the STORAGE layer. I also acknowledge that this scenario would be wicked good – for EMC! However, this is not the case with MOST of the real world, who do deal with (gasp!) non-EMC systems, and even those who deal with multiple EMC systems – it’s a fact that there are lots and lots of disparate elements – and that is the real issue. Plus, your argument falls down if the real issue turns out to be external to the actual storage layer. If the problem is at the load balancer, the server, or the network, you could sell me an array made out of pure gold that squirts Silver Oak and it won’t do me any good. Until I get drunk and sell the thing, of course.
3. In one sense the unrealistic answer to this is that we take all of the individual elements we deal with and throw them out and buy a 1973 Mainframe. Surely as an industry we are attempting to do that by virualizing the disparat elements. Pragmatically however, I’m realistic enough to know that we aren’t going to be able to do that, not in short order anyway, and as such will continue to have a zillion elements that comprise our “infrastructure”. Storage is only 1 layer. Storage by itself is useless. We either implode back to a single system or we deal with the distributed nature of the stack as best we can.
4. Finally, in terms of making things harder, I revert to the statements above. FAST is an awesome concept but is predicated upon having the data under it’s control – and that the storage layer is the entire issue. What happens if after I have FAST put my proper data into my properly sized SSD and Caches – and the user experience still sucks? What happens is the storage guy says, “ain’t me man”, and drinks the Silver Oak while the server, application, and network guys sweat out the problem. They ultimately find out that because the VMware experiment that moved Exchange from 25 shittly little servers down to 4 mongo-ass kicking blade servers as well as 11 Sharepoint servers they piled on but forgot to upgrade to SQL 2007, that the workload got all funky and even though CPU is only at 45% now, performance still sucks. So it ain’t the storage, and it ain’t the processor – but it sure is something. Your system is spitting love on demand, but do you think the user cares? Do you think the operations staff is psyched with their Flash investment now or the magic new Nehalems? Nope. They want to shoot someone.
Which I guess is my point here – systemic issues can only be masked by point improvements – but as long as change is the rule of the day, eventually the next weakest link will be exposed. Flash, nor FAST, nor Silver Oak can solve every problem – only the ones it can touch. —–Steve

By: Tony Asaro

Tony Asaro — Fri, 29 May 2009 18:09:15 +0000

Steve – disk drives are the slowest thing in the performance chain which is why solving this problem does become valuable. There already is caching in multiple places – the host systems and the storage controllers – and cache + cache is additive. However, there will be cache misses and that is why technologies like wide stripping make a real impact on overall performance. Additionally, there is a need for faster processors – if Intel didn’t drive this then VMware couldn’t exist. Who knows what unforeseen value improving performance at levels will manifest? I agree that SSD doesn’t necessarily solve all problems and remember it is price/performance that is what ultimately matters and right now SSD has not hit the “no-brainer” curve. There is no panacea for solving this problem when all the parts of the data center are so disaggregated and heterogeneous. That is why having the market drive improvement with each vendor motivated to do so.
—-I agree that we absolutely need to constantly improve at the component level, and apologize if I didn’t state that clearly. However, the concept of wide-striping is a great example of my point – it is only relevant if the data I’m striping is within that system. If the data most important at this point in time sits elsewhere, the fact that array A is wide striped is meaningless to me. The same holds true if that array contains zero disks – and all SSD. Success is still predicated on the data required BEING in that place.
With data growth never abating, it signifies that you will never have all your data in any one container, thus there will always be a growing percentage of that data that is sub-optimzed. I guess what I’m rambling about is that even if EVERY storage array were 100% SSD – it would still not make for 100% optimized IT infrastructure hollistically – it would just remove the storage end of the wire from being the issue and push it elsewhere. Thanks — Steve