I've seen you posted this in the Intel Software forum.Can you comment on their reply?
here is the answer from my Buddy Mark.
Hi Professor Fog,
Thank you for your detailed and insightful comments.
Among the important boundary conditions to add to your list is that some of today’s drivers use legacy SSE. These are being reached via interrupt and the existing drivers can’t save the upper part of the live YMM registers. Theoretically a new OS could take care of this for the ISR, but we didn’t want to penalize users of the legacy architecture and could not mandate a major OS re-write.
Over a decade ago when we defined legacy SSE instructions we had no vision of the way we were going to extending the vector length. The new Intel AVX 128 instructions are defined as zeroing the upper part of the register (bit 128 to infinity), and we have a new state management (XSAVE/RSTOR) that can manage state in a forward compatible manner. So if and when we will extend the vector size further the magnitude of this issue should be much smaller (driver writers, you have been warned J).
So in brief, our solution was:
1) Legacy and new code can intermix, the first transition from 256b usage to legacy 128b will have a transition penalty (in Sandy bridge implementation it will cost several tens of cycles depending on what’s in the pipe, in extreme cases it can be much longer). The HW will take care to save the upper parts and later restore (with a similar performance penalty) when returning to 256b code.
2) The compiler will map C or 128b instrisics to either the legacy 128b or the new zeroing-upper 128b forms based on a switch. A programmer could also write in assembly, and they need to keep the above in mind. Inline assembly can also be encoded to the new zeroing-upper 128b forms based on a switch. The expected result of this is that mixing legacy SSE and new 256b instructions would mostly happen at or above the function level.
3) The vZeroUpper instruction is cheap and the recommended ABI is to save the live upper part of the YMM register and issue vZeroUpper prior to calling to an external function. BTW, it will be a good programming practice to "deallocate" the upper part (using vZeroUpper) or the whole registers (using vZeroAll) when finishing part of the program that uses this registers. (optimization recommendation: This will help free up resources in the OOO machine and reduce time spent in task switches). The result of this will be that only the small overhead of vZeroUpper will be the cost of this scheme.
4) We will add performance events that count the two transitions from #1 above to help with debugging any SSE->256b transitions.
To help understand the decision process a bit better, allow me to provide some historical context. Over the past several years we evaluated all the options you have above, but particularly options 1 and 5 (and of course the current option...). Options involving partial dependencies or partial state management were difficult to make fast and/or power efficient. It also did not seem right that the new state should penalize all users of the legacy architecture".
Your "option 1" is in a couple of ways a desirable architecture. It avoids any concern of legacy code perturbing your new state. On the other hand, there are advantages to being able to interoperate cleanly with the current XMM register state: it is entrenched in the ABI's (as you note, XMM is used for argument passing), and we wanted to bring the encoding advantages of the NDS and new operations to scalar and short vector (scalar dominates most FP code today).. The big problem is that doubling the name space for what is, in effect, the same functionality, is inefficient both inside the processor as well as in the state save image.
Your "option 5" (the legacy instructions to zero the new upper state) is really attractive. The real issue here is that popular operating systems do not preserve all state on switching to an interrupt service routine (ISR). A surprising number of Ring-0 components active during ISR use XMM's - if only to move or initialize data. There are only two solutions to this problem: the OS must save all the new state or the drivers must be rewritten. There is no way we could compel the industry to rewrite/fix all of their existing drivers (for example to use XSAVE) and no way to guarantee they would have done so successfully. Consider for example the pain the industry is still going through on the transition from 32 to 64-bit operating systems! The feedback we have from OS vendors also precluded adding overhead to the ISR servicing to add the state management overhead on every interrupt. We didn't want to inflict either of these costs on portions of the industry that don't even typically use wide vectors. Architecturally, therefore, we had to prevent legacy drivers from having side effects on the new (upper) state. This means they had to merge. New drivers, aware of how to use XSAVE to manage AVX state in a forward compatible manner so as not to break apps. So for the AVX-prefixed (short vector or scalar) instructions we allow a zeroing behavior.
What we decided to do was to optimize for the common scenario of uniform blocks of 128-bit (or all scalar) code separated from uniform blocks of 256-bit code. We maintain an internal record of when we transition between states where the upper bits contain something nonzero - to a point where the state is guaranteed to be zero. We give you a fast (1* cycle throughput) way to the second state –VZEROUPPER (though VZEROALL, XRSTOR and reboot also work). Once you're in that state of Zeroed-upperness, you can execute 128-bit code (or scalar) - VEX prefixed or not - and you pay no transition penalty. You can also transition back to executing 256-bit instructions – also with no penalty. You can transition freely between any VEXed instruction of any width and pay no penalty. The downside is that if you try to move from 256-bit instructions to legacy 128-bit instructions without that VZEROUPPER, you're going to pay. The way we chose to make you pay is to optimize for the common use of long blocks of SSE code – you pay once during the transition to legacy 128-bit code instead of on every instruction. We do it by copying the upper 128-bits of all 16 registers to a special scratchpad and this copying takes time (something like 50 cycles - still TBD). Then the legacy SSE code can operate as long as it wants with no penalty. When you transition back to a VEX-prefixed instruction, you have to pay the penalty again to restore state from that scratchpad.
The solution for this problem is for software to use VZEROUPPER prior to leaving your basic block of 256-bit code. Use it prior to calling any (ABI compliant) function, and prior to any blind return jump.
We've also expanded the ABI in a backwards compatible way - functions can declare and pass 256-bit arguments if so declared, but >>new state is caller save<<. Making ymm6-15 as callee save on Windows-64 would require changes in several runtime libraries related to exception handling and stack unwinding, making the changes incompatible with the current libraries. Having all new state as caller-save is slightly inefficient but the gain due to full backward compatibility outweighs the loss of performance. For full details see the ABI extensions for each OS see the Spring Intel Developer Forum AVX presentation (
https://intel.wingateweb.com/SHchina/pu ... 0r_eng.pdf). For code that doesn't vectorize, mixture of scalar (or 128-bit vector) AVX and legacy 128-bit instructions is completely painless so long as the upper state is zero previously - I would rely on compliant callers to issue VZEROUPPER or if paranoid you could zero yourself prior to the executing the function body).
So back to your scenario: A function using 256-bit AVX instructions cannot assume the callee will not modify the high bits in the YMM’s. Bits 128-255 of the YMM’s are caller save. The caller must also issue a VZEROUPPER in case that callee hadn't ported itself to use AVX. There's always a tradeoff in caller/callee save and the benefit here is that the callee that wasn’t to use 256-INT doesn't have to worry about preserving this new state.
> Now, I wonder if we really need the complexity of having two versions of all 128-bit instructions
The non-destructive source and the compact encoding are >major< performance features that apply to scalar and short vector forms. Indeed, I expect the performance upside of these on general purpose, compiled code (that doesn't always vectorize so well) to match the upside of the wider vector width. New operations like broadcast and true masked loads and stores and the 4th operand (for the new 3-source instructions) are only available under the VEX prefix. You can freely intermix 128-bit VEX instructions with legacy 128-bit instructions. And you can intermix blocks of 256-bit instructions with legacy 128-bit instructions so long as you follow two rules: use VZEROUPPER prior to leaving a basic block containing 256-bit and adhere to caller-save ABI semantics on new state.
> There will be a penalty for mixing the legacy XMM instructions using partial register writes with any of the full YMM instructions. Is there a penalty only when reading a full register after writing to the partial register, or are there other situations where mixing instructions with and without VEX causes delays?
Without VZEROUPPER, there will be a penalty when moving from 256-bit instructions to legacy 128-bit instructions, and another penalty when moving from 128-bit instructions back to 256-bit instructions. Consider this code:
VADDPS ymm0, ymm1, ymm2
ADDSS xmm3, xmm4
VSUBPS ymm0, ymm0, ymm2
Here we would have a penalty on the ADDSS and a second penalty on the VSUBPS. (In this case it would be best just to use the VEX form of ADDSS).
> Compilers will need a switch for compiling 128-bit XMM instructions with or without VEX prefix. Software developers will have problems avoiding the penalty of mixing code with and without VEX prefixes.
Absolutely correct on the compiler switch (the compiler has a switch to select between AVX and legacy SSE generation – on the Intel compiler it’s QxG). But since the compilers (at least for high level languages) will generate VZEROUPPER and caller-save semantics on new state whenever they succeed in autovectorizing, there won't be penalties. There are still transitions penalties that happen due to asynchronous events (such as ISR's that use XMM's) but these are not common enough to be a significant performance concern.
The tools will not generate VZEROUPPER on intrinsics or assembly - here, we give the developer the full right/control to shoot their performance in the foot. It is a particularly concern of mine that people writing intrinsics be aware they still have to use VZEROUPPER!
> It will be very hard for programmers using vector intrinsics to avoid mixing the different kinds of instructions.
I hope it will not be too hard, but it is a concern. I've ported a number of applications with both an internal version of the Intel(R) compiler - with and without intrinsic and assembly versions of some hotspots - and the transitions haven't shown up as a noticeable contribution. Autovectorizing compilers like the Intel(R) compiler don't have any problems. In the Intel compiler, when you compile with QxG, all your intrinsics (128-bit or 256-bit) get a VEX prefix. The only diligence required on the part of the programmer is to ensure that you have a VZEROUPPER prior to leaving that block of intrinsics. We have a tool used internally (the Intel(R) software development emulator) that identifies any transitions. We also plan to have a performance monitoring counter in the hardware to allow tools like Intel(R) VTune to show transition penalties.
> Do you expect all function libraries using XMM registers to have two versions of every function: a legacy version for backwards compatibility, and a version with VEX prefixes on all XMM instructions for calling from procedures that use YMM registers?
What we wouldn't do is have lots of library versions just to work around the lack of a callee-save ABI. We would have (and do today) support multiple versions of our performance libraries optimized for different processors. But the 80/20 rule applies: most library functions are not critical performance bottlenecks and the legacy (non-VEXed) implementations work there just fine - with no transition penalties to apps that use 256-bit AVX. In the long run of course, there are no advantages to the non-VEX forms and we would like to move the industry in the direction of AVX.
> The solution of having two versions of all XMM instructions looks to me like a shortsighted patch in an otherwise well designed and future-oriented ISA extension
> The problem will appear again in all future extensions of the size of the vector registers. How do you plan to solve the problem next time the register size is increased? Will we have two versions of every YMM instruction when ZMM is introduced, and two versions of every ZMM instruction when.... This would be a waste of the few unused bits that are left in the VEX prefix.
I have a much stronger feeling that we have reached an excellent compromise. There are significant benefits to the non-destructive destination and new instruction forms that apply to scalar and short vector operations - and if you don't want/need to port, we interoperate at very high performance with the legacy instructions. Software and tools developers have very few rules to avoid performance overhead 1) when you use 256-bit forms of instructions, make sure you VZEROUPPER when leaving the block and 2) realize that the ABI remains caller save for new state. And of course - don't intentionally write code like the example above J
I'm glad you raised the future of ZMM. If/when we adopt the next natural extension (to 512 bits or whatever), we would face the same problems with 'Legacy 256-bit' so long as the asynchronous parts of the system are not saving state in a forward-looking way, or the OS does not decide to step in and manage state on ISR's. So this is a call to the industry planning to adopt AVX in their interrupt service routines: use XSAVE. It's there to guarantee you can save both today’s <and> tomorrow's state!
> It is probably too late to change the AVX spec now, although it looks to me like a draft, published prematurely as a response to AMD's SSE5 stunt?
Let's keep having the dialogue, but I believe the software apps will be greatly benefitted by the present architecture (and microarchitecture). As we did with the Nehalem instructions, we believe its best for the software to have lead time to prepare the OS, compilers, tools, and apps early. The AVX spec is not a draft :)
Regards, Mark
