Code and Memory Optimisation Tricks
Download as PPTX, PDF0 likes457 views
This document discusses code and memory optimization techniques for software engineers developing AAA game titles. It begins with an introduction to the speaker and provides an overview of hardware architecture including CPU registers, caches, and memory access times. The bulk of the document focuses on optimizing for data caches through techniques like improving data layout, prefetching, and utilizing cache lines efficiently. It also discusses optimizing branches through removing branches, computing both paths, and splitting data to avoid branches. Resources for further reading are provided.
![• Linear data access is the best you can do to
help hardware prefetching
• Processor recognizes pattern and preload data for next iterations
beforehand
Vec4D in[SIZE]; // Offset from origin
float ChebyshevDist[SIZE]; // Chebyshev distance from origin
for (auto i = 0; i < SIZE; ++i)
{
ChebyshevDist[i] = Max(in[i].x, in[i].y, in[i].z, in[i].w);
}
Optimizing for data cache](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-12-320.jpg)
![struct FooBonus
{
float fooBonus;
float otherData[15];
};
// For every character…
// Assume we have array<FooBonus> structs;
float Sum{0.0f};
for (auto i = 0; i < SIZE; ++i)
{
Actor->Total += FooArray[i].fooBonus;
}
Example of poor data layout:
Optimizing for data cache](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-23-320.jpg)
![• 64 byte offset between loads
• Each is on separate cache line
• 60 from 64 bytes are wasted
addss xmm6,dword ptr [rax-40h]
addss xmm6,dword ptr [rax]
addss xmm6,dword ptr [rax+40h]
addss xmm6,dword ptr [rax+80h]
addss xmm6,dword ptr [rax+0C0h]
addss xmm6,dword ptr [rax+100h]
addss xmm6,dword ptr [rax+140h]
addss xmm6,dword ptr [rax+180h]
add rax,200h
cmp rax,rcx
jl main+0A0h
*MSVC loves x8 loop unrolling
Optimizing for data cache](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-24-320.jpg)
![Cold fields
struct FooBonus
{
MiscData* otherData;
float fooBonus;
};
struct MiscData
{
float otherData[15];
};
Optimizing for data cache
+ 4 bytes for memory alignment on 64bit](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-26-320.jpg)
![• 12 byte offset
• Much less bandwidth is wasted
• Can do better?!
addss xmm6,dword ptr [rax-0Ch]
addss xmm6,dword ptr [rax]
addss xmm6,dword ptr [rax+0Ch]
addss xmm6,dword ptr [rax+18h]
addss xmm6,dword ptr [rax+24h]
addss xmm6,dword ptr [rax+30h]
addss xmm6,dword ptr [rax+3Ch]
addss xmm6,dword ptr [rax+48h]
add rax,60h
cmp rax,rcx
jl main+0A0h
Optimizing for data cache](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-27-320.jpg)
![• Maybe no need to make a pointer to the cold fields?
• Make use of Structure of Arrays
• Store and index different arrays
struct FooBonus
{
float fooBonus;
};
struct MiscData
{
float otherData[15];
};
Optimizing for data cache](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-28-320.jpg)
![• 100% bandwidth utilization
• If everything is 64byte aligned
addss xmm6,dword ptr [rax-4]
addss xmm6,dword ptr [rax]
addss xmm6,dword ptr [rax+4]
addss xmm6,dword ptr [rax+8]
addss xmm6,dword ptr [rax+0Ch]
addss xmm6,dword ptr [rax+10h]
addss xmm6,dword ptr [rax+14h]
addss xmm6,dword ptr [rax+18h]
add rax,20h
cmp rax,rcx
jl main+0A0h
Optimizing for data cache](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-29-320.jpg)
![struct CombatActor
{
// Data…
EUnitType Type; //ARCHER or SWORDSMAN
};
struct Squad
{
CombatActor Units[SIZE][SIZE];
};
void UpdateArmy(const Squad& squad)
{
for (auto i = 0; i < SIZE; ++i)
for (auto j = 0; j < SIZE; ++j)
{
const auto & Unit = squad.Units[i][j];
switch (Unit.Type)
{
case EElementType::ARCHER:
// Process archer
break;
case EElementType::SWORDSMAN:
// Process swordsman
break;
default:
// Handle default
break;
}
}
}
• Branching every iteration?
• Bad performance for hard-to-predict branches
Optimizing branches](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-49-320.jpg)
![struct CombatActor
{
// Data…
EUnitType Type; //ARCHER or SWORDSMAN
};
struct Squad
{
CombatActor Archers[A_SIZE];
CombatActor Swordsmen[S_SIZE];
};
void UpdateArchers(const Squad & squad)
{
// Just iterate and process, no branching here
// Update archers
}
• Split! And process separately
• No branching in processing methods
• + Better utilization of I-cache!
void UpdateSwordsmen(const Squad & squad)
{
// Just iterate and process, no branching here
// Update swordsmen
}
Optimizing branches](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-50-320.jpg)
![; function prologue
cmp dword ptr [data], 0
je END
; set of some ALU instructions…
;…
END:
; function epilogue
; function prologue
cmp dword ptr [data], 0
jne COMP
jmp END
COMP:
; set of some ALU instructions…
;…
END:
; function epilogue
• Imagine cmp dword ptr [data], 0 – likely to evaluate to “false”
• Prefer predicted not taken
Predicted not taken Predicted taken
Optimizing branches](https://image.slidesharecdn.com/devgammoptimizationtricksevgenymuralev-161003150120/85/Code-and-Memory-Optimisation-Tricks-52-320.jpg)
Code and Memory Optimisation Tricks
- 1. Code and memory optimization tricks Evgeny Muralev Software Engineer Sperasoft Inc.
- 2. About me • Software engineer at Sperasoft • Worked on code for EA Sports games (FIFA, NFL, Madden); Now Ubisoft AAA title • Indie game developer in my free time
- 3. Our Clients Electronic Arts Riot Games Wargaming BioWare Ubisoft Disney Sony Our Projects Dragon Age: Inquisition FIFA 14 SIMS 4 Mass Effect 2 League of Legends Grand Theft Auto V About us Our Office Locations USA Poland Russia The Facts Founded in 2004 300+ employees Sperasoft on-line sperasoft.com linkedin.com/company/sperasoft twitter.com/sperasoft facebook.com/sperasoft
- 4. Agenda • Brief architecture overview • Optimizing for data cache • Optimizing branches (and I-cache)
- 5. Developing AAA title • Fixed performance requirements • Min 30fps (33.3ms per frame) • Performance is a king • a LOT of work to do in one frame!
- 6. Make code faster?… • Improved hardware • Wait for another generation • Fixed on consoles • Improved algorithm • Very important • Hardware-aware optimization • Optimizing for (limited) range of hardware • Microoptimizations for specific architecture!
- 7. Brief overview CPU REG L1 I L1 D L2 I/D RAM REG ~2 cycles ~20 cycles ~200 cycles
- 8. • Last level cache (LLC) miss cost ~ 200 cycles • Intel Skylake instruction latencies • ADDPS/ADDSS 4 cycles • MULPS/MULSS 4 cycles • DIVSS/DIVPS 11 cycles • SQRTPS/SQRTSS 13 cycles Brief overview
- 10. Intel Skylake case study: Level Capacity/ Associativity Fastest Latency Peak Bandwidth (B/cycle) L1/D 32Kb/8 4 cycles 96 (2x32 Load + 1x32 Store) L1/I 32Kb/8 N/A N/A L2 256Kb/4 12 cycles 64B/cycle L3 (shared) Up to 2Mb per core/Up to 16 44 cycles 32B/cycle http://www.intel.ru/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf Brief overview
- 11. • Out-of-order execution cannot hide big latencies like access to main memory • That’s why processor always tries to prefetch ahead • Both instructions and data Brief overview
- 12. • Linear data access is the best you can do to help hardware prefetching • Processor recognizes pattern and preload data for next iterations beforehand Vec4D in[SIZE]; // Offset from origin float ChebyshevDist[SIZE]; // Chebyshev distance from origin for (auto i = 0; i < SIZE; ++i) { ChebyshevDist[i] = Max(in[i].x, in[i].y, in[i].z, in[i].w); } Optimizing for data cache
- 13. • Access patterns must be trivial • Triggering prefetching after every cache miss will pollute a cache • Prefetching cannot happen across page boundaries • Might trigger invalid page table walk (on TLB miss) Optimizing for data cache
- 14. • What about traversal of pointer-based data structures? • Spoiler: It sucks Optimizing for data cache
- 15. • Prefetching is blocked • next->next is not known • Cache miss every iteration! • Increases chance of TLB misses • * Depending on your memory allocator current current->next->nextcurrent->next struct GameActor { // Data… GameActor* next; }; while (current != nullptr) { // Do some operations on current // actor… current = current->next; } LLC miss! LLC miss! LLC miss! Optimizing for data cache
- 16. Array vs Linked List traversal Linear Data Random access Optimizing for data cache Time N of elements
- 17. • Load from memory: • auto data = *pointerToData; • Special instructions: • use intrinsics: _mm_prefetch(void *p, enum _mmhint h) Configurable! Optimizing for data cache
- 18. • Usually retire after virtual to physical address translation is completed • In case of exception such as page fault software prefetch retired without prefetching any data e.g. Intel guide on prefetch instructions: • Load from memory != prefetch instructions • Prefetch instructions may differ depending on H/W vendor Optimizing for data cache
- 19. • Probably won’t help • Computations don’t overlap memory access time enough • Remember LLC miss ~ 200c vs trivial ALUs ~ 3-4c while (current != nullptr) { Prefetch(current->next) // Trivial ALU computations on current actor current = current->next; } Optimizing for data cache
- 20. while (current != nullptr) { Prefetch(current->next) //HighLatencyComputation… current = current->next; } • May help around high latency • Make sure data is not evicted from cache before use Optimizing for data cache
- 21. • Prefetch far enough to overlap memory access time • Prefetch near enough so it’s not evicted from data cache • Do NOT overprefetch • Prefetching is not free • Polluting cache • Always profile when using software prefetching Optimizing for data cache
- 22. RAM: … … … … … … a … … … … … … … … … … … … … … … … … … … … … … … … … Cache: • Cache operates with blocks called “cache lines” • When accessing “a” whole cache line is loaded • You can expect 64 bytes wide cache line on x64 a … … … … … … … … … … … … … … … Optimizing for data cache
- 23. struct FooBonus { float fooBonus; float otherData[15]; }; // For every character… // Assume we have array<FooBonus> structs; float Sum{0.0f}; for (auto i = 0; i < SIZE; ++i) { Actor->Total += FooArray[i].fooBonus; } Example of poor data layout: Optimizing for data cache
- 24. • 64 byte offset between loads • Each is on separate cache line • 60 from 64 bytes are wasted addss xmm6,dword ptr [rax-40h] addss xmm6,dword ptr [rax] addss xmm6,dword ptr [rax+40h] addss xmm6,dword ptr [rax+80h] addss xmm6,dword ptr [rax+0C0h] addss xmm6,dword ptr [rax+100h] addss xmm6,dword ptr [rax+140h] addss xmm6,dword ptr [rax+180h] add rax,200h cmp rax,rcx jl main+0A0h *MSVC loves x8 loop unrolling Optimizing for data cache
- 25. • Look for patterns how your data is accessed • Split the data based on access patterns • Data used together should be located together • Look for most common case Optimizing for data cache
- 26. Cold fields struct FooBonus { MiscData* otherData; float fooBonus; }; struct MiscData { float otherData[15]; }; Optimizing for data cache + 4 bytes for memory alignment on 64bit
- 27. • 12 byte offset • Much less bandwidth is wasted • Can do better?! addss xmm6,dword ptr [rax-0Ch] addss xmm6,dword ptr [rax] addss xmm6,dword ptr [rax+0Ch] addss xmm6,dword ptr [rax+18h] addss xmm6,dword ptr [rax+24h] addss xmm6,dword ptr [rax+30h] addss xmm6,dword ptr [rax+3Ch] addss xmm6,dword ptr [rax+48h] add rax,60h cmp rax,rcx jl main+0A0h Optimizing for data cache
- 28. • Maybe no need to make a pointer to the cold fields? • Make use of Structure of Arrays • Store and index different arrays struct FooBonus { float fooBonus; }; struct MiscData { float otherData[15]; }; Optimizing for data cache
- 29. • 100% bandwidth utilization • If everything is 64byte aligned addss xmm6,dword ptr [rax-4] addss xmm6,dword ptr [rax] addss xmm6,dword ptr [rax+4] addss xmm6,dword ptr [rax+8] addss xmm6,dword ptr [rax+0Ch] addss xmm6,dword ptr [rax+10h] addss xmm6,dword ptr [rax+14h] addss xmm6,dword ptr [rax+18h] add rax,20h cmp rax,rcx jl main+0A0h Optimizing for data cache
- 30. B/D Utilization Attempt 1 Attempt 2 Attempt 3 Optimizing for data cache Time N of elements
- 31. • Poor data utilization: • Wasted bandwidth • Increasing probability of TLB misses • More cache misses due to crossing page boundary Optimizing for data cache
- 32. • Recognize data access patterns: • Just analyze the data and how it’s used • Include logging to getters/setters • Collect any other useful data (time/counters) float GameCharacter::GetStamina() const { // Active only in debug build CollectData(“GameCharacter::Stamina”); return Stamina; } Optimizing for data cache
- 33. • What to consider: • What data is accessed together • How often data is accessed? • From where it’s accessed? Optimizing for data cache
- 34. • Instruction fetch • Decoding • Execution • Memory Access • Retirement *of course it is more complex on real hardware Optimizing branches Instruction lifetime:
- 35. IF ID EX MEM WB I1 Optimizing branches
- 36. IF ID EX MEM WB I1 I2 I1 Optimizing branches
- 37. IF ID EX MEM WB I1 I2 I1 I3 I2 I1 Optimizing branches
- 38. IF ID EX MEM WB I1 I2 I1 I3 I2 I1 I4 I3 I2 I1 Optimizing branches
- 39. IF ID EX MEM WB I1 I2 I1 I3 I2 I1 I4 I3 I2 I1 I5 I4 I3 I2 I1 Optimizing branches
- 40. • What instructions to fetch after Inst A? • Condition hasn’t been evaluated yet • Processor speculatively chooses one of the paths • Wrong guess is called branch misprediction // Instruction A if (Condition == true) { // Instruction B // Instruction C } else { // Instruction D // Instruction E } Optimizing branches
- 41. IF ID EX MEM WB A B A C B A A D A • Pipeline Flush • A lot of wasted cycles Mispredicted branch! Optimizing branches
- 42. • Try to remove branches at all • Especially hard to predict branches • Reduces chance of branch misprediction • Doesn’t take resources of Branch Target Buffer Optimizing branches
- 43. Know bit tricks! Example: Negate number based on flag value int In; int Out; bool bDontNegate; r = (bDontNegate ^ (bDontNegate– 1)) * v; int In; int Out; bool bDontNegate; Out = In; if (bDontNegate == false) { out *= -1; } Branchy version: Branchless version: https://graphics.stanford.edu/~seander/bithacks.html Optimizing branches
- 44. • Compute both branches Example: X = (A < B) ? CONST1 : CONST2 Optimizing branches
- 45. • Conditional instructions (setCC and cmovCC) cmp a, b ;Condition jbe L30 ;Conditional branch mov ebx const1 ;ebx holds X jmp L31 ;Unconditional branch L30: mov ebx const2 L31: X = (A < B) ? CONST1 : CONST2 xor ebx, ebx ;Clear ebx (X in the C code) cmp A, B setge bl ;When ebx = 0 or 1 ;OR the complement condition sub ebx, 1 ;ebx=11..11 or 00..00 and ebx, const3 ;const3 = const1-const2 add ebx, const2 ;ebx=const1 or const2 Branchy version: Branchless version: http://www.intel.ru/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf Optimizing branches
- 46. • SIMD mask + blending example X = (A < B) ? CONST1 : CONST2 // create selector mask = __mm_cmplt_ps(a, b); // blend values res = __mm_blendv_ps(const2, const1, mask); mask = 0xffffffff if (a < b); 0 otherwise blend values using mask Optimizing branches
- 47. • Do it only for hard to predict branches • Obviously have to compute both results • Introduces data-dependency blocking out-of- order execution • Profile! Compute both summary: Optimizing branches
- 48. • Blue nodes - archers • Red nodes - swordsmen Optimizing branches Example: Need to updatea squad
- 49. struct CombatActor { // Data… EUnitType Type; //ARCHER or SWORDSMAN }; struct Squad { CombatActor Units[SIZE][SIZE]; }; void UpdateArmy(const Squad& squad) { for (auto i = 0; i < SIZE; ++i) for (auto j = 0; j < SIZE; ++j) { const auto & Unit = squad.Units[i][j]; switch (Unit.Type) { case EElementType::ARCHER: // Process archer break; case EElementType::SWORDSMAN: // Process swordsman break; default: // Handle default break; } } } • Branching every iteration? • Bad performance for hard-to-predict branches Optimizing branches
- 50. struct CombatActor { // Data… EUnitType Type; //ARCHER or SWORDSMAN }; struct Squad { CombatActor Archers[A_SIZE]; CombatActor Swordsmen[S_SIZE]; }; void UpdateArchers(const Squad & squad) { // Just iterate and process, no branching here // Update archers } • Split! And process separately • No branching in processing methods • + Better utilization of I-cache! void UpdateSwordsmen(const Squad & squad) { // Just iterate and process, no branching here // Update swordsmen } Optimizing branches
- 51. • For very predictable branches: • Generally prefer predicted not taken conditional branches • Depending on architecture predicted taken branch may take little more latency Optimizing branches
- 52. ; function prologue cmp dword ptr [data], 0 je END ; set of some ALU instructions… ;… END: ; function epilogue ; function prologue cmp dword ptr [data], 0 jne COMP jmp END COMP: ; set of some ALU instructions… ;… END: ; function epilogue • Imagine cmp dword ptr [data], 0 – likely to evaluate to “false” • Prefer predicted not taken Predicted not taken Predicted taken Optimizing branches
- 53. • Study branch predictor on target architecture • Consider whether you really need a branch • Compute both results • Bit/Math hacks • Study the data and split it • Based on access patterns • Based on performed computation Optimizing branches
- 54. Conclusion • Know your hardware • Architecture matters! • Design code around data, not abstractions • Hardware is a real thing
- 55. Resources • http://www.agner.org/optimize/microarchitecture.pdf • https://people.freebsd.org/~lstewart/articles/cpumemory.pdf • http://support.amd.com/TechDocs/47414_15h_sw_opt_guide.pdf • http://www.intel.ru/content/dam/www/public/us/en/documents/manual s/64-ia-32-architectures-optimization-manual.pdf • https://graphics.stanford.edu/~seander/bithacks.html
- 56. Questions? • E-mail: [email protected] • Twitter: @EvgenyGD • Web: evgenymuralev.com
- 57. • www.sperasoft.com • Follow us on Twitter, LinkedIn and Facebook! Sperasoft
Editor's Notes
- #12: OOO hides small cache latencies The drawbacks, as just described, are that the access patterns must be trivial and that prefetching cannot happen across page boundaries