Changeset 215565 in webkit for trunk/Source/JavaScriptCore/assembler

Timestamp:

Apr 20, 2017, 10:55:44 AM (8 years ago)

Author:

[email protected]

Message:

Optimize SharedArrayBuffer in the DFG+FTL
​https://bugs.webkit.org/show_bug.cgi?id=164108

Reviewed by Saam Barati.

JSTests:

Added a fairly comprehensive test of the intrinsics. This creates a function for each possible
combination of type and operation, and then first uses it nicely and then tries a bunch of
erroneous conditions like OOB.

• stress/SharedArrayBuffer-opt.js: Added.

(string_appeared_here.switch):
(string_appeared_here.str):
(runAtomic):
(shouldFail):
(Symbol):
(string_appeared_here.a.of.arrays.m.of.atomics):

• stress/SharedArrayBuffer.js:

Source/JavaScriptCore:

This adds atomics intrinsics to the DFG and wires them through to the DFG and FTL backends. This
was super easy in the FTL since B3 already has comprehensive atomic intrinsics, which are more
powerful than what we need right now. In the DFG backend, I went with an easy-to-write
implementation that just reduces everything to a weak CAS loop. It's very inefficient with
registers (it needs ~8) but it's the DFG backend, so it's not obvious how much we care.

To make the rare cases easy to handle, I refactored AtomicsObject.cpp so that the operations for
the slow paths can share code with the native functions.

This also fixes register handling in the X86 implementations of CAS, in the case that
expectedAndResult is not %rax. This also fixes the ARM64 implementation of branchWeakCAS.

I adapted the CascadeLock from WTF/benchmarks/ToyLocks.h as a microbenchmark of lock performance.
This benchmark performs 2.5x faster, in both the contended and uncontended case, thanks to this
change. It's still about 3x slower than native. I investigated this only a bit. I suspect that
the story will be different in asm.js code, which will get constant-folding of the typed array
backing store by virtue of how it uses lexically scoped variables as pointers to the heap arrays.
It's worth noting that the native lock I was comparing against, the very nicely-tuned
CascadeLock, is at the very high end of lock throughput under virtually all conditions
(uncontended, microcontended, held for a long time). I also compared to WTF::Lock and others, and
the only ones that performed better in this microbenchmark were spinlocks. I don't recommend
using those. So, when I say this is 3x slower than native, I really mean that it's 3x slower than
the fastest native lock that I have in my arsenal.

Also worth noting is that I experimented with exposing Atomics.yield(), which uses sched_yield,
as a way of testing if adding a yield loop to the JS cascadeLock would help. It does not help. I
did not investigate why.

• assembler/AbstractMacroAssembler.h:

(JSC::AbstractMacroAssembler::JumpList::append):

• assembler/CPU.h:

(JSC::is64Bit):
(JSC::is32Bit):

• b3/B3Common.h:

(JSC::B3::is64Bit): Deleted.
(JSC::B3::is32Bit): Deleted.

• b3/B3LowerToAir.cpp:

(JSC::B3::Air::LowerToAir::appendTrapping):
(JSC::B3::Air::LowerToAir::appendCAS):
(JSC::B3::Air::LowerToAir::appendGeneralAtomic):

• dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

• dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::handleIntrinsicCall):

• dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

• dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

• dfg/DFGNode.h:

(JSC::DFG::Node::hasHeapPrediction):
(JSC::DFG::Node::hasArrayMode):

• dfg/DFGNodeType.h:

(JSC::DFG::isAtomicsIntrinsic):
(JSC::DFG::numExtraAtomicsArgs):

• dfg/DFGPredictionPropagationPhase.cpp:
• dfg/DFGSSALoweringPhase.cpp:

(JSC::DFG::SSALoweringPhase::handleNode):

• dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

• dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::loadFromIntTypedArray):
(JSC::DFG::SpeculativeJIT::setIntTypedArrayLoadResult):
(JSC::DFG::SpeculativeJIT::compileGetByValOnIntTypedArray):
(JSC::DFG::SpeculativeJIT::getIntTypedArrayStoreOperand):
(JSC::DFG::SpeculativeJIT::compilePutByValForIntTypedArray):

• dfg/DFGSpeculativeJIT.h:

(JSC::DFG::SpeculativeJIT::callOperation):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• ftl/FTLAbstractHeapRepository.cpp:

(JSC::FTL::AbstractHeapRepository::decorateFencedAccess):
(JSC::FTL::AbstractHeapRepository::computeRangesAndDecorateInstructions):

• ftl/FTLAbstractHeapRepository.h:
• ftl/FTLCapabilities.cpp:

(JSC::FTL::canCompile):

• ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileAtomicsReadModifyWrite):
(JSC::FTL::DFG::LowerDFGToB3::compileAtomicsIsLockFree):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compilePutByVal):
(JSC::FTL::DFG::LowerDFGToB3::pointerIntoTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::loadFromIntTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::storeType):
(JSC::FTL::DFG::LowerDFGToB3::setIntTypedArrayLoadResult):
(JSC::FTL::DFG::LowerDFGToB3::getIntTypedArrayStoreOperand):
(JSC::FTL::DFG::LowerDFGToB3::vmCall):

• ftl/FTLOutput.cpp:

(JSC::FTL::Output::store):
(JSC::FTL::Output::store32As8):
(JSC::FTL::Output::store32As16):
(JSC::FTL::Output::atomicXchgAdd):
(JSC::FTL::Output::atomicXchgAnd):
(JSC::FTL::Output::atomicXchgOr):
(JSC::FTL::Output::atomicXchgSub):
(JSC::FTL::Output::atomicXchgXor):
(JSC::FTL::Output::atomicXchg):
(JSC::FTL::Output::atomicStrongCAS):

• ftl/FTLOutput.h:

(JSC::FTL::Output::store32):
(JSC::FTL::Output::store64):
(JSC::FTL::Output::storePtr):
(JSC::FTL::Output::storeFloat):
(JSC::FTL::Output::storeDouble):

• jit/JITOperations.h:
• runtime/AtomicsObject.cpp:

(JSC::atomicsFuncAdd):
(JSC::atomicsFuncAnd):
(JSC::atomicsFuncCompareExchange):
(JSC::atomicsFuncExchange):
(JSC::atomicsFuncIsLockFree):
(JSC::atomicsFuncLoad):
(JSC::atomicsFuncOr):
(JSC::atomicsFuncStore):
(JSC::atomicsFuncSub):
(JSC::atomicsFuncWait):
(JSC::atomicsFuncWake):
(JSC::atomicsFuncXor):
(JSC::operationAtomicsAdd):
(JSC::operationAtomicsAnd):
(JSC::operationAtomicsCompareExchange):
(JSC::operationAtomicsExchange):
(JSC::operationAtomicsIsLockFree):
(JSC::operationAtomicsLoad):
(JSC::operationAtomicsOr):
(JSC::operationAtomicsStore):
(JSC::operationAtomicsSub):
(JSC::operationAtomicsXor):

• runtime/AtomicsObject.h:

Source/WTF:

Made small changes as part of benchmarking the JS versions of these locks.

• benchmarks/LockSpeedTest.cpp:
• benchmarks/ToyLocks.h:
• wtf/Range.h:

(WTF::Range::dump):

LayoutTests:

Add a test of futex performance.

• workers/sab/cascade_lock-worker.js: Added.

(onmessage):

• workers/sab/cascade_lock.html: Added.
• workers/sab/worker-resources.js:

(cascadeLockSlow):
(cascadeLock):
(cascadeUnlock):

Location:

trunk/Source/JavaScriptCore/assembler

Files:

• AbstractMacroAssembler.h (modified) (4 diffs)
• CPU.h (modified) (2 diffs)
• MacroAssemblerARM64.h (modified) (6 diffs)
• MacroAssemblerX86Common.h (modified) (4 diffs)
• MacroAssemblerX86_64.h (modified) (1 diff)

trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h

@@ -118 +118 @@
     struct BaseIndex;
     
+    static RegisterID withSwappedRegister(RegisterID original, RegisterID left, RegisterID right)
+    {
+        if (original == left)
+            return right;
+        if (original == right)
+            return left;
+        return original;
+    }
+    
     // Address:
     //
@@ -131 +140 @@
         {
             return Address(base, offset + additionalOffset);
+        }
+        
+        Address withSwappedRegister(RegisterID left, RegisterID right)
+        {
+            return Address(AbstractMacroAssembler::withSwappedRegister(base, left, right), offset);
         }
         
@@ -201 +215 @@
         {
             return BaseIndex(base, index, scale, offset + additionalOffset);
+        }
+
+        BaseIndex withSwappedRegister(RegisterID left, RegisterID right)
+        {
+            return BaseIndex(AbstractMacroAssembler::withSwappedRegister(base, left, right), AbstractMacroAssembler::withSwappedRegister(index, left, right), scale, offset);
         }
     };
@@ -703 +722 @@
         void append(Jump jump)
         {
-            m_jumps.append(jump);
+            if (jump.isSet())
+                m_jumps.append(jump);
         }
         

trunk/Source/JavaScriptCore/assembler/CPU.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2008, 2012-2015 Apple Inc. All rights reserved.
+ * Copyright (C) 2008-2017 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -66 +66 @@
 }
 
+inline bool is64Bit()
+{
+#if USE(JSVALUE64)
+    return true;
+#else
+    return false;
+#endif
+}
+
+inline bool is32Bit()
+{
+    return !is64Bit();
+}
+
 inline bool isMIPS()
 {

trunk/Source/JavaScriptCore/assembler/MacroAssemblerARM64.h

@@ -3509 +3509 @@
     }
     
-    void atomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicStrongCAS<8>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicStrongCAS<16>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicStrongCAS<32>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicStrongCAS<64>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicRelaxedStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicRelaxedStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicRelaxedStrongCAS<8>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicRelaxedStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicRelaxedStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicRelaxedStrongCAS<16>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicRelaxedStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicRelaxedStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicRelaxedStrongCAS<32>(cond, expectedAndResult, newValue, address, result);
     }
     
-    void atomicRelaxedStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<typename AddressType>
+    void atomicRelaxedStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         atomicRelaxedStrongCAS<64>(cond, expectedAndResult, newValue, address, result);
     }
     
-    JumpList branchAtomicWeakCAS8(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicWeakCAS8(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicWeakCAS<8>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicWeakCAS16(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicWeakCAS16(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicWeakCAS<16>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicWeakCAS32(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicWeakCAS32(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicWeakCAS<32>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicWeakCAS<64>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicRelaxedWeakCAS8(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS8(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicRelaxedWeakCAS<8>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicRelaxedWeakCAS16(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS16(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicRelaxedWeakCAS<16>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicRelaxedWeakCAS32(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS32(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicRelaxedWeakCAS<32>(cond, expectedAndClobbered, newValue, address);
     }
     
-    JumpList branchAtomicRelaxedWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS64(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         return branchAtomicRelaxedWeakCAS<64>(cond, expectedAndClobbered, newValue, address);
@@ -4173 +4189 @@
     void storeCondRel(RegisterID src, RegisterID dest, RegisterID result)
     {
-        m_assembler.stlxr<datasize>(src, dest, result);
+        m_assembler.stlxr<datasize>(result, src, dest);
     }
     
@@ -4214 +4230 @@
     }
     
-    template<int datasize>
-    void atomicRelaxedStrongCAS(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
+    template<int datasize, typename AddressType>
+    void atomicRelaxedStrongCAS(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, AddressType address, RegisterID result)
     {
         signExtend<datasize>(expectedAndResult, expectedAndResult);
@@ -4238 +4254 @@
     }
     
-    template<int datasize>
-    JumpList branchAtomicWeakCAS(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<int datasize, typename AddressType>
+    JumpList branchAtomicWeakCAS(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         signExtend<datasize>(expectedAndClobbered, expectedAndClobbered);
@@ -4266 +4282 @@
     }
     
-    template<int datasize>
-    JumpList branchAtomicRelaxedWeakCAS(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, Address address)
+    template<int datasize, typename AddressType>
+    JumpList branchAtomicRelaxedWeakCAS(StatusCondition cond, RegisterID expectedAndClobbered, RegisterID newValue, AddressType address)
     {
         signExtend<datasize>(expectedAndClobbered, expectedAndClobbered);
@@ -4304 +4320 @@
     }
 
+    // This uses both the memory and data temp, but only returns the memorty temp. So you can use the
+    // data temp after this finishes.
+    RegisterID extractSimpleAddress(BaseIndex address)
+    {
+        RegisterID result = getCachedMemoryTempRegisterIDAndInvalidate();
+        lshift64(address.index, TrustedImm32(address.scale), result);
+        add64(address.base, result);
+        add64(TrustedImm32(address.offset), result);
+        return result;
+    }
+
     Jump jumpAfterFloatingPointCompare(DoubleCondition cond)
     {

trunk/Source/JavaScriptCore/assembler/MacroAssemblerX86Common.h

@@ -3024 +3024 @@
     void atomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS8(RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS8(RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS16(RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS16(RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS32(RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS32(RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     Jump branchAtomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base); });
     }
 
     Jump branchAtomicStrongCAS8(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgb_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     Jump branchAtomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base); });
     }
 
     Jump branchAtomicStrongCAS16(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgw_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     Jump branchAtomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base); });
     }
 
     Jump branchAtomicStrongCAS32(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgl_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
@@ -4073 +4073 @@
     }
     
-    template<typename Func>
-    void atomicStrongCAS(StatusCondition cond, RegisterID expectedAndResult, RegisterID result, const Func& func)
-    {
+    template<typename AddressType, typename Func>
+    void atomicStrongCAS(StatusCondition cond, RegisterID expectedAndResult, RegisterID result, AddressType& address, const Func& func)
+    {
+        address = address.withSwappedRegister(X86Registers::eax, expectedAndResult);
         swap(expectedAndResult, X86Registers::eax);
         m_assembler.lock();
@@ -4083 +4084 @@
     }
 
-    template<typename Func>
-    void atomicStrongCAS(RegisterID expectedAndResult, const Func& func)
-    {
+    template<typename AddressType, typename Func>
+    void atomicStrongCAS(RegisterID expectedAndResult, AddressType& address, const Func& func)
+    {
+        address = address.withSwappedRegister(X86Registers::eax, expectedAndResult);
         swap(expectedAndResult, X86Registers::eax);
         m_assembler.lock();
@@ -4092 +4094 @@
     }
 
-    template<typename Func>
-    Jump branchAtomicStrongCAS(StatusCondition cond, RegisterID expectedAndResult, const Func& func)
-    {
+    template<typename AddressType, typename Func>
+    Jump branchAtomicStrongCAS(StatusCondition cond, RegisterID expectedAndResult, AddressType& address, const Func& func)
+    {
+        address = address.withSwappedRegister(X86Registers::eax, expectedAndResult);
         swap(expectedAndResult, X86Registers::eax);
         m_assembler.lock();

trunk/Source/JavaScriptCore/assembler/MacroAssemblerX86_64.h

@@ -1436 +1436 @@
     void atomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address, RegisterID result)
     {
-        atomicStrongCAS(cond, expectedAndResult, result, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(cond, expectedAndResult, result, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     void atomicStrongCAS64(RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
     }
 
     void atomicStrongCAS64(RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        atomicStrongCAS(expectedAndResult, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        atomicStrongCAS(expectedAndResult, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }
 
     Jump branchAtomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, Address address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base); });
     }
 
     Jump branchAtomicStrongCAS64(StatusCondition cond, RegisterID expectedAndResult, RegisterID newValue, BaseIndex address)
     {
-        return branchAtomicStrongCAS(cond, expectedAndResult, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
+        return branchAtomicStrongCAS(cond, expectedAndResult, address, [&] { m_assembler.cmpxchgq_rm(newValue, address.offset, address.base, address.index, address.scale); });
     }