Changeset 120172 in webkit for trunk/Source/JavaScriptCore/runtime/SymbolTable.h

Timestamp:

Jun 13, 2012, 1:20:39 AM (13 years ago)

Author:

[email protected]

Message:

DFG should be able to set watchpoints on global variables
​https://bugs.webkit.org/show_bug.cgi?id=88692

Source/JavaScriptCore:

Reviewed by Geoffrey Garen.

This implements global variable constant folding by allowing the optimizing
compiler to set a "watchpoint" on globals that it wishes to constant fold.
If the watchpoint fires, then an OSR exit is forced by overwriting the
machine code that the optimizing compiler generated with a jump.

As such, this patch is adding quite a bit of stuff:

• Jump replacement on those hardware targets supported by the optimizing JIT. It is now possible to patch in a jump instruction over any recorded watchpoint label. The jump must be "local" in the sense that it must be within the range of the largest jump distance supported by a one instruction jump.

• WatchpointSets and Watchpoints. A Watchpoint is a doubly-linked list node that records the location where a jump must be inserted and the destination to which it should jump. Watchpoints can be added to a WatchpointSet. The WatchpointSet can be fired all at once, which plants all jumps. WatchpointSet also remembers if it had ever been invalidated, which allows for monotonicity: we typically don't want to optimize using watchpoints on something for which watchpoints had previously fired. The act of notifying a WatchpointSet has a trivial fast path in case no Watchpoints are registered (one-byte load+branch).

• SpeculativeJIT::speculationWatchpoint(). It's like speculationCheck(), except that you don't have to emit branches. But, you need to know what WatchpointSet to add the resulting Watchpoint to. Not everything that you could write a speculationCheck() for will have a WatchpointSet that would get notified if the condition you were speculating against became invalid.

• SymbolTableEntry now has the ability to refer to a WatchpointSet. It can do so without incurring any space overhead for those entries that don't have WatchpointSets.

• The bytecode generator infers all global function variables to be watchable, and makes all stores perform the WatchpointSet's write check, and marks all loads as being potentially watchable (i.e. you can compile them to a watchpoint and a constant).

Put together, this allows for fully sleazy inlining of calls to globally
declared functions. The inline prologue will no longer contain the load of
the function, or any checks of the function you're calling. I.e. it's
pretty much like the kind of inlining you would see in Java or C++.
Furthermore, the watchpointing functionality is built to be fairly general,
and should allow setting watchpoints on all sorts of interesting things
in the future.

The sleazy inlining means that we will now sometimes inline in code paths
that have never executed. Previously, to inline we would have either had
to have executed the call (to read the call's inline cache) or have
executed the method check (to read the method check's inline cache). Now,
we might inline when the callee is a watched global variable. This
revealed some humorous bugs. First, constant folding disagreed with CFA
over what kinds of operations can clobber (example: code path A is dead
but stores a String into variable X, all other code paths store 0 into
X, and then you do CompareEq(X, 0) - CFA will say that this is a non-
clobbering constant, but constant folding thought it was clobbering
because it saw the String prediction). Second, inlining would crash if
the inline callee had not been compiled. This patch fixes both bugs,
since otherwise run-javascriptcore-tests would report regressions.

• CMakeLists.txt:
• GNUmakefile.list.am:
• JavaScriptCore.vcproj/JavaScriptCore/JavaScriptCore.vcproj:
• JavaScriptCore.xcodeproj/project.pbxproj:
• Target.pri:
• assembler/ARMv7Assembler.h:

(ARMv7Assembler):
(JSC::ARMv7Assembler::ARMv7Assembler):
(JSC::ARMv7Assembler::labelForWatchpoint):
(JSC::ARMv7Assembler::label):
(JSC::ARMv7Assembler::replaceWithJump):
(JSC::ARMv7Assembler::maxJumpReplacementSize):

• assembler/AbstractMacroAssembler.h:

(JSC):
(AbstractMacroAssembler):
(Label):
(JSC::AbstractMacroAssembler::watchpointLabel):

• assembler/AssemblerBuffer.h:
• assembler/MacroAssemblerARM.h:

(JSC::MacroAssemblerARM::replaceWithJump):
(MacroAssemblerARM):
(JSC::MacroAssemblerARM::maxJumpReplacementSize):

• assembler/MacroAssemblerARMv7.h:

(MacroAssemblerARMv7):
(JSC::MacroAssemblerARMv7::replaceWithJump):
(JSC::MacroAssemblerARMv7::maxJumpReplacementSize):
(JSC::MacroAssemblerARMv7::branchTest8):
(JSC::MacroAssemblerARMv7::jump):
(JSC::MacroAssemblerARMv7::makeBranch):

• assembler/MacroAssemblerMIPS.h:

(JSC::MacroAssemblerMIPS::replaceWithJump):
(MacroAssemblerMIPS):
(JSC::MacroAssemblerMIPS::maxJumpReplacementSize):

• assembler/MacroAssemblerSH4.h:

(JSC::MacroAssemblerSH4::replaceWithJump):
(MacroAssemblerSH4):
(JSC::MacroAssemblerSH4::maxJumpReplacementSize):

• assembler/MacroAssemblerX86.h:

(MacroAssemblerX86):
(JSC::MacroAssemblerX86::branchTest8):

• assembler/MacroAssemblerX86Common.h:

(JSC::MacroAssemblerX86Common::replaceWithJump):
(MacroAssemblerX86Common):
(JSC::MacroAssemblerX86Common::maxJumpReplacementSize):

• assembler/MacroAssemblerX86_64.h:

(MacroAssemblerX86_64):
(JSC::MacroAssemblerX86_64::branchTest8):

• assembler/X86Assembler.h:

(JSC::X86Assembler::X86Assembler):
(X86Assembler):
(JSC::X86Assembler::cmpb_im):
(JSC::X86Assembler::testb_im):
(JSC::X86Assembler::labelForWatchpoint):
(JSC::X86Assembler::label):
(JSC::X86Assembler::replaceWithJump):
(JSC::X86Assembler::maxJumpReplacementSize):
(JSC::X86Assembler::X86InstructionFormatter::memoryModRM):

• bytecode/CodeBlock.cpp:

(JSC::CodeBlock::dump):

• bytecode/CodeBlock.h:

(JSC::CodeBlock::appendOSRExit):
(JSC::CodeBlock::appendSpeculationRecovery):
(CodeBlock):
(JSC::CodeBlock::appendWatchpoint):
(JSC::CodeBlock::numberOfWatchpoints):
(JSC::CodeBlock::watchpoint):
(DFGData):

• bytecode/DFGExitProfile.h:

(JSC::DFG::exitKindToString):
(JSC::DFG::exitKindIsCountable):

• bytecode/Instruction.h:

(Instruction):
(JSC::Instruction::Instruction):

• bytecode/Opcode.h:

(JSC):
(JSC::padOpcodeName):

• bytecode/Watchpoint.cpp: Added.

(JSC):
(JSC::Watchpoint::~Watchpoint):
(JSC::Watchpoint::correctLabels):
(JSC::Watchpoint::fire):
(JSC::WatchpointSet::WatchpointSet):
(JSC::WatchpointSet::~WatchpointSet):
(JSC::WatchpointSet::add):
(JSC::WatchpointSet::notifyWriteSlow):
(JSC::WatchpointSet::fireAllWatchpoints):

• bytecode/Watchpoint.h: Added.

(JSC):
(Watchpoint):
(JSC::Watchpoint::Watchpoint):
(JSC::Watchpoint::setDestination):
(WatchpointSet):
(JSC::WatchpointSet::isStillValid):
(JSC::WatchpointSet::hasBeenInvalidated):
(JSC::WatchpointSet::startWatching):
(JSC::WatchpointSet::notifyWrite):
(JSC::WatchpointSet::addressOfIsWatched):

• bytecompiler/BytecodeGenerator.cpp:

(JSC::ResolveResult::checkValidity):
(JSC::BytecodeGenerator::addGlobalVar):
(JSC::BytecodeGenerator::BytecodeGenerator):
(JSC::BytecodeGenerator::resolve):
(JSC::BytecodeGenerator::emitResolve):
(JSC::BytecodeGenerator::emitResolveWithBase):
(JSC::BytecodeGenerator::emitResolveWithThis):
(JSC::BytecodeGenerator::emitGetStaticVar):
(JSC::BytecodeGenerator::emitPutStaticVar):

• bytecompiler/BytecodeGenerator.h:

(BytecodeGenerator):

• bytecompiler/NodesCodegen.cpp:

(JSC::FunctionCallResolveNode::emitBytecode):
(JSC::PostfixResolveNode::emitBytecode):
(JSC::PrefixResolveNode::emitBytecode):
(JSC::ReadModifyResolveNode::emitBytecode):
(JSC::AssignResolveNode::emitBytecode):
(JSC::ConstDeclNode::emitCodeSingle):

• dfg/DFGAbstractState.cpp:

(JSC::DFG::AbstractState::execute):
(JSC::DFG::AbstractState::clobberStructures):

• dfg/DFGAbstractState.h:

(AbstractState):
(JSC::DFG::AbstractState::didClobber):

• dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::handleInlining):
(JSC::DFG::ByteCodeParser::parseBlock):

• dfg/DFGCCallHelpers.h:

(CCallHelpers):
(JSC::DFG::CCallHelpers::setupArguments):

• dfg/DFGCSEPhase.cpp:

(JSC::DFG::CSEPhase::globalVarWatchpointElimination):
(CSEPhase):
(JSC::DFG::CSEPhase::globalVarStoreElimination):
(JSC::DFG::CSEPhase::performNodeCSE):

• dfg/DFGCapabilities.h:

(JSC::DFG::canCompileOpcode):

• dfg/DFGConstantFoldingPhase.cpp:

(JSC::DFG::ConstantFoldingPhase::run):

• dfg/DFGCorrectableJumpPoint.h:

(JSC::DFG::CorrectableJumpPoint::isSet):
(CorrectableJumpPoint):

• dfg/DFGJITCompiler.cpp:

(JSC::DFG::JITCompiler::linkOSRExits):
(JSC::DFG::JITCompiler::link):

• dfg/DFGNode.h:

(JSC::DFG::Node::hasIdentifierNumberForCheck):
(Node):
(JSC::DFG::Node::identifierNumberForCheck):
(JSC::DFG::Node::hasRegisterPointer):

• dfg/DFGNodeType.h:

(DFG):

• dfg/DFGOSRExit.cpp:

(JSC::DFG::OSRExit::OSRExit):

• dfg/DFGOSRExit.h:

(OSRExit):

• dfg/DFGOperations.cpp:
• dfg/DFGOperations.h:
• dfg/DFGPredictionPropagationPhase.cpp:

(JSC::DFG::PredictionPropagationPhase::propagate):

• dfg/DFGSpeculativeJIT.h:

(JSC::DFG::SpeculativeJIT::callOperation):
(JSC::DFG::SpeculativeJIT::appendCall):
(SpeculativeJIT):
(JSC::DFG::SpeculativeJIT::speculationWatchpoint):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• jit/JIT.cpp:

(JSC::JIT::privateCompileMainPass):
(JSC::JIT::privateCompileSlowCases):

• jit/JIT.h:
• jit/JITPropertyAccess.cpp:

(JSC::JIT::emit_op_put_global_var_check):
(JSC):
(JSC::JIT::emitSlow_op_put_global_var_check):

• jit/JITPropertyAccess32_64.cpp:

(JSC::JIT::emit_op_put_global_var_check):
(JSC):
(JSC::JIT::emitSlow_op_put_global_var_check):

• jit/JITStubs.cpp:

(JSC::JITThunks::JITThunks):
(JSC::DEFINE_STUB_FUNCTION):
(JSC):

• jit/JITStubs.h:
• llint/LLIntSlowPaths.cpp:

(JSC::LLInt::LLINT_SLOW_PATH_DECL):
(LLInt):

• llint/LLIntSlowPaths.h:

(LLInt):

• llint/LowLevelInterpreter32_64.asm:
• llint/LowLevelInterpreter64.asm:
• runtime/JSObject.cpp:

(JSC::JSObject::removeDirect):

• runtime/JSObject.h:

(JSObject):

• runtime/JSSymbolTableObject.h:

(JSC::symbolTableGet):
(JSC::symbolTablePut):
(JSC::symbolTablePutWithAttributes):

• runtime/SymbolTable.cpp: Added.

(JSC):
(JSC::SymbolTableEntry::copySlow):
(JSC::SymbolTableEntry::freeFatEntrySlow):
(JSC::SymbolTableEntry::couldBeWatched):
(JSC::SymbolTableEntry::attemptToWatch):
(JSC::SymbolTableEntry::addressOfIsWatched):
(JSC::SymbolTableEntry::addWatchpoint):
(JSC::SymbolTableEntry::notifyWriteSlow):
(JSC::SymbolTableEntry::inflateSlow):

• runtime/SymbolTable.h:

(JSC):
(SymbolTableEntry):
(Fast):
(JSC::SymbolTableEntry::Fast::Fast):
(JSC::SymbolTableEntry::Fast::isNull):
(JSC::SymbolTableEntry::Fast::getIndex):
(JSC::SymbolTableEntry::Fast::isReadOnly):
(JSC::SymbolTableEntry::Fast::getAttributes):
(JSC::SymbolTableEntry::Fast::isFat):
(JSC::SymbolTableEntry::SymbolTableEntry):
(JSC::SymbolTableEntry::~SymbolTableEntry):
(JSC::SymbolTableEntry::operator=):
(JSC::SymbolTableEntry::isNull):
(JSC::SymbolTableEntry::getIndex):
(JSC::SymbolTableEntry::getFast):
(JSC::SymbolTableEntry::getAttributes):
(JSC::SymbolTableEntry::isReadOnly):
(JSC::SymbolTableEntry::watchpointSet):
(JSC::SymbolTableEntry::notifyWrite):
(FatEntry):
(JSC::SymbolTableEntry::FatEntry::FatEntry):
(JSC::SymbolTableEntry::isFat):
(JSC::SymbolTableEntry::fatEntry):
(JSC::SymbolTableEntry::inflate):
(JSC::SymbolTableEntry::bits):
(JSC::SymbolTableEntry::freeFatEntry):
(JSC::SymbolTableEntry::pack):
(JSC::SymbolTableEntry::isValidIndex):

Source/WTF:

Reviewed by Geoffrey Garen.

Added ability to set the inline capacity of segmented vectors.

• wtf/SegmentedVector.h:

(WTF):
(SegmentedVectorIterator):
(WTF::SegmentedVectorIterator::operator=):
(WTF::SegmentedVectorIterator::SegmentedVectorIterator):
(SegmentedVector):

LayoutTests:

Rubber stamped by Geoffrey Garen.

Added a test for watchpoints. Also updated the jsc-test-list to include the latest
tests.

• fast/js/dfg-call-function-hit-watchpoint-expected.txt: Added.
• fast/js/dfg-call-function-hit-watchpoint.html: Added.
• fast/js/jsc-test-list:
• fast/js/script-tests/dfg-call-function-hit-watchpoint.js: Added.

(foo):
(bar):
(.foo):

File:

• trunk/Source/JavaScriptCore/runtime/SymbolTable.h (modified) (7 diffs)

trunk/Source/JavaScriptCore/runtime/SymbolTable.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2007, 2008 Apple Inc. All rights reserved.
+ * Copyright (C) 2007, 2008, 2012 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -32 +32 @@
 #include "JSObject.h"
 #include "UString.h"
+#include "Watchpoint.h"
 #include <wtf/AlwaysInline.h>
 #include <wtf/HashTraits.h>
 
 namespace JSC {
+
+    class Watchpoint;
+    class WatchpointSet;
 
     static ALWAYS_INLINE int missingSymbolMarker() { return std::numeric_limits<int>::max(); }
@@ -43 +47 @@
     // four bits all set or all unset.
 
+    // In addition to implementing semantics-mandated variable attributes and
+    // implementation-mandated variable indexing, this class also implements
+    // watchpoints to be used for JIT optimizations. Because watchpoints are
+    // meant to be relatively rare, this class optimizes heavily for the case
+    // that they are not being used. To that end, this class uses the thin-fat
+    // idiom: either it is thin, in which case it contains an in-place encoded
+    // word that consists of attributes, the index, and a bit saying that it is
+    // thin; or it is fat, in which case it contains a pointer to a malloc'd
+    // data structure and a bit saying that it is fat. The malloc'd data
+    // structure will be malloced a second time upon copy, to preserve the
+    // property that in-place edits to SymbolTableEntry do not manifest in any
+    // copies. However, the malloc'd FatEntry data structure contains a ref-
+    // counted pointer to a shared WatchpointSet. Thus, in-place edits of the
+    // WatchpointSet will manifest in all copies. Here's a picture:
+    //
+    // SymbolTableEntry --> FatEntry --> WatchpointSet
+    //
+    // If you make a copy of a SymbolTableEntry, you will have:
+    //
+    // original: SymbolTableEntry --> FatEntry --> WatchpointSet
+    // copy:     SymbolTableEntry --> FatEntry -----^
+
     struct SymbolTableEntry {
+        // Use the SymbolTableEntry::Fast class, either via implicit cast or by calling
+        // getFast(), when you (1) only care about isNull(), getIndex(), and isReadOnly(),
+        // and (2) you are in a hot path where you need to minimize the number of times
+        // that you branch on isFat() when getting the bits().
+        class Fast {
+        public:
+            Fast()
+                : m_bits(0)
+            {
+            }
+            
+            ALWAYS_INLINE Fast(const SymbolTableEntry& entry)
+                : m_bits(entry.bits())
+            {
+            }
+        
+            bool isNull() const
+            {
+                return !m_bits;
+            }
+
+            int getIndex() const
+            {
+                return static_cast<int>(m_bits >> FlagBits);
+            }
+        
+            bool isReadOnly() const
+            {
+                return m_bits & ReadOnlyFlag;
+            }
+            
+            unsigned getAttributes() const
+            {
+                unsigned attributes = 0;
+                if (m_bits & ReadOnlyFlag)
+                    attributes |= ReadOnly;
+                if (m_bits & DontEnumFlag)
+                    attributes |= DontEnum;
+                return attributes;
+            }
+
+            bool isFat() const
+            {
+                return m_bits & FatFlag;
+            }
+            
+        private:
+            friend struct SymbolTableEntry;
+            intptr_t m_bits;
+        };
+
         SymbolTableEntry()
             : m_bits(0)
@@ -50 +127 @@
 
         SymbolTableEntry(int index)
+            : m_bits(0)
         {
             ASSERT(isValidIndex(index));
@@ -56 +134 @@
 
         SymbolTableEntry(int index, unsigned attributes)
+            : m_bits(0)
         {
             ASSERT(isValidIndex(index));
@@ -61 +140 @@
         }
         
+        ~SymbolTableEntry()
+        {
+            freeFatEntry();
+        }
+        
+        SymbolTableEntry(const SymbolTableEntry& other)
+            : m_bits(0)
+        {
+            *this = other;
+        }
+        
+        SymbolTableEntry& operator=(const SymbolTableEntry& other)
+        {
+            if (UNLIKELY(other.isFat()))
+                return copySlow(other);
+            freeFatEntry();
+            m_bits = other.m_bits;
+            return *this;
+        }
+        
         bool isNull() const
         {
-            return !m_bits;
+            return !bits();
         }
 
         int getIndex() const
         {
-            return m_bits >> FlagBits;
-        }
-
+            return static_cast<int>(bits() >> FlagBits);
+        }
+        
+        ALWAYS_INLINE Fast getFast() const
+        {
+            return Fast(*this);
+        }
+        
+        ALWAYS_INLINE Fast getFast(bool& wasFat) const
+        {
+            Fast result;
+            wasFat = isFat();
+            if (wasFat)
+                result.m_bits = fatEntry()->m_bits;
+            else
+                result.m_bits = m_bits;
+            return result;
+        }
+        
         unsigned getAttributes() const
         {
-            unsigned attributes = 0;
-            if (m_bits & ReadOnlyFlag)
-                attributes |= ReadOnly;
-            if (m_bits & DontEnumFlag)
-                attributes |= DontEnum;
-            return attributes;
+            return getFast().getAttributes();
         }
 
@@ -88 +198 @@
         bool isReadOnly() const
         {
-            return m_bits & ReadOnlyFlag;
-        }
-
+            return bits() & ReadOnlyFlag;
+        }
+        
+        bool couldBeWatched();
+        
+        // Notify an opportunity to create a watchpoint for a variable. This is
+        // idempotent and fail-silent. It is idempotent in the sense that if
+        // a watchpoint set had already been created, then another one will not
+        // be created. Hence two calls to this method have the same effect as
+        // one call. It is also fail-silent, in the sense that if a watchpoint
+        // set had been created and had already been invalidated, then this will
+        // just return. This means that couldBeWatched() may return false even
+        // immediately after a call to attemptToWatch().
+        void attemptToWatch();
+        
+        bool* addressOfIsWatched();
+        
+        void addWatchpoint(Watchpoint*);
+        
+        WatchpointSet* watchpointSet()
+        {
+            return fatEntry()->m_watchpoints.get();
+        }
+        
+        ALWAYS_INLINE void notifyWrite()
+        {
+            if (LIKELY(!isFat()))
+                return;
+            notifyWriteSlow();
+        }
+        
     private:
-        static const unsigned ReadOnlyFlag = 0x1;
-        static const unsigned DontEnumFlag = 0x2;
-        static const unsigned NotNullFlag = 0x4;
-        static const unsigned FlagBits = 3;
+        static const intptr_t FatFlag = 0x1;
+        static const intptr_t ReadOnlyFlag = 0x2;
+        static const intptr_t DontEnumFlag = 0x4;
+        static const intptr_t NotNullFlag = 0x8;
+        static const intptr_t FlagBits = 4;
+        
+        class FatEntry {
+            WTF_MAKE_FAST_ALLOCATED;
+        public:
+            FatEntry(intptr_t bits)
+                : m_bits(bits | FatFlag)
+            {
+            }
+            
+            intptr_t m_bits; // always has FatFlag set and exactly matches what the bits would have been if this wasn't fat.
+            
+            RefPtr<WatchpointSet> m_watchpoints;
+        };
+        
+        SymbolTableEntry& copySlow(const SymbolTableEntry&);
+        JS_EXPORT_PRIVATE void notifyWriteSlow();
+        
+        bool isFat() const
+        {
+            return m_bits & FatFlag;
+        }
+        
+        const FatEntry* fatEntry() const
+        {
+            ASSERT(isFat());
+            return bitwise_cast<const FatEntry*>(m_bits & ~FatFlag);
+        }
+        
+        FatEntry* fatEntry()
+        {
+            ASSERT(isFat());
+            return bitwise_cast<FatEntry*>(m_bits & ~FatFlag);
+        }
+        
+        FatEntry* inflate()
+        {
+            if (LIKELY(isFat()))
+                return fatEntry();
+            return inflateSlow();
+        }
+        
+        FatEntry* inflateSlow();
+        
+        ALWAYS_INLINE intptr_t bits() const
+        {
+            if (isFat())
+                return fatEntry()->m_bits;
+            return m_bits;
+        }
+        
+        ALWAYS_INLINE intptr_t& bits()
+        {
+            if (isFat())
+                return fatEntry()->m_bits;
+            return m_bits;
+        }
+        
+        void freeFatEntry()
+        {
+            if (LIKELY(!isFat()))
+                return;
+            freeFatEntrySlow();
+        }
+        
+        void freeFatEntrySlow();
 
         void pack(int index, bool readOnly, bool dontEnum)
         {
-            m_bits = (index << FlagBits) | NotNullFlag;
+            intptr_t& bitsRef = bits();
+            bitsRef = (static_cast<intptr_t>(index) << FlagBits) | NotNullFlag;
             if (readOnly)
-                m_bits |= ReadOnlyFlag;
+                bitsRef |= ReadOnlyFlag;
             if (dontEnum)
-                m_bits |= DontEnumFlag;
+                bitsRef |= DontEnumFlag;
         }
         
         bool isValidIndex(int index)
         {
-            return ((index << FlagBits) >> FlagBits) == index;
-        }
-
-        int m_bits;
+            return ((static_cast<intptr_t>(index) << FlagBits) >> FlagBits) == static_cast<intptr_t>(index);
+        }
+
+        intptr_t m_bits;
     };