Changeset 179357 in webkit

Timestamp:

Jan 29, 2015, 12:33:45 PM (11 years ago)

Author:

[email protected]

Message:

Polymorphic call inlining should be based on polymorphic call inline caching rather than logging
​https://bugs.webkit.org/show_bug.cgi?id=140660

Reviewed by Geoffrey Garen.

When we first implemented polymorphic call inlining, we did the profiling based on a call
edge log. The idea was to store each call edge (a tuple of call site and callee) into a
global log that was processed lazily. Processing the log would give precise counts of call
edges, and could be used to drive well-informed inlining decisions - polymorphic or not.
This was a speed-up on throughput tests but a slow-down for latency tests. It was a net win
nonetheless.

Experience with this code shows three things. First, the call edge profiler is buggy and
complex. It would take work to fix the bugs. Second, the call edge profiler incurs lots of
overhead for latency code that we care deeply about. Third, it's not at all clear that
having call edge counts for every possible callee is any better than just having call edge
counts for the limited number of callees that an inline cache would catch.

So, this patch removes the call edge profiler and replaces it with a polymorphic call inline
cache. If we miss the basic call inline cache, we inflate the cache to be a jump to an
out-of-line stub that cases on the previously known callees. If that misses again, then we
rewrite that stub to include the new callee. We do this up to some number of callees. If we
hit the limit then we switch to using a plain virtual call.

Substantial speed-up on V8Spider; undoes the slow-down that the original call edge profiler
caused. Might be a SunSpider speed-up (below 1%), depending on hardware.

• CMakeLists.txt:
• JavaScriptCore.vcxproj/JavaScriptCore.vcxproj:
• JavaScriptCore.xcodeproj/project.pbxproj:
• bytecode/CallEdge.h:

(JSC::CallEdge::count):
(JSC::CallEdge::CallEdge):

• bytecode/CallEdgeProfile.cpp: Removed.
• bytecode/CallEdgeProfile.h: Removed.
• bytecode/CallEdgeProfileInlines.h: Removed.
• bytecode/CallLinkInfo.cpp:

(JSC::CallLinkInfo::unlink):
(JSC::CallLinkInfo::visitWeak):

• bytecode/CallLinkInfo.h:
• bytecode/CallLinkStatus.cpp:

(JSC::CallLinkStatus::CallLinkStatus):
(JSC::CallLinkStatus::computeFor):
(JSC::CallLinkStatus::computeFromCallLinkInfo):
(JSC::CallLinkStatus::isClosureCall):
(JSC::CallLinkStatus::makeClosureCall):
(JSC::CallLinkStatus::dump):
(JSC::CallLinkStatus::computeFromCallEdgeProfile): Deleted.

• bytecode/CallLinkStatus.h:

(JSC::CallLinkStatus::CallLinkStatus):
(JSC::CallLinkStatus::isSet):
(JSC::CallLinkStatus::variants):
(JSC::CallLinkStatus::size):
(JSC::CallLinkStatus::at):
(JSC::CallLinkStatus::operator[]):
(JSC::CallLinkStatus::canOptimize):
(JSC::CallLinkStatus::edges): Deleted.
(JSC::CallLinkStatus::canTrustCounts): Deleted.

• bytecode/CallVariant.cpp:

(JSC::variantListWithVariant):
(JSC::despecifiedVariantList):

• bytecode/CallVariant.h:
• bytecode/CodeBlock.cpp:

(JSC::CodeBlock::~CodeBlock):
(JSC::CodeBlock::linkIncomingPolymorphicCall):
(JSC::CodeBlock::unlinkIncomingCalls):
(JSC::CodeBlock::noticeIncomingCall):

• bytecode/CodeBlock.h:

(JSC::CodeBlock::isIncomingCallAlreadyLinked): Deleted.

• dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

• dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::addCallWithoutSettingResult):
(JSC::DFG::ByteCodeParser::handleCall):
(JSC::DFG::ByteCodeParser::handleInlining):

• dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

• dfg/DFGConstantFoldingPhase.cpp:

(JSC::DFG::ConstantFoldingPhase::foldConstants):

• dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

• dfg/DFGDriver.cpp:

(JSC::DFG::compileImpl):

• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

• dfg/DFGNode.h:

(JSC::DFG::Node::hasHeapPrediction):

• dfg/DFGNodeType.h:
• dfg/DFGOperations.cpp:
• dfg/DFGPredictionPropagationPhase.cpp:

(JSC::DFG::PredictionPropagationPhase::propagate):

• dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGTierUpCheckInjectionPhase.cpp:

(JSC::DFG::TierUpCheckInjectionPhase::run):
(JSC::DFG::TierUpCheckInjectionPhase::removeFTLProfiling): Deleted.

• ftl/FTLCapabilities.cpp:

(JSC::FTL::canCompile):

• heap/Heap.cpp:

(JSC::Heap::collect):

• jit/BinarySwitch.h:
• jit/ClosureCallStubRoutine.cpp: Removed.
• jit/ClosureCallStubRoutine.h: Removed.
• jit/JITCall.cpp:

(JSC::JIT::compileOpCall):

• jit/JITCall32_64.cpp:

(JSC::JIT::compileOpCall):

• jit/JITOperations.cpp:
• jit/JITOperations.h:

(JSC::operationLinkPolymorphicCallFor):
(JSC::operationLinkClosureCallFor): Deleted.

• jit/JITStubRoutine.h:
• jit/JITWriteBarrier.h:
• jit/PolymorphicCallStubRoutine.cpp: Added.

(JSC::PolymorphicCallNode::~PolymorphicCallNode):
(JSC::PolymorphicCallNode::unlink):
(JSC::PolymorphicCallCase::dump):
(JSC::PolymorphicCallStubRoutine::PolymorphicCallStubRoutine):
(JSC::PolymorphicCallStubRoutine::~PolymorphicCallStubRoutine):
(JSC::PolymorphicCallStubRoutine::variants):
(JSC::PolymorphicCallStubRoutine::edges):
(JSC::PolymorphicCallStubRoutine::visitWeak):
(JSC::PolymorphicCallStubRoutine::markRequiredObjectsInternal):

• jit/PolymorphicCallStubRoutine.h: Added.

(JSC::PolymorphicCallNode::PolymorphicCallNode):
(JSC::PolymorphicCallCase::PolymorphicCallCase):
(JSC::PolymorphicCallCase::variant):
(JSC::PolymorphicCallCase::codeBlock):

• jit/Repatch.cpp:

(JSC::linkSlowFor):
(JSC::linkFor):
(JSC::revertCall):
(JSC::unlinkFor):
(JSC::linkVirtualFor):
(JSC::linkPolymorphicCall):
(JSC::linkClosureCall): Deleted.

• jit/Repatch.h:
• jit/ThunkGenerators.cpp:

(JSC::linkPolymorphicCallForThunkGenerator):
(JSC::linkPolymorphicCallThunkGenerator):
(JSC::linkPolymorphicCallThatPreservesRegsThunkGenerator):
(JSC::linkClosureCallForThunkGenerator): Deleted.
(JSC::linkClosureCallThunkGenerator): Deleted.
(JSC::linkClosureCallThatPreservesRegsThunkGenerator): Deleted.

• jit/ThunkGenerators.h:

(JSC::linkPolymorphicCallThunkGeneratorFor):
(JSC::linkClosureCallThunkGeneratorFor): Deleted.

• llint/LLIntSlowPaths.cpp:

(JSC::LLInt::jitCompileAndSetHeuristics):

• runtime/Options.h:
• runtime/VM.cpp:

(JSC::VM::prepareToDiscardCode):
(JSC::VM::ensureCallEdgeLog): Deleted.

• runtime/VM.h:

Location:

trunk/Source/JavaScriptCore

Files:

• CMakeLists.txt (modified) (3 diffs)
• ChangeLog (modified) (1 diff)
• JavaScriptCore.vcxproj/JavaScriptCore.vcxproj (modified) (6 diffs)
• JavaScriptCore.xcodeproj/project.pbxproj (modified) (21 diffs)
• bytecode/CallEdge.h (modified) (3 diffs)
• bytecode/CallEdgeProfile.cpp (deleted)
• bytecode/CallEdgeProfile.h (deleted)
• bytecode/CallEdgeProfileInlines.h (deleted)
• bytecode/CallLinkInfo.cpp (modified) (4 diffs)
• bytecode/CallLinkInfo.h (modified) (5 diffs)
• bytecode/CallLinkStatus.cpp (modified) (8 diffs)
• bytecode/CallLinkStatus.h (modified) (7 diffs)
• bytecode/CallVariant.cpp (modified) (2 diffs)
• bytecode/CallVariant.h (modified) (3 diffs)
• bytecode/CodeBlock.cpp (modified) (7 diffs)
• bytecode/CodeBlock.h (modified) (2 diffs)
• dfg/DFGAbstractInterpreterInlines.h (modified) (1 diff)
• dfg/DFGByteCodeParser.cpp (modified) (12 diffs)
• dfg/DFGClobberize.h (modified) (1 diff)
• dfg/DFGConstantFoldingPhase.cpp (modified) (1 diff)
• dfg/DFGDoesGC.cpp (modified) (1 diff)
• dfg/DFGDriver.cpp (modified) (2 diffs)
• dfg/DFGFixupPhase.cpp (modified) (1 diff)
• dfg/DFGNode.h (modified) (1 diff)
• dfg/DFGNodeType.h (modified) (1 diff)
• dfg/DFGOperations.cpp (modified) (2 diffs)
• dfg/DFGPredictionPropagationPhase.cpp (modified) (1 diff)
• dfg/DFGSafeToExecute.h (modified) (1 diff)
• dfg/DFGSpeculativeJIT32_64.cpp (modified) (3 diffs)
• dfg/DFGSpeculativeJIT64.cpp (modified) (3 diffs)
• dfg/DFGTierUpCheckInjectionPhase.cpp (modified) (2 diffs)
• ftl/FTLCapabilities.cpp (modified) (1 diff)
• heap/Heap.cpp (modified) (1 diff)
• jit/BinarySwitch.h (modified) (1 diff)
• jit/ClosureCallStubRoutine.cpp (deleted)
• jit/ClosureCallStubRoutine.h (deleted)
• jit/JITCall.cpp (modified) (1 diff)
• jit/JITCall32_64.cpp (modified) (1 diff)
• jit/JITOperations.cpp (modified) (4 diffs)
• jit/JITOperations.h (modified) (2 diffs)
• jit/JITStubRoutine.h (modified) (1 diff)
• jit/JITWriteBarrier.h (modified) (1 diff)
• jit/PolymorphicCallStubRoutine.cpp (added)
• jit/PolymorphicCallStubRoutine.h (added)
• jit/Repatch.cpp (modified) (8 diffs)
• jit/Repatch.h (modified) (3 diffs)
• jit/ThunkGenerators.cpp (modified) (1 diff)
• jit/ThunkGenerators.h (modified) (1 diff)
• llint/LLIntSlowPaths.cpp (modified) (1 diff)
• runtime/Options.h (modified) (1 diff)
• runtime/VM.cpp (modified) (2 diffs)
• runtime/VM.h (modified) (2 diffs)

trunk/Source/JavaScriptCore/CMakeLists.txt

@@ -67 +67 @@
     bytecode/BytecodeLivenessAnalysis.cpp
     bytecode/CallEdge.cpp
-    bytecode/CallEdgeProfile.cpp
     bytecode/CallLinkInfo.cpp
     bytecode/CallLinkStatus.cpp
@@ -328 +327 @@
     jit/ArityCheckFailReturnThunks.cpp
     jit/BinarySwitch.cpp
-    jit/ClosureCallStubRoutine.cpp
     jit/ExecutableAllocator.cpp
     jit/ExecutableAllocatorFixedVMPool.cpp
@@ -351 +349 @@
     jit/JITThunks.cpp
     jit/JITToDFGDeferredCompilationCallback.cpp
+    jit/PolymorphicCallStubRoutine.cpp
     jit/Reg.cpp
     jit/RegisterPreservationWrapperGenerator.cpp

trunk/Source/JavaScriptCore/ChangeLog

@@ +1 @@
+2015-01-28  Filip Pizlo  <[email protected]>
+
+        Polymorphic call inlining should be based on polymorphic call inline caching rather than logging
+        https://bugs.webkit.org/show_bug.cgi?id=140660
+
+        Reviewed by Geoffrey Garen.
+        
+        When we first implemented polymorphic call inlining, we did the profiling based on a call
+        edge log. The idea was to store each call edge (a tuple of call site and callee) into a
+        global log that was processed lazily. Processing the log would give precise counts of call
+        edges, and could be used to drive well-informed inlining decisions - polymorphic or not.
+        This was a speed-up on throughput tests but a slow-down for latency tests. It was a net win
+        nonetheless.
+        
+        Experience with this code shows three things. First, the call edge profiler is buggy and
+        complex. It would take work to fix the bugs. Second, the call edge profiler incurs lots of
+        overhead for latency code that we care deeply about. Third, it's not at all clear that
+        having call edge counts for every possible callee is any better than just having call edge
+        counts for the limited number of callees that an inline cache would catch.
+        
+        So, this patch removes the call edge profiler and replaces it with a polymorphic call inline
+        cache. If we miss the basic call inline cache, we inflate the cache to be a jump to an
+        out-of-line stub that cases on the previously known callees. If that misses again, then we
+        rewrite that stub to include the new callee. We do this up to some number of callees. If we
+        hit the limit then we switch to using a plain virtual call.
+        
+        Substantial speed-up on V8Spider; undoes the slow-down that the original call edge profiler
+        caused. Might be a SunSpider speed-up (below 1%), depending on hardware.
+
+        * CMakeLists.txt:
+        * JavaScriptCore.vcxproj/JavaScriptCore.vcxproj:
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * bytecode/CallEdge.h:
+        (JSC::CallEdge::count):
+        (JSC::CallEdge::CallEdge):
+        * bytecode/CallEdgeProfile.cpp: Removed.
+        * bytecode/CallEdgeProfile.h: Removed.
+        * bytecode/CallEdgeProfileInlines.h: Removed.
+        * bytecode/CallLinkInfo.cpp:
+        (JSC::CallLinkInfo::unlink):
+        (JSC::CallLinkInfo::visitWeak):
+        * bytecode/CallLinkInfo.h:
+        * bytecode/CallLinkStatus.cpp:
+        (JSC::CallLinkStatus::CallLinkStatus):
+        (JSC::CallLinkStatus::computeFor):
+        (JSC::CallLinkStatus::computeFromCallLinkInfo):
+        (JSC::CallLinkStatus::isClosureCall):
+        (JSC::CallLinkStatus::makeClosureCall):
+        (JSC::CallLinkStatus::dump):
+        (JSC::CallLinkStatus::computeFromCallEdgeProfile): Deleted.
+        * bytecode/CallLinkStatus.h:
+        (JSC::CallLinkStatus::CallLinkStatus):
+        (JSC::CallLinkStatus::isSet):
+        (JSC::CallLinkStatus::variants):
+        (JSC::CallLinkStatus::size):
+        (JSC::CallLinkStatus::at):
+        (JSC::CallLinkStatus::operator[]):
+        (JSC::CallLinkStatus::canOptimize):
+        (JSC::CallLinkStatus::edges): Deleted.
+        (JSC::CallLinkStatus::canTrustCounts): Deleted.
+        * bytecode/CallVariant.cpp:
+        (JSC::variantListWithVariant):
+        (JSC::despecifiedVariantList):
+        * bytecode/CallVariant.h:
+        * bytecode/CodeBlock.cpp:
+        (JSC::CodeBlock::~CodeBlock):
+        (JSC::CodeBlock::linkIncomingPolymorphicCall):
+        (JSC::CodeBlock::unlinkIncomingCalls):
+        (JSC::CodeBlock::noticeIncomingCall):
+        * bytecode/CodeBlock.h:
+        (JSC::CodeBlock::isIncomingCallAlreadyLinked): Deleted.
+        * dfg/DFGAbstractInterpreterInlines.h:
+        (JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
+        * dfg/DFGByteCodeParser.cpp:
+        (JSC::DFG::ByteCodeParser::addCallWithoutSettingResult):
+        (JSC::DFG::ByteCodeParser::handleCall):
+        (JSC::DFG::ByteCodeParser::handleInlining):
+        * dfg/DFGClobberize.h:
+        (JSC::DFG::clobberize):
+        * dfg/DFGConstantFoldingPhase.cpp:
+        (JSC::DFG::ConstantFoldingPhase::foldConstants):
+        * dfg/DFGDoesGC.cpp:
+        (JSC::DFG::doesGC):
+        * dfg/DFGDriver.cpp:
+        (JSC::DFG::compileImpl):
+        * dfg/DFGFixupPhase.cpp:
+        (JSC::DFG::FixupPhase::fixupNode):
+        * dfg/DFGNode.h:
+        (JSC::DFG::Node::hasHeapPrediction):
+        * dfg/DFGNodeType.h:
+        * dfg/DFGOperations.cpp:
+        * dfg/DFGPredictionPropagationPhase.cpp:
+        (JSC::DFG::PredictionPropagationPhase::propagate):
+        * dfg/DFGSafeToExecute.h:
+        (JSC::DFG::safeToExecute):
+        * dfg/DFGSpeculativeJIT32_64.cpp:
+        (JSC::DFG::SpeculativeJIT::emitCall):
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGSpeculativeJIT64.cpp:
+        (JSC::DFG::SpeculativeJIT::emitCall):
+        (JSC::DFG::SpeculativeJIT::compile):
+        * dfg/DFGTierUpCheckInjectionPhase.cpp:
+        (JSC::DFG::TierUpCheckInjectionPhase::run):
+        (JSC::DFG::TierUpCheckInjectionPhase::removeFTLProfiling): Deleted.
+        * ftl/FTLCapabilities.cpp:
+        (JSC::FTL::canCompile):
+        * heap/Heap.cpp:
+        (JSC::Heap::collect):
+        * jit/BinarySwitch.h:
+        * jit/ClosureCallStubRoutine.cpp: Removed.
+        * jit/ClosureCallStubRoutine.h: Removed.
+        * jit/JITCall.cpp:
+        (JSC::JIT::compileOpCall):
+        * jit/JITCall32_64.cpp:
+        (JSC::JIT::compileOpCall):
+        * jit/JITOperations.cpp:
+        * jit/JITOperations.h:
+        (JSC::operationLinkPolymorphicCallFor):
+        (JSC::operationLinkClosureCallFor): Deleted.
+        * jit/JITStubRoutine.h:
+        * jit/JITWriteBarrier.h:
+        * jit/PolymorphicCallStubRoutine.cpp: Added.
+        (JSC::PolymorphicCallNode::~PolymorphicCallNode):
+        (JSC::PolymorphicCallNode::unlink):
+        (JSC::PolymorphicCallCase::dump):
+        (JSC::PolymorphicCallStubRoutine::PolymorphicCallStubRoutine):
+        (JSC::PolymorphicCallStubRoutine::~PolymorphicCallStubRoutine):
+        (JSC::PolymorphicCallStubRoutine::variants):
+        (JSC::PolymorphicCallStubRoutine::edges):
+        (JSC::PolymorphicCallStubRoutine::visitWeak):
+        (JSC::PolymorphicCallStubRoutine::markRequiredObjectsInternal):
+        * jit/PolymorphicCallStubRoutine.h: Added.
+        (JSC::PolymorphicCallNode::PolymorphicCallNode):
+        (JSC::PolymorphicCallCase::PolymorphicCallCase):
+        (JSC::PolymorphicCallCase::variant):
+        (JSC::PolymorphicCallCase::codeBlock):
+        * jit/Repatch.cpp:
+        (JSC::linkSlowFor):
+        (JSC::linkFor):
+        (JSC::revertCall):
+        (JSC::unlinkFor):
+        (JSC::linkVirtualFor):
+        (JSC::linkPolymorphicCall):
+        (JSC::linkClosureCall): Deleted.
+        * jit/Repatch.h:
+        * jit/ThunkGenerators.cpp:
+        (JSC::linkPolymorphicCallForThunkGenerator):
+        (JSC::linkPolymorphicCallThunkGenerator):
+        (JSC::linkPolymorphicCallThatPreservesRegsThunkGenerator):
+        (JSC::linkClosureCallForThunkGenerator): Deleted.
+        (JSC::linkClosureCallThunkGenerator): Deleted.
+        (JSC::linkClosureCallThatPreservesRegsThunkGenerator): Deleted.
+        * jit/ThunkGenerators.h:
+        (JSC::linkPolymorphicCallThunkGeneratorFor):
+        (JSC::linkClosureCallThunkGeneratorFor): Deleted.
+        * llint/LLIntSlowPaths.cpp:
+        (JSC::LLInt::jitCompileAndSetHeuristics):
+        * runtime/Options.h:
+        * runtime/VM.cpp:
+        (JSC::VM::prepareToDiscardCode):
+        (JSC::VM::ensureCallEdgeLog): Deleted.
+        * runtime/VM.h:
+
 2015-01-29  Joseph Pecoraro  <[email protected]>
 

trunk/Source/JavaScriptCore/JavaScriptCore.vcxproj/JavaScriptCore.vcxproj

@@ -315 +315 @@
     <ClCompile Include="..\bytecode\BytecodeLivenessAnalysis.cpp" />
     <ClCompile Include="..\bytecode\CallEdge.cpp" />
-    <ClCompile Include="..\bytecode\CallEdgeProfile.cpp" />
     <ClCompile Include="..\bytecode\CallLinkInfo.cpp" />
     <ClCompile Include="..\bytecode\CallLinkStatus.cpp" />
@@ -599 +598 @@
     <ClCompile Include="..\jit\AssemblyHelpers.cpp" />
     <ClCompile Include="..\jit\BinarySwitch.cpp" />
-    <ClCompile Include="..\jit\ClosureCallStubRoutine.cpp" />
     <ClCompile Include="..\jit\ExecutableAllocator.cpp" />
     <ClCompile Include="..\jit\GCAwareJITStubRoutine.cpp" />
@@ -622 +620 @@
     <ClCompile Include="..\jit\JITThunks.cpp" />
     <ClCompile Include="..\jit\JITToDFGDeferredCompilationCallback.cpp" />
+    <ClCompile Include="..\jit\PolymorphicCallStubRoutine.cpp" />
     <ClCompile Include="..\jit\Reg.cpp" />
     <ClCompile Include="..\jit\RegisterPreservationWrapperGenerator.cpp" />
@@ -932 +931 @@
     <ClInclude Include="..\bytecode\BytecodeUseDef.h" />
     <ClInclude Include="..\bytecode\CallEdge.h" />
-    <ClInclude Include="..\bytecode\CallEdgeProfile.h" />
-    <ClInclude Include="..\bytecode\CallEdgeProfileInlines.h" />
     <ClInclude Include="..\bytecode\CallLinkInfo.h" />
     <ClInclude Include="..\bytecode\CallLinkStatus.h" />
@@ -1334 +1331 @@
     <ClInclude Include="..\jit\BinarySwitch.h" />
     <ClInclude Include="..\jit\CCallHelpers.h" />
-    <ClInclude Include="..\jit\ClosureCallStubRoutine.h" />
     <ClInclude Include="..\jit\CompactJITCodeMap.h" />
     <ClInclude Include="..\jit\ExecutableAllocator.h" />
@@ -1358 +1354 @@
     <ClInclude Include="..\jit\JITWriteBarrier.h" />
     <ClInclude Include="..\jit\JSInterfaceJIT.h" />
+    <ClInclude Include="..\jit\PolymorphicCallStubRoutine.h" />
     <ClInclude Include="..\jit\Reg.h" />
     <ClInclude Include="..\jit\RegisterPreservationWrapperGenerator.h" />

trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

@@ -282 +282 @@
                 0F3B3A2B15475000003ED0FF /* DFGValidate.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F3B3A2915474FF4003ED0FF /* DFGValidate.cpp */; };
                 0F3B3A2C15475002003ED0FF /* DFGValidate.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F3B3A2A15474FF4003ED0FF /* DFGValidate.h */; settings = {ATTRIBUTES = (Private, ); }; };
-                0F3B7E2619A11B8000D9BC56 /* CallEdge.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F3B7E2019A11B8000D9BC56 /* CallEdge.h */; settings = {ATTRIBUTES = (Private, ); }; };
-                0F3B7E2719A11B8000D9BC56 /* CallEdgeProfile.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F3B7E2119A11B8000D9BC56 /* CallEdgeProfile.cpp */; };
-                0F3B7E2819A11B8000D9BC56 /* CallEdgeProfile.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F3B7E2219A11B8000D9BC56 /* CallEdgeProfile.h */; settings = {ATTRIBUTES = (Private, ); }; };
-                0F3B7E2919A11B8000D9BC56 /* CallEdgeProfileInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F3B7E2319A11B8000D9BC56 /* CallEdgeProfileInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 0F3B7E2A19A11B8000D9BC56 /* CallVariant.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F3B7E2419A11B8000D9BC56 /* CallVariant.cpp */; };
                 0F3B7E2B19A11B8000D9BC56 /* CallVariant.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F3B7E2519A11B8000D9BC56 /* CallVariant.h */; settings = {ATTRIBUTES = (Private, ); }; };
-                0F3B7E2D19A12AAE00D9BC56 /* CallEdge.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F3B7E2C19A12AAE00D9BC56 /* CallEdge.cpp */; };
                 0F3D0BBC194A414300FC9CF9 /* ConstantStructureCheck.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F3D0BBA194A414300FC9CF9 /* ConstantStructureCheck.cpp */; };
                 0F3D0BBD194A414300FC9CF9 /* ConstantStructureCheck.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F3D0BBB194A414300FC9CF9 /* ConstantStructureCheck.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -356 +351 @@
                 0F64B2711A784BAF006E4E66 /* BinarySwitch.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F64B26F1A784BAF006E4E66 /* BinarySwitch.cpp */; };
                 0F64B2721A784BAF006E4E66 /* BinarySwitch.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F64B2701A784BAF006E4E66 /* BinarySwitch.h */; settings = {ATTRIBUTES = (Private, ); }; };
+                0F64B2791A7957B2006E4E66 /* CallEdge.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F64B2771A7957B2006E4E66 /* CallEdge.cpp */; };
+                0F64B27A1A7957B2006E4E66 /* CallEdge.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F64B2781A7957B2006E4E66 /* CallEdge.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 0F666EC0183566F900D017F1 /* BytecodeLivenessAnalysisInlines.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F666EBE183566F900D017F1 /* BytecodeLivenessAnalysisInlines.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 0F666EC1183566F900D017F1 /* FullBytecodeLiveness.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F666EBF183566F900D017F1 /* FullBytecodeLiveness.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -387 +384 @@
                 0F714CA416EA92F000F3EBEB /* DFGBackwardsPropagationPhase.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F714CA116EA92ED00F3EBEB /* DFGBackwardsPropagationPhase.cpp */; };
                 0F714CA516EA92F200F3EBEB /* DFGBackwardsPropagationPhase.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F714CA216EA92ED00F3EBEB /* DFGBackwardsPropagationPhase.h */; settings = {ATTRIBUTES = (Private, ); }; };
-                0F73D7AE165A142D00ACAB71 /* ClosureCallStubRoutine.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F73D7AB165A142A00ACAB71 /* ClosureCallStubRoutine.cpp */; };
-                0F73D7AF165A143000ACAB71 /* ClosureCallStubRoutine.h in Headers */ = {isa = PBXBuildFile; fileRef = 0F73D7AC165A142A00ACAB71 /* ClosureCallStubRoutine.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 0F743BAA16B88249009F9277 /* ARM64Disassembler.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 652A3A201651C66100A80AFE /* ARM64Disassembler.cpp */; };
                 0F7576D218E1FEE9002EF4CD /* AccessorCallJITStubRoutine.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0F7576D018E1FEE9002EF4CD /* AccessorCallJITStubRoutine.cpp */; };
@@ -603 +598 @@
                 0FE7211D193B9C590031F6ED /* DFGTransition.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FE7211B193B9C590031F6ED /* DFGTransition.cpp */; };
                 0FE7211E193B9C590031F6ED /* DFGTransition.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FE7211C193B9C590031F6ED /* DFGTransition.h */; settings = {ATTRIBUTES = (Private, ); }; };
+                0FE834171A6EF97B00D04847 /* PolymorphicCallStubRoutine.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FE834151A6EF97B00D04847 /* PolymorphicCallStubRoutine.cpp */; };
+                0FE834181A6EF97B00D04847 /* PolymorphicCallStubRoutine.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FE834161A6EF97B00D04847 /* PolymorphicCallStubRoutine.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 0FE8534B1723CDA500B618F5 /* DFGDesiredWatchpoints.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FE853491723CDA500B618F5 /* DFGDesiredWatchpoints.cpp */; };
                 0FE8534C1723CDA500B618F5 /* DFGDesiredWatchpoints.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FE8534A1723CDA500B618F5 /* DFGDesiredWatchpoints.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -1951 +1948 @@
                 0F3B3A2915474FF4003ED0FF /* DFGValidate.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGValidate.cpp; path = dfg/DFGValidate.cpp; sourceTree = "<group>"; };
                 0F3B3A2A15474FF4003ED0FF /* DFGValidate.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGValidate.h; path = dfg/DFGValidate.h; sourceTree = "<group>"; };
-                0F3B7E2019A11B8000D9BC56 /* CallEdge.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CallEdge.h; sourceTree = "<group>"; };
-                0F3B7E2119A11B8000D9BC56 /* CallEdgeProfile.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallEdgeProfile.cpp; sourceTree = "<group>"; };
-                0F3B7E2219A11B8000D9BC56 /* CallEdgeProfile.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CallEdgeProfile.h; sourceTree = "<group>"; };
-                0F3B7E2319A11B8000D9BC56 /* CallEdgeProfileInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CallEdgeProfileInlines.h; sourceTree = "<group>"; };
                 0F3B7E2419A11B8000D9BC56 /* CallVariant.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallVariant.cpp; sourceTree = "<group>"; };
                 0F3B7E2519A11B8000D9BC56 /* CallVariant.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CallVariant.h; sourceTree = "<group>"; };
-                0F3B7E2C19A12AAE00D9BC56 /* CallEdge.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallEdge.cpp; sourceTree = "<group>"; };
                 0F3D0BBA194A414300FC9CF9 /* ConstantStructureCheck.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ConstantStructureCheck.cpp; sourceTree = "<group>"; };
                 0F3D0BBB194A414300FC9CF9 /* ConstantStructureCheck.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ConstantStructureCheck.h; sourceTree = "<group>"; };
@@ -2026 +2018 @@
                 0F64B26F1A784BAF006E4E66 /* BinarySwitch.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = BinarySwitch.cpp; sourceTree = "<group>"; };
                 0F64B2701A784BAF006E4E66 /* BinarySwitch.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BinarySwitch.h; sourceTree = "<group>"; };
+                0F64B2771A7957B2006E4E66 /* CallEdge.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = CallEdge.cpp; sourceTree = "<group>"; };
+                0F64B2781A7957B2006E4E66 /* CallEdge.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = CallEdge.h; sourceTree = "<group>"; };
                 0F666EBE183566F900D017F1 /* BytecodeLivenessAnalysisInlines.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = BytecodeLivenessAnalysisInlines.h; sourceTree = "<group>"; };
                 0F666EBF183566F900D017F1 /* FullBytecodeLiveness.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = FullBytecodeLiveness.h; sourceTree = "<group>"; };
@@ -2057 +2051 @@
                 0F714CA116EA92ED00F3EBEB /* DFGBackwardsPropagationPhase.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGBackwardsPropagationPhase.cpp; path = dfg/DFGBackwardsPropagationPhase.cpp; sourceTree = "<group>"; };
                 0F714CA216EA92ED00F3EBEB /* DFGBackwardsPropagationPhase.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGBackwardsPropagationPhase.h; path = dfg/DFGBackwardsPropagationPhase.h; sourceTree = "<group>"; };
-                0F73D7AB165A142A00ACAB71 /* ClosureCallStubRoutine.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ClosureCallStubRoutine.cpp; sourceTree = "<group>"; };
-                0F73D7AC165A142A00ACAB71 /* ClosureCallStubRoutine.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = ClosureCallStubRoutine.h; sourceTree = "<group>"; };
                 0F7576D018E1FEE9002EF4CD /* AccessorCallJITStubRoutine.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = AccessorCallJITStubRoutine.cpp; sourceTree = "<group>"; };
                 0F7576D118E1FEE9002EF4CD /* AccessorCallJITStubRoutine.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = AccessorCallJITStubRoutine.h; sourceTree = "<group>"; };
@@ -2285 +2277 @@
                 0FE7211B193B9C590031F6ED /* DFGTransition.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGTransition.cpp; path = dfg/DFGTransition.cpp; sourceTree = "<group>"; };
                 0FE7211C193B9C590031F6ED /* DFGTransition.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGTransition.h; path = dfg/DFGTransition.h; sourceTree = "<group>"; };
+                0FE834151A6EF97B00D04847 /* PolymorphicCallStubRoutine.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = PolymorphicCallStubRoutine.cpp; sourceTree = "<group>"; };
+                0FE834161A6EF97B00D04847 /* PolymorphicCallStubRoutine.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = PolymorphicCallStubRoutine.h; sourceTree = "<group>"; };
                 0FE853491723CDA500B618F5 /* DFGDesiredWatchpoints.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = DFGDesiredWatchpoints.cpp; path = dfg/DFGDesiredWatchpoints.cpp; sourceTree = "<group>"; };
                 0FE8534A1723CDA500B618F5 /* DFGDesiredWatchpoints.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGDesiredWatchpoints.h; path = dfg/DFGDesiredWatchpoints.h; sourceTree = "<group>"; };
@@ -3748 +3742 @@
                                 0F64B2701A784BAF006E4E66 /* BinarySwitch.h */,
                                 0F24E53D17EA9F5900ABB217 /* CCallHelpers.h */,
-                                0F73D7AB165A142A00ACAB71 /* ClosureCallStubRoutine.cpp */,
-                                0F73D7AC165A142A00ACAB71 /* ClosureCallStubRoutine.h */,
                                 0FD82E37141AB14200179C94 /* CompactJITCodeMap.h */,
                                 A7B48DB60EE74CFC00DCBDB6 /* ExecutableAllocator.cpp */,
@@ -3797 +3789 @@
                                 A76F54A213B28AAB00EF2BCE /* JITWriteBarrier.h */,
                                 A76C51741182748D00715B05 /* JSInterfaceJIT.h */,
+                                0FE834151A6EF97B00D04847 /* PolymorphicCallStubRoutine.cpp */,
+                                0FE834161A6EF97B00D04847 /* PolymorphicCallStubRoutine.h */,
                                 0FA7A8E918B413C80052371D /* Reg.cpp */,
                                 0FA7A8EA18B413C80052371D /* Reg.h */,
@@ -5028 +5022 @@
                                 0F885E101849A3BE00F1E3FA /* BytecodeUseDef.h */,
                                 0F8023E91613832300A0BA45 /* ByValInfo.h */,
-                                0F3B7E2C19A12AAE00D9BC56 /* CallEdge.cpp */,
-                                0F3B7E2019A11B8000D9BC56 /* CallEdge.h */,
-                                0F3B7E2119A11B8000D9BC56 /* CallEdgeProfile.cpp */,
-                                0F3B7E2219A11B8000D9BC56 /* CallEdgeProfile.h */,
-                                0F3B7E2319A11B8000D9BC56 /* CallEdgeProfileInlines.h */,
+                                0F64B2771A7957B2006E4E66 /* CallEdge.cpp */,
+                                0F64B2781A7957B2006E4E66 /* CallEdge.h */,
                                 0F0B83AE14BCF71400885B4F /* CallLinkInfo.cpp */,
                                 0F0B83AF14BCF71400885B4F /* CallLinkInfo.h */,
@@ -5484 +5475 @@
                                 0F24E54217EA9F5900ABB217 /* CCallHelpers.h in Headers */,
                                 BC6AAAE50E1F426500AD87D8 /* ClassInfo.h in Headers */,
-                                0F73D7AF165A143000ACAB71 /* ClosureCallStubRoutine.h in Headers */,
                                 969A07970ED1D3AE00F1F681 /* CodeBlock.h in Headers */,
                                 0F8F94411667633200D61971 /* CodeBlockHash.h in Headers */,
@@ -5528 +5518 @@
                                 41359CF30FDD89AD00206180 /* DateConversion.h in Headers */,
                                 BC1166020E1997B4008066DD /* DateInstance.h in Headers */,
+                                0F64B27A1A7957B2006E4E66 /* CallEdge.h in Headers */,
                                 14A1563210966365006FA260 /* DateInstanceCache.h in Headers */,
                                 BCD2034C0E17135E002C7E82 /* DatePrototype.h in Headers */,
@@ -5575 +5566 @@
                                 0F2FC77316E12F740038D976 /* DFGDCEPhase.h in Headers */,
                                 0F8F2B9A172F0501007DBDA5 /* DFGDesiredIdentifiers.h in Headers */,
-                                0F3B7E2819A11B8000D9BC56 /* CallEdgeProfile.h in Headers */,
                                 C2C0F7CE17BBFC5B00464FE4 /* DFGDesiredTransitions.h in Headers */,
                                 0FE8534C1723CDA500B618F5 /* DFGDesiredWatchpoints.h in Headers */,
@@ -5866 +5856 @@
                                 BC18C4160E16F5CD00B34460 /* JSLexicalEnvironment.h in Headers */,
                                 840480131021A1D9008E7F01 /* JSAPIValueWrapper.h in Headers */,
-                                0F3B7E2919A11B8000D9BC56 /* CallEdgeProfileInlines.h in Headers */,
                                 C2CF39C216E15A8100DD69BE /* JSAPIWrapperObject.h in Headers */,
                                 A76140D2182982CB00750624 /* JSArgumentsIterator.h in Headers */,
@@ -6190 +6179 @@
                                 E49DC16D12EF295300184A1F /* SourceProviderCacheItem.h in Headers */,
                                 0FB7F39E15ED8E4600F167B2 /* SparseArrayValueMap.h in Headers */,
-                                0F3B7E2619A11B8000D9BC56 /* CallEdge.h in Headers */,
                                 A7386554118697B400540279 /* SpecializedThunkJIT.h in Headers */,
                                 0F5541B21613C1FB00CE3E25 /* SpecialPointer.h in Headers */,
@@ -6237 +6225 @@
                                 A7E5AB391799E4B200D2833D /* UDis86Disassembler.h in Headers */,
                                 A7A8AF4117ADB5F3005AB174 /* Uint16Array.h in Headers */,
+                                0FE834181A6EF97B00D04847 /* PolymorphicCallStubRoutine.h in Headers */,
                                 866739D313BFDE710023D87C /* Uint16WithFraction.h in Headers */,
                                 A7A8AF4217ADB5F3005AB174 /* Uint32Array.h in Headers */,
@@ -6775 +6764 @@
                                 0F93329D14CA7DC30085F3C6 /* CallLinkStatus.cpp in Sources */,
                                 0F2B9CE419D0BA7D00B1D1B5 /* DFGInsertOSRHintsForUpdate.cpp in Sources */,
-                                0F73D7AE165A142D00ACAB71 /* ClosureCallStubRoutine.cpp in Sources */,
                                 969A07960ED1D3AE00F1F681 /* CodeBlock.cpp in Sources */,
                                 0F8F94401667633000D61971 /* CodeBlockHash.cpp in Sources */,
@@ -6790 +6778 @@
                                 A709F2F217A0AC2A00512E98 /* CommonSlowPaths.cpp in Sources */,
                                 6553A33117A1F1EE008CF6F3 /* CommonSlowPathsExceptions.cpp in Sources */,
+                                0F64B2791A7957B2006E4E66 /* CallEdge.cpp in Sources */,
                                 A7E5A3A71797432D00E893C0 /* CompilationResult.cpp in Sources */,
                                 147F39C2107EC37600427A48 /* Completion.cpp in Sources */,
@@ -6961 +6950 @@
                                 0F235BD817178E1C00690C7F /* FTLExitThunkGenerator.cpp in Sources */,
                                 0F235BDA17178E1C00690C7F /* FTLExitValue.cpp in Sources */,
-                                0F3B7E2719A11B8000D9BC56 /* CallEdgeProfile.cpp in Sources */,
                                 A7F2996B17A0BB670010417A /* FTLFail.cpp in Sources */,
+                                0FE834171A6EF97B00D04847 /* PolymorphicCallStubRoutine.cpp in Sources */,
                                 0FD8A31917D51F2200CA2C40 /* FTLForOSREntryJITCode.cpp in Sources */,
                                 0F25F1AF181635F300522F39 /* FTLInlineCacheSize.cpp in Sources */,
@@ -7154 +7143 @@
                                 0F38B01117CF078000B144D3 /* LLIntEntrypoint.cpp in Sources */,
                                 0F4680A814BA7FAB00BFE272 /* LLIntExceptions.cpp in Sources */,
-                                0F3B7E2D19A12AAE00D9BC56 /* CallEdge.cpp in Sources */,
                                 0F4680A414BA7F8D00BFE272 /* LLIntSlowPaths.cpp in Sources */,
                                 0F0B839C14BCF46300885B4F /* LLIntThunks.cpp in Sources */,

trunk/Source/JavaScriptCore/bytecode/CallEdge.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -31 +31 @@
 namespace JSC {
 
-typedef uint16_t CallEdgeCountType;
-
 class CallEdge {
 public:
     CallEdge();
-    CallEdge(CallVariant, CallEdgeCountType);
+    CallEdge(CallVariant, uint32_t);
     
     bool operator!() const { return !m_callee; }
     
     CallVariant callee() const { return m_callee; }
-    CallEdgeCountType count() const { return m_count; }
+    uint32_t count() const { return m_count; }
     
     CallEdge despecifiedClosure() const
@@ -50 +48 @@
     void dump(PrintStream&) const;
     
-public:
+private:
     CallVariant m_callee;
-    CallEdgeCountType m_count;
+    uint32_t m_count;
 };
 
-inline CallEdge::CallEdge(CallVariant callee, CallEdgeCountType count)
+inline CallEdge::CallEdge(CallVariant callee, uint32_t count)
     : m_callee(callee)
     , m_count(count)

trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.cpp

@@ -30 +30 @@
 #include "DFGThunks.h"
 #include "JSCInlines.h"
+#include "Repatch.h"
 #include "RepatchBuffer.h"
+#include <wtf/ListDump.h>
 #include <wtf/NeverDestroyed.h>
 
@@ -38 +40 @@
 void CallLinkInfo::unlink(RepatchBuffer& repatchBuffer)
 {
-    ASSERT(isLinked());
+    if (!isLinked()) {
+        // We could be called even if we're not linked anymore because of how polymorphic calls
+        // work. Each callsite within the polymorphic call stub may separately ask us to unlink().
+        RELEASE_ASSERT(!isOnList());
+        return;
+    }
     
-    if (Options::showDisassembly())
-        dataLog("Unlinking call from ", callReturnLocation, " to ", pointerDump(repatchBuffer.codeBlock()), "\n");
-
-    repatchBuffer.revertJumpReplacementToBranchPtrWithPatch(RepatchBuffer::startOfBranchPtrWithPatchOnRegister(hotPathBegin), static_cast<MacroAssembler::RegisterID>(calleeGPR), 0);
-    repatchBuffer.relink(
-        callReturnLocation,
-        repatchBuffer.codeBlock()->vm()->getCTIStub(linkThunkGeneratorFor(
-            (callType == Construct || callType == ConstructVarargs)? CodeForConstruct : CodeForCall,
-            isFTL ? MustPreserveRegisters : RegisterPreservationNotRequired)).code());
-    hasSeenShouldRepatch = false;
-    callee.clear();
-    stub.clear();
+    unlinkFor(
+        repatchBuffer, *this,
+        (callType == Construct || callType == ConstructVarargs)? CodeForConstruct : CodeForCall,
+        isFTL ? MustPreserveRegisters : RegisterPreservationNotRequired);
 
     // It will be on a list if the callee has a code block.
@@ -62 +61 @@
     if (isLinked()) {
         if (stub) {
-            if (!Heap::isMarked(stub->executable())) {
+            if (!stub->visitWeak(repatchBuffer)) {
                 if (Options::verboseOSR()) {
                     dataLog(
                         "Clearing closure call from ", *repatchBuffer.codeBlock(), " to ",
-                        stub->executable()->hashFor(specializationKind()),
-                        ", stub routine ", RawPointer(stub.get()), ".\n");
+                        listDump(stub->variants()), ", stub routine ", RawPointer(stub.get()),
+                        ".\n");
                 }
                 unlink(repatchBuffer);
@@ -84 +83 @@
     if (!!lastSeenCallee && !Heap::isMarked(lastSeenCallee.get()))
         lastSeenCallee.clear();
-    
-    if (callEdgeProfile) {
-        WTF::loadLoadFence();
-        callEdgeProfile->visitWeak();
-    }
 }
 

trunk/Source/JavaScriptCore/bytecode/CallLinkInfo.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2012, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2012, 2014, 2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -27 +27 @@
 #define CallLinkInfo_h
 
-#include "CallEdgeProfile.h"
-#include "ClosureCallStubRoutine.h"
 #include "CodeLocation.h"
 #include "CodeSpecializationKind.h"
@@ -34 +32 @@
 #include "JSFunction.h"
 #include "Opcode.h"
+#include "PolymorphicCallStubRoutine.h"
 #include "WriteBarrier.h"
 #include <wtf/OwnPtr.h>
@@ -83 +82 @@
     JITWriteBarrier<JSFunction> callee;
     WriteBarrier<JSFunction> lastSeenCallee;
-    RefPtr<ClosureCallStubRoutine> stub;
+    RefPtr<PolymorphicCallStubRoutine> stub;
     bool isFTL : 1;
     bool hasSeenShouldRepatch : 1;
@@ -89 +88 @@
     unsigned callType : 5; // CallType
     unsigned calleeGPR : 8;
-    unsigned slowPathCount;
+    uint32_t slowPathCount;
     CodeOrigin codeOrigin;
-    OwnPtr<CallEdgeProfile> callEdgeProfile;
 
     bool isLinked() { return stub || callee; }

trunk/Source/JavaScriptCore/bytecode/CallLinkStatus.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2012, 2013, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2012-2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -48 +48 @@
     }
     
-    m_edges.append(CallEdge(CallVariant(value.asCell()), 1));
+    m_variants.append(CallVariant(value.asCell()));
 }
 
@@ -130 +130 @@
     UNUSED_PARAM(profiledBlock);
     
-    if (Options::callStatusShouldUseCallEdgeProfile()) {
-        // Always trust the call edge profile over anything else since this has precise counts.
-        // It can make the best possible decision because it never "forgets" what happened for any
-        // call, with the exception of fading out the counts of old calls (for example if the
-        // counter type is 16-bit then calls that happened more than 2^16 calls ago are given half
-        // weight, and this compounds for every 2^15 [sic] calls after that). The combination of
-        // high fidelity for recent calls and fading for older calls makes this the most useful
-        // mechamism of choosing how to optimize future calls.
-        CallEdgeProfile* edgeProfile = callLinkInfo.callEdgeProfile.get();
-        WTF::loadLoadFence();
-        if (edgeProfile) {
-            CallLinkStatus result = computeFromCallEdgeProfile(edgeProfile);
-            if (!!result)
-                return result;
-        }
-    }
-    
     return computeFromCallLinkInfo(locker, callLinkInfo);
 }
@@ -166 +149 @@
     // is probably OK for now.
     
+    // PolymorphicCallStubRoutine is a GCAwareJITStubRoutine, so if non-null, it will stay alive
+    // until next GC even if the CallLinkInfo is concurrently cleared. Also, the variants list is
+    // never mutated after the PolymorphicCallStubRoutine is instantiated. We have some conservative
+    // fencing in place to make sure that we see the variants list after construction.
+    if (PolymorphicCallStubRoutine* stub = callLinkInfo.stub.get()) {
+        WTF::loadLoadFence();
+        
+        CallEdgeList edges = stub->edges();
+        
+        // Now that we've loaded the edges list, there are no further concurrency concerns. We will
+        // just manipulate and prune this list to our liking - mostly removing entries that are too
+        // infrequent and ensuring that it's sorted in descending order of frequency.
+        
+        RELEASE_ASSERT(edges.size());
+        
+        std::sort(
+            edges.begin(), edges.end(),
+            [] (CallEdge a, CallEdge b) {
+                return a.count() > b.count();
+            });
+        RELEASE_ASSERT(edges.first().count() >= edges.last().count());
+        
+        double totalCallsToKnown = 0;
+        double totalCallsToUnknown = callLinkInfo.slowPathCount;
+        CallVariantList variants;
+        for (size_t i = 0; i < edges.size(); ++i) {
+            CallEdge edge = edges[i];
+            // If the call is at the tail of the distribution, then we don't optimize it and we
+            // treat it as if it was a call to something unknown. We define the tail as being either
+            // a call that doesn't belong to the N most frequent callees (N =
+            // maxPolymorphicCallVariantsForInlining) or that has a total call count that is too
+            // small.
+            if (i >= Options::maxPolymorphicCallVariantsForInlining()
+                || edge.count() < Options::frequentCallThreshold())
+                totalCallsToUnknown += edge.count();
+            else {
+                totalCallsToKnown += edge.count();
+                variants.append(edge.callee());
+            }
+        }
+        
+        // Bail if we didn't find any calls that qualified.
+        RELEASE_ASSERT(!!totalCallsToKnown == !!variants.size());
+        if (variants.isEmpty())
+            return takesSlowPath();
+        
+        // We require that the distribution of callees is skewed towards a handful of common ones.
+        if (totalCallsToKnown / totalCallsToUnknown < Options::minimumCallToKnownRate())
+            return takesSlowPath();
+        
+        RELEASE_ASSERT(totalCallsToKnown);
+        RELEASE_ASSERT(variants.size());
+        
+        CallLinkStatus result;
+        result.m_variants = variants;
+        result.m_couldTakeSlowPath = !!totalCallsToUnknown;
+        return result;
+    }
+    
     if (callLinkInfo.slowPathCount >= Options::couldTakeSlowCaseMinimumCount())
         return takesSlowPath();
-    
-    if (ClosureCallStubRoutine* stub = callLinkInfo.stub.get())
-        return CallLinkStatus(stub->executable());
     
     JSFunction* target = callLinkInfo.lastSeenCallee.get();
@@ -180 +219 @@
 
     return CallLinkStatus(target);
-}
-
-CallLinkStatus CallLinkStatus::computeFromCallEdgeProfile(CallEdgeProfile* edgeProfile)
-{
-    // In cases where the call edge profile saw nothing, use the CallLinkInfo instead.
-    if (!edgeProfile->totalCalls())
-        return CallLinkStatus();
-    
-    // To do anything meaningful, we require that the majority of calls are to something we
-    // know how to handle.
-    unsigned numCallsToKnown = edgeProfile->numCallsToKnownCells();
-    unsigned numCallsToUnknown = edgeProfile->numCallsToNotCell() + edgeProfile->numCallsToUnknownCell();
-    
-    // We require that the majority of calls were to something that we could possibly inline.
-    if (numCallsToKnown <= numCallsToUnknown)
-        return takesSlowPath();
-    
-    // We require that the number of such calls is greater than some minimal threshold, so that we
-    // avoid inlining completely cold calls.
-    if (numCallsToKnown < Options::frequentCallThreshold())
-        return takesSlowPath();
-    
-    CallLinkStatus result;
-    result.m_edges = edgeProfile->callEdges();
-    result.m_couldTakeSlowPath = !!numCallsToUnknown;
-    result.m_canTrustCounts = true;
-    
-    return result;
 }
 
@@ -283 +294 @@
 bool CallLinkStatus::isClosureCall() const
 {
-    for (unsigned i = m_edges.size(); i--;) {
-        if (m_edges[i].callee().isClosureCall())
+    for (unsigned i = m_variants.size(); i--;) {
+        if (m_variants[i].isClosureCall())
             return true;
     }
@@ -292 +303 @@
 void CallLinkStatus::makeClosureCall()
 {
-    ASSERT(!m_isProved);
-    for (unsigned i = m_edges.size(); i--;)
-        m_edges[i] = m_edges[i].despecifiedClosure();
-    
-    if (!ASSERT_DISABLED) {
-        // Doing this should not have created duplicates, because the CallEdgeProfile
-        // should despecify closures if doing so would reduce the number of known callees.
-        for (unsigned i = 0; i < m_edges.size(); ++i) {
-            for (unsigned j = i + 1; j < m_edges.size(); ++j)
-                ASSERT(m_edges[i].callee() != m_edges[j].callee());
-        }
-    }
+    m_variants = despecifiedVariantList(m_variants);
 }
 
@@ -321 +321 @@
         out.print(comma, "Could Take Slow Path");
     
-    out.print(listDump(m_edges));
+    if (!m_variants.isEmpty())
+        out.print(comma, listDump(m_variants));
 }
 

trunk/Source/JavaScriptCore/bytecode/CallLinkStatus.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2012, 2013, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2012-2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -28 +28 @@
 
 #include "CallLinkInfo.h"
+#include "CallVariant.h"
 #include "CodeOrigin.h"
 #include "CodeSpecializationKind.h"
@@ -49 +50 @@
         : m_couldTakeSlowPath(false)
         , m_isProved(false)
-        , m_canTrustCounts(false)
     {
     }
@@ -63 +63 @@
     
     CallLinkStatus(CallVariant variant)
-        : m_edges(1, CallEdge(variant, 1))
+        : m_variants(1, variant)
         , m_couldTakeSlowPath(false)
         , m_isProved(false)
-        , m_canTrustCounts(false)
     {
     }
@@ -110 +109 @@
         CodeBlock*, CodeOrigin, const CallLinkInfoMap&, const ContextMap&);
     
-    bool isSet() const { return !m_edges.isEmpty() || m_couldTakeSlowPath; }
+    bool isSet() const { return !m_variants.isEmpty() || m_couldTakeSlowPath; }
     
     bool operator!() const { return !isSet(); }
@@ -116 +115 @@
     bool couldTakeSlowPath() const { return m_couldTakeSlowPath; }
     
-    CallEdgeList edges() const { return m_edges; }
-    unsigned size() const { return m_edges.size(); }
-    CallEdge at(unsigned i) const { return m_edges[i]; }
-    CallEdge operator[](unsigned i) const { return at(i); }
+    CallVariantList variants() const { return m_variants; }
+    unsigned size() const { return m_variants.size(); }
+    CallVariant at(unsigned i) const { return m_variants[i]; }
+    CallVariant operator[](unsigned i) const { return at(i); }
     bool isProved() const { return m_isProved; }
-    bool canOptimize() const { return !m_edges.isEmpty(); }
-    bool canTrustCounts() const { return m_canTrustCounts; }
+    bool canOptimize() const { return !m_variants.isEmpty(); }
     
     bool isClosureCall() const; // Returns true if any callee is a closure call.
@@ -133 +131 @@
     static CallLinkStatus computeFromLLInt(const ConcurrentJITLocker&, CodeBlock*, unsigned bytecodeIndex);
 #if ENABLE(JIT)
-    static CallLinkStatus computeFromCallEdgeProfile(CallEdgeProfile*);
     static CallLinkStatus computeFromCallLinkInfo(
         const ConcurrentJITLocker&, CallLinkInfo&);
 #endif
     
-    CallEdgeList m_edges;
+    CallVariantList m_variants;
     bool m_couldTakeSlowPath;
     bool m_isProved;
-    bool m_canTrustCounts;
 };
 

trunk/Source/JavaScriptCore/bytecode/CallVariant.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -51 +51 @@
 }
 
+CallVariantList variantListWithVariant(const CallVariantList& list, CallVariant variantToAdd)
+{
+    ASSERT(variantToAdd);
+    CallVariantList result;
+    for (CallVariant variant : list) {
+        ASSERT(variant);
+        if (!!variantToAdd) {
+            if (variant == variantToAdd)
+                variantToAdd = CallVariant();
+            else if (variant.despecifiedClosure() == variantToAdd.despecifiedClosure()) {
+                variant = variant.despecifiedClosure();
+                variantToAdd = CallVariant();
+            }
+        }
+        result.append(variant);
+    }
+    if (!!variantToAdd)
+        result.append(variantToAdd);
+    return result;
+}
+
+CallVariantList despecifiedVariantList(const CallVariantList& list)
+{
+    CallVariantList result;
+    for (CallVariant variant : list)
+        result = variantListWithVariant(result, variant.despecifiedClosure());
+    return result;
+}
+
 } // namespace JSC
 

trunk/Source/JavaScriptCore/bytecode/CallVariant.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2014, 2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -57 +57 @@
 //
 // This class serves as a kind of union over these four things. It does so by just holding a
-// JSCell*. We determine which of the modes its in by doing type checks on the cell. Note that there
-// is no lifecycle management for the cell because this class is always used in contexts where we
-// either do custom weak reference logic over instances of this class (see CallEdgeProfile), or we
-// are inside the compiler and we assume that the compiler runs in between collections and so can
-// touch the heap without notifying anyone.
+// JSCell*. We determine which of the modes its in by doing type checks on the cell. Note that we
+// cannot use WriteBarrier<> here because this gets used inside the compiler.
 
 class CallVariant {
@@ -182 +179 @@
 typedef Vector<CallVariant, 1> CallVariantList;
 
+// Returns a new variant list by attempting to either append the given variant or merge it with one
+// of the variants we already have by despecifying closures.
+CallVariantList variantListWithVariant(const CallVariantList&, CallVariant);
+
+// Returns a new list where every element is despecified, and the list is deduplicated.
+CallVariantList despecifiedVariantList(const CallVariantList&);
+
 } // namespace JSC
 

trunk/Source/JavaScriptCore/bytecode/CodeBlock.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2008, 2009, 2010, 2012, 2013, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2008-2010, 2012-2015 Apple Inc. All rights reserved.
  * Copyright (C) 2008 Cameron Zwarich <[email protected]>
  *
@@ -2182 +2182 @@
     while (m_incomingCalls.begin() != m_incomingCalls.end())
         m_incomingCalls.begin()->remove();
+    while (m_incomingPolymorphicCalls.begin() != m_incomingPolymorphicCalls.end())
+        m_incomingPolymorphicCalls.begin()->remove();
     
     // Note that our outgoing calls will be removed from other CodeBlocks'
@@ -3048 +3050 @@
     m_incomingCalls.push(incoming);
 }
+
+void CodeBlock::linkIncomingPolymorphicCall(ExecState* callerFrame, PolymorphicCallNode* incoming)
+{
+    noticeIncomingCall(callerFrame);
+    m_incomingPolymorphicCalls.push(incoming);
+}
 #endif // ENABLE(JIT)
 
@@ -3055 +3063 @@
         m_incomingLLIntCalls.begin()->unlink();
 #if ENABLE(JIT)
-    if (m_incomingCalls.isEmpty())
+    if (m_incomingCalls.isEmpty() && m_incomingPolymorphicCalls.isEmpty())
         return;
     RepatchBuffer repatchBuffer(this);
     while (m_incomingCalls.begin() != m_incomingCalls.end())
         m_incomingCalls.begin()->unlink(repatchBuffer);
+    while (m_incomingPolymorphicCalls.begin() != m_incomingPolymorphicCalls.end())
+        m_incomingPolymorphicCalls.begin()->unlink(repatchBuffer);
 #endif // ENABLE(JIT)
 }
@@ -3253 +3263 @@
     
     if (Options::verboseCallLink())
-        dataLog("Noticing call link from ", *callerCodeBlock, " to ", *this, "\n");
-    
+        dataLog("Noticing call link from ", pointerDump(callerCodeBlock), " to ", *this, "\n");
+    
+#if ENABLE(DFG_JIT)
     if (!m_shouldAlwaysBeInlined)
         return;
-
-#if ENABLE(DFG_JIT)
+    
+    if (!callerCodeBlock) {
+        m_shouldAlwaysBeInlined = false;
+        if (Options::verboseCallLink())
+            dataLog("    Clearing SABI because caller is native.\n");
+        return;
+    }
+
     if (!hasBaselineJITProfiling())
         return;
@@ -3286 +3303 @@
     }
     
+    if (JITCode::isOptimizingJIT(callerCodeBlock->jitType())) {
+        m_shouldAlwaysBeInlined = false;
+        if (Options::verboseCallLink())
+            dataLog("    Clearing SABI bcause caller was already optimized.\n");
+        return;
+    }
+    
     if (callerCodeBlock->codeType() != FunctionCode) {
         // If the caller is either eval or global code, assume that that won't be
@@ -3306 +3330 @@
         return;
     }
-
-    RELEASE_ASSERT(callerCodeBlock->m_capabilityLevelState != DFG::CapabilityLevelNotSet);
+    
+    if (callerCodeBlock->m_capabilityLevelState == DFG::CapabilityLevelNotSet) {
+        dataLog("In call from ", *callerCodeBlock, " ", callerFrame->codeOrigin(), " to ", *this, ": caller's DFG capability level is not set.\n");
+        CRASH();
+    }
     
     if (canCompile(callerCodeBlock->m_capabilityLevelState))

trunk/Source/JavaScriptCore/bytecode/CodeBlock.h

@@ -230 +230 @@
         
     void linkIncomingCall(ExecState* callerFrame, CallLinkInfo*);
-        
-    bool isIncomingCallAlreadyLinked(CallLinkInfo* incoming)
-    {
-        return m_incomingCalls.isOnList(incoming);
-    }
+    void linkIncomingPolymorphicCall(ExecState* callerFrame, PolymorphicCallNode*);
 #endif // ENABLE(JIT)
 
@@ -1078 +1074 @@
     Bag<CallLinkInfo> m_callLinkInfos;
     SentinelLinkedList<CallLinkInfo, BasicRawSentinelNode<CallLinkInfo>> m_incomingCalls;
+    SentinelLinkedList<PolymorphicCallNode, BasicRawSentinelNode<PolymorphicCallNode>> m_incomingPolymorphicCalls;
 #endif
     std::unique_ptr<CompactJITCodeMap> m_jitCodeMap;

trunk/Source/JavaScriptCore/dfg/DFGAbstractInterpreterInlines.h

@@ -1969 +1969 @@
         break;
 
-    case ProfiledCall:
-    case ProfiledConstruct:
-        if (forNode(m_graph.varArgChild(node, 0)).m_value)
-            m_state.setFoundConstants(true);
-        clobberWorld(node->origin.semantic, clobberLimit);
-        forNode(node).makeHeapTop();
-        break;
-
     case ForceOSRExit:
     case CheckBadCell:

trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

@@ -1 +1 @@
  /*
- * Copyright (C) 2011, 2012, 2013, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2011-2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -676 +676 @@
             m_parameterSlots = parameterSlots;
 
-        int dummyThisArgument = op == Call || op == NativeCall || op == ProfiledCall ? 0 : 1;
+        int dummyThisArgument = op == Call || op == NativeCall ? 0 : 1;
         for (int i = 0 + dummyThisArgument; i < argCount; ++i)
             addVarArgChild(get(virtualRegisterForArgument(i, registerOffset)));
@@ -1045 +1045 @@
         callLinkStatus = CallLinkStatus(callTarget->asJSValue()).setIsProved(true);
     
-    if ((!callLinkStatus.canOptimize() || callLinkStatus.size() != 1)
-        && !isFTL(m_graph.m_plan.mode) && Options::useFTLJIT()
-        && InlineCallFrame::isNormalCall(kind)
-        && CallEdgeLog::isEnabled()
-        && Options::dfgDoesCallEdgeProfiling()) {
-        ASSERT(op == Call || op == Construct);
-        if (op == Call)
-            op = ProfiledCall;
-        else
-            op = ProfiledConstruct;
-    }
+    if (Options::verboseDFGByteCodeParsing())
+        dataLog("    Handling call at ", currentCodeOrigin(), ": ", callLinkStatus, "\n");
     
     if (!callLinkStatus.canOptimize()) {
@@ -1077 +1068 @@
 #if ENABLE(FTL_NATIVE_CALL_INLINING)
     if (isFTL(m_graph.m_plan.mode) && Options::optimizeNativeCalls() && callLinkStatus.size() == 1 && !callLinkStatus.couldTakeSlowPath()) {
-        CallVariant callee = callLinkStatus[0].callee();
+        CallVariant callee = callLinkStatus[0];
         JSFunction* function = callee.function();
         CodeSpecializationKind specializationKind = InlineCallFrame::specializationKindFor(kind);
@@ -1084 +1075 @@
             callOpInfo = OpInfo(m_graph.freeze(function));
 
-            if (op == Call || op == ProfiledCall)
+            if (op == Call)
                 op = NativeCall;
             else {
-                ASSERT(op == Construct || op == ProfiledConstruct);
+                ASSERT(op == Construct);
                 op = NativeConstruct;
             }
@@ -1427 +1418 @@
     if (!callLinkStatus.couldTakeSlowPath() && callLinkStatus.size() == 1) {
         emitFunctionChecks(
-            callLinkStatus[0].callee(), callTargetNode, registerOffset, specializationKind);
+            callLinkStatus[0], callTargetNode, registerOffset, specializationKind);
         bool result = attemptToInlineCall(
-            callTargetNode, resultOperand, callLinkStatus[0].callee(), registerOffset,
+            callTargetNode, resultOperand, callLinkStatus[0], registerOffset,
             argumentCountIncludingThis, nextOffset, kind, CallerDoesNormalLinking, prediction,
             inliningBalance);
         if (!result && !callLinkStatus.isProved())
-            undoFunctionChecks(callLinkStatus[0].callee());
+            undoFunctionChecks(callLinkStatus[0]);
         if (verbose) {
             dataLog("Done inlining (simple).\n");
@@ -1463 +1454 @@
     bool allAreDirectCalls = true;
     for (unsigned i = callLinkStatus.size(); i--;) {
-        if (callLinkStatus[i].callee().isClosureCall())
+        if (callLinkStatus[i].isClosureCall())
             allAreDirectCalls = false;
         else
@@ -1476 +1467 @@
     else {
         // FIXME: We should be able to handle this case, but it's tricky and we don't know of cases
-        // where it would be beneficial. Also, CallLinkStatus would make all callees appear like
-        // closure calls if any calls were closure calls - except for calls to internal functions.
-        // So this will only arise if some callees are internal functions and others are closures.
+        // where it would be beneficial. It might be best to handle these cases as if all calls were
+        // closure calls.
         // https://bugs.webkit.org/show_bug.cgi?id=136020
         if (verbose) {
@@ -1518 +1508 @@
     Vector<BasicBlock*> landingBlocks;
     
-    // We make force this true if we give up on inlining any of the edges.
+    // We may force this true if we give up on inlining any of the edges.
     bool couldTakeSlowPath = callLinkStatus.couldTakeSlowPath();
     
@@ -1534 +1524 @@
         
         bool inliningResult = attemptToInlineCall(
-            myCallTargetNode, resultOperand, callLinkStatus[i].callee(), registerOffset,
+            myCallTargetNode, resultOperand, callLinkStatus[i], registerOffset,
             argumentCountIncludingThis, nextOffset, kind, CallerLinksManually, prediction,
             inliningBalance);
@@ -1553 +1543 @@
         JSCell* thingToCaseOn;
         if (allAreDirectCalls)
-            thingToCaseOn = callLinkStatus[i].callee().nonExecutableCallee();
+            thingToCaseOn = callLinkStatus[i].nonExecutableCallee();
         else {
             ASSERT(allAreClosureCalls);
-            thingToCaseOn = callLinkStatus[i].callee().executable();
+            thingToCaseOn = callLinkStatus[i].executable();
         }
         data.cases.append(SwitchCase(m_graph.freeze(thingToCaseOn), block.get()));
@@ -1568 +1558 @@
 
         if (verbose)
-            dataLog("Finished inlining ", callLinkStatus[i].callee(), " at ", currentCodeOrigin(), ".\n");
+            dataLog("Finished inlining ", callLinkStatus[i], " at ", currentCodeOrigin(), ".\n");
     }
     

trunk/Source/JavaScriptCore/dfg/DFGClobberize.h

@@ -364 +364 @@
     case Call:
     case Construct:
-    case ProfiledCall:
-    case ProfiledConstruct:
     case NativeCall:
     case NativeConstruct:

trunk/Source/JavaScriptCore/dfg/DFGConstantFoldingPhase.cpp

@@ -421 +421 @@
             }
                 
-            case ProfiledCall:
-            case ProfiledConstruct: {
-                if (!m_state.forNode(m_graph.varArgChild(node, 0)).m_value)
-                    break;
-                
-                // If we were able to prove that the callee is a constant then the normal call
-                // inline cache will record this callee. This means that there is no need to do any
-                // additional profiling.
-                m_interpreter.execute(indexInBlock);
-                node->setOp(node->op() == ProfiledCall ? Call : Construct);
-                eliminated = true;
-                break;
-            }
-
             default:
                 break;

trunk/Source/JavaScriptCore/dfg/DFGDoesGC.cpp

@@ -119 +119 @@
     case NativeCall:
     case NativeConstruct:
-    case ProfiledCall:
-    case ProfiledConstruct:
     case Breakpoint:
     case ProfileWillCall:

trunk/Source/JavaScriptCore/dfg/DFGDriver.cpp

@@ -80 +80 @@
         vm.getCTIStub(linkCallThunkGenerator);
         vm.getCTIStub(linkConstructThunkGenerator);
-        vm.getCTIStub(linkClosureCallThunkGenerator);
+        vm.getCTIStub(linkPolymorphicCallThunkGenerator);
         vm.getCTIStub(virtualCallThunkGenerator);
         vm.getCTIStub(virtualConstructThunkGenerator);
@@ -86 +86 @@
         vm.getCTIStub(linkCallThatPreservesRegsThunkGenerator);
         vm.getCTIStub(linkConstructThatPreservesRegsThunkGenerator);
-        vm.getCTIStub(linkClosureCallThatPreservesRegsThunkGenerator);
+        vm.getCTIStub(linkPolymorphicCallThatPreservesRegsThunkGenerator);
         vm.getCTIStub(virtualCallThatPreservesRegsThunkGenerator);
         vm.getCTIStub(virtualConstructThatPreservesRegsThunkGenerator);
     }
     
-    if (CallEdgeLog::isEnabled())
-        vm.ensureCallEdgeLog().processLog();
-
     if (vm.typeProfiler())
         vm.typeProfilerLog()->processLogEntries(ASCIILiteral("Preparing for DFG compilation."));

trunk/Source/JavaScriptCore/dfg/DFGFixupPhase.cpp

@@ -1203 +1203 @@
         case Call:
         case Construct:
-        case ProfiledCall:
-        case ProfiledConstruct:
         case ProfileControlFlow:
         case NativeCall:

trunk/Source/JavaScriptCore/dfg/DFGNode.h

@@ -1058 +1058 @@
         case Call:
         case Construct:
-        case ProfiledCall:
-        case ProfiledConstruct:
         case NativeCall:
         case NativeConstruct:

trunk/Source/JavaScriptCore/dfg/DFGNodeType.h

@@ -218 +218 @@
     macro(Call, NodeResultJS | NodeMustGenerate | NodeHasVarArgs | NodeClobbersWorld) \
     macro(Construct, NodeResultJS | NodeMustGenerate | NodeHasVarArgs | NodeClobbersWorld) \
-    macro(ProfiledCall, NodeResultJS | NodeMustGenerate | NodeHasVarArgs | NodeClobbersWorld) \
-    macro(ProfiledConstruct, NodeResultJS | NodeMustGenerate | NodeHasVarArgs | NodeClobbersWorld) \
     macro(NativeCall, NodeResultJS | NodeMustGenerate | NodeHasVarArgs | NodeClobbersWorld) \
     macro(NativeConstruct, NodeResultJS | NodeMustGenerate | NodeHasVarArgs | NodeClobbersWorld) \

trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

@@ -1220 +1220 @@
     CodeBlock* codeBlock = exec->codeBlock();
     
+    if (codeBlock->jitType() != JITCode::DFGJIT) {
+        dataLog("Unexpected code block in DFG->FTL tier-up: ", *codeBlock, "\n");
+        RELEASE_ASSERT_NOT_REACHED();
+    }
+    
     JITCode* jitCode = codeBlock->jitCode()->dfg();
     
@@ -1238 +1243 @@
     DeferGC deferGC(vm->heap);
     CodeBlock* codeBlock = exec->codeBlock();
+    
+    if (codeBlock->jitType() != JITCode::DFGJIT) {
+        dataLog("Unexpected code block in DFG->FTL tier-up: ", *codeBlock, "\n");
+        RELEASE_ASSERT_NOT_REACHED();
+    }
     
     JITCode* jitCode = codeBlock->jitCode()->dfg();

trunk/Source/JavaScriptCore/dfg/DFGPredictionPropagationPhase.cpp

@@ -189 +189 @@
         case Call:
         case Construct:
-        case ProfiledCall:
-        case ProfiledConstruct:
         case NativeCall:
         case NativeConstruct:

trunk/Source/JavaScriptCore/dfg/DFGSafeToExecute.h

@@ -190 +190 @@
     case Call:
     case Construct:
-    case ProfiledCall:
-    case ProfiledConstruct:
     case NewObject:
     case NewArray:

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT32_64.cpp

@@ -639 +639 @@
 void SpeculativeJIT::emitCall(Node* node)
 {
-    bool isCall = node->op() == Call || node->op() == ProfiledCall;
+    bool isCall = node->op() == Call;
     if (!isCall)
-        ASSERT(node->op() == Construct || node->op() == ProfiledConstruct);
+        ASSERT(node->op() == Construct);
 
     // For constructors, the this argument is not passed but we have to make space
@@ -690 +690 @@
     CallLinkInfo* info = m_jit.codeBlock()->addCallLinkInfo();
 
-    if (node->op() == ProfiledCall || node->op() == ProfiledConstruct) {
-        m_jit.vm()->callEdgeLog->emitLogCode(
-            m_jit, info->callEdgeProfile, callee.jsValueRegs());
-    }
-    
     slowPath.append(branchNotCell(callee.jsValueRegs()));
     slowPath.append(m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleePayloadGPR, targetToCheck));
@@ -4170 +4165 @@
     case Call:
     case Construct:
-    case ProfiledCall:
-    case ProfiledConstruct:
         emitCall(node);
         break;

trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp

@@ -625 +625 @@
 void SpeculativeJIT::emitCall(Node* node)
 {
-    bool isCall = node->op() == Call || node->op() == ProfiledCall;
+    bool isCall = node->op() == Call;
     if (!isCall)
-        DFG_ASSERT(m_jit.graph(), node, node->op() == Construct || node->op() == ProfiledConstruct);
+        DFG_ASSERT(m_jit.graph(), node, node->op() == Construct);
     
     // For constructors, the this argument is not passed but we have to make space
@@ -670 +670 @@
     CallLinkInfo* callLinkInfo = m_jit.codeBlock()->addCallLinkInfo();
     
-    if (node->op() == ProfiledCall || node->op() == ProfiledConstruct) {
-        m_jit.vm()->callEdgeLog->emitLogCode(
-            m_jit, callLinkInfo->callEdgeProfile, JSValueRegs(calleeGPR));
-    }
-
     slowPath = m_jit.branchPtrWithPatch(MacroAssembler::NotEqual, calleeGPR, targetToCheck, MacroAssembler::TrustedImmPtr(0));
 
@@ -4237 +4232 @@
     case Call:
     case Construct:
-    case ProfiledCall:
-    case ProfiledConstruct:
         emitCall(node);
         break;

trunk/Source/JavaScriptCore/dfg/DFGTierUpCheckInjectionPhase.cpp

@@ -51 +51 @@
             return false;
         
-        if (m_graph.m_profiledBlock->m_didFailFTLCompilation) {
-            removeFTLProfiling();
+        if (m_graph.m_profiledBlock->m_didFailFTLCompilation)
             return false;
-        }
         
 #if ENABLE(FTL_JIT)
         FTL::CapabilityLevel level = FTL::canCompile(m_graph);
-        if (level == FTL::CannotCompile) {
-            removeFTLProfiling();
+        if (level == FTL::CannotCompile)
             return false;
-        }
         
         if (!Options::enableOSREntryToFTL())
@@ -123 +119 @@
 #endif // ENABLE(FTL_JIT)
     }
-
-private:
-    void removeFTLProfiling()
-    {
-        for (BlockIndex blockIndex = m_graph.numBlocks(); blockIndex--;) {
-            BasicBlock* block = m_graph.block(blockIndex);
-            if (!block)
-                continue;
-            
-            for (unsigned nodeIndex = 0; nodeIndex < block->size(); ++nodeIndex) {
-                Node* node = block->at(nodeIndex);
-                switch (node->op()) {
-                case ProfiledCall:
-                    node->setOp(Call);
-                    break;
-                    
-                case ProfiledConstruct:
-                    node->setOp(Construct);
-                    break;
-                    
-                default:
-                    break;
-                }
-            }
-        }
-    }
 };
 

trunk/Source/JavaScriptCore/ftl/FTLCapabilities.cpp

@@ -177 +177 @@
         // These are OK.
         break;
-    case ProfiledCall:
-    case ProfiledConstruct:
-        // These are OK not because the FTL can support them, but because if the DFG sees one of
-        // these then the FTL will see a normal Call/Construct.
-        break;
     case Identity:
         // No backend handles this because it will be optimized out. But we may check

trunk/Source/JavaScriptCore/heap/Heap.cpp

@@ -994 +994 @@
     }
     
-    if (vm()->callEdgeLog) {
-        DeferGCForAWhile awhile(*this);
-        vm()->callEdgeLog->processLog();
-    }
-    
     RELEASE_ASSERT(!m_deferralDepth);
     ASSERT(vm()->currentThreadIsHoldingAPILock());

trunk/Source/JavaScriptCore/jit/BinarySwitch.h

@@ -55 +55 @@
 //     unsigned index = switch.caseIndex(); // index into casesVector, above
 //     ... // generate code for this case
+//     ... = jit.jump(); // you have to jump out yourself; falling through causes undefined behavior
 // }
 // switch.fallThrough().link(&jit);

trunk/Source/JavaScriptCore/jit/JITCall.cpp

@@ -215 +215 @@
     CallLinkInfo* info = m_codeBlock->addCallLinkInfo();
 
-    if (CallEdgeLog::isEnabled() && shouldEmitProfiling()
-        && Options::baselineDoesCallEdgeProfiling())
-        m_vm->ensureCallEdgeLog().emitLogCode(*this, info->callEdgeProfile, JSValueRegs(regT0));
-
     if (opcodeID == op_call_eval) {
         compileCallEval(instruction);

trunk/Source/JavaScriptCore/jit/JITCall32_64.cpp

@@ -301 +301 @@
     CallLinkInfo* info = m_codeBlock->addCallLinkInfo();
 
-    if (CallEdgeLog::isEnabled() && shouldEmitProfiling()
-        && Options::baselineDoesCallEdgeProfiling()) {
-        m_vm->ensureCallEdgeLog().emitLogCode(
-            *this, info->callEdgeProfile, JSValueRegs(regT1, regT0));
-    }
-
     if (opcodeID == op_call_eval) {
         compileCallEval(instruction);

trunk/Source/JavaScriptCore/jit/JITOperations.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2013, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2013-2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -777 +777 @@
 }
 
-static bool attemptToOptimizeClosureCall(
-    ExecState* execCallee, RegisterPreservationMode registers, JSCell* calleeAsFunctionCell,
-    CallLinkInfo& callLinkInfo)
-{
-    if (!calleeAsFunctionCell)
-        return false;
-    
-    JSFunction* callee = jsCast<JSFunction*>(calleeAsFunctionCell);
-    JSFunction* oldCallee = callLinkInfo.callee.get();
-    
-    if (!oldCallee || oldCallee->executable() != callee->executable())
-        return false;
-    
-    ASSERT(callee->executable()->hasJITCodeForCall());
-    MacroAssemblerCodePtr codePtr =
-        callee->executable()->generatedJITCodeForCall()->addressForCall(
-            *execCallee->callerFrame()->codeBlock()->vm(), callee->executable(),
-            ArityCheckNotRequired, registers);
-    
-    CodeBlock* codeBlock;
-    if (callee->executable()->isHostFunction())
-        codeBlock = 0;
-    else {
-        codeBlock = jsCast<FunctionExecutable*>(callee->executable())->codeBlockForCall();
-        if (execCallee->argumentCountIncludingThis() < static_cast<size_t>(codeBlock->numParameters()) || callLinkInfo.callType == CallLinkInfo::CallVarargs || callLinkInfo.callType == CallLinkInfo::ConstructVarargs)
-            return false;
-    }
-    
-    linkClosureCall(
-        execCallee, callLinkInfo, codeBlock, callee->executable(), codePtr, registers);
-    
-    return true;
-}
-
-char* JIT_OPERATION operationLinkClosureCall(ExecState* execCallee, CallLinkInfo* callLinkInfo)
+char* JIT_OPERATION operationLinkPolymorphicCall(ExecState* execCallee, CallLinkInfo* callLinkInfo)
 {
     JSCell* calleeAsFunctionCell;
     char* result = virtualForWithFunction(execCallee, CodeForCall, RegisterPreservationNotRequired, calleeAsFunctionCell);
 
-    if (!attemptToOptimizeClosureCall(execCallee, RegisterPreservationNotRequired, calleeAsFunctionCell, *callLinkInfo))
-        linkSlowFor(execCallee, *callLinkInfo, CodeForCall, RegisterPreservationNotRequired);
+    linkPolymorphicCall(execCallee, *callLinkInfo, CallVariant(calleeAsFunctionCell), RegisterPreservationNotRequired);
     
     return result;
@@ -832 +797 @@
 }
 
-char* JIT_OPERATION operationLinkClosureCallThatPreservesRegs(ExecState* execCallee, CallLinkInfo* callLinkInfo)
+char* JIT_OPERATION operationLinkPolymorphicCallThatPreservesRegs(ExecState* execCallee, CallLinkInfo* callLinkInfo)
 {
     JSCell* calleeAsFunctionCell;
     char* result = virtualForWithFunction(execCallee, CodeForCall, MustPreserveRegisters, calleeAsFunctionCell);
 
-    if (!attemptToOptimizeClosureCall(execCallee, MustPreserveRegisters, calleeAsFunctionCell, *callLinkInfo))
-        linkSlowFor(execCallee, *callLinkInfo, CodeForCall, MustPreserveRegisters);
+    linkPolymorphicCall(execCallee, *callLinkInfo, CallVariant(calleeAsFunctionCell), MustPreserveRegisters);
     
     return result;
@@ -1034 +998 @@
     
     CodeBlock* codeBlock = exec->codeBlock();
+    if (codeBlock->jitType() != JITCode::BaselineJIT) {
+        dataLog("Unexpected code block in Baseline->DFG tier-up: ", *codeBlock, "\n");
+        RELEASE_ASSERT_NOT_REACHED();
+    }
     
     if (bytecodeIndex) {

trunk/Source/JavaScriptCore/jit/JITOperations.h

@@ -247 +247 @@
 EncodedJSValue JIT_OPERATION operationCallEval(ExecState*, ExecState*) WTF_INTERNAL;
 char* JIT_OPERATION operationLinkCall(ExecState*, CallLinkInfo*) WTF_INTERNAL;
-char* JIT_OPERATION operationLinkClosureCall(ExecState*, CallLinkInfo*) WTF_INTERNAL;
+char* JIT_OPERATION operationLinkPolymorphicCall(ExecState*, CallLinkInfo*) WTF_INTERNAL;
 char* JIT_OPERATION operationVirtualCall(ExecState*, CallLinkInfo*) WTF_INTERNAL;
 char* JIT_OPERATION operationVirtualConstruct(ExecState*, CallLinkInfo*) WTF_INTERNAL;
 char* JIT_OPERATION operationLinkConstruct(ExecState*, CallLinkInfo*) WTF_INTERNAL;
 char* JIT_OPERATION operationLinkCallThatPreservesRegs(ExecState*, CallLinkInfo*) WTF_INTERNAL;
-char* JIT_OPERATION operationLinkClosureCallThatPreservesRegs(ExecState*, CallLinkInfo*) WTF_INTERNAL;
+char* JIT_OPERATION operationLinkPolymorphicCallThatPreservesRegs(ExecState*, CallLinkInfo*) WTF_INTERNAL;
 char* JIT_OPERATION operationVirtualCallThatPreservesRegs(ExecState*, CallLinkInfo*) WTF_INTERNAL;
 char* JIT_OPERATION operationVirtualConstructThatPreservesRegs(ExecState*, CallLinkInfo*) WTF_INTERNAL;
@@ -392 +392 @@
 }
 
-inline P_JITOperation_ECli operationLinkClosureCallFor(RegisterPreservationMode registers)
+inline P_JITOperation_ECli operationLinkPolymorphicCallFor(RegisterPreservationMode registers)
 {
     switch (registers) {
     case RegisterPreservationNotRequired:
-        return operationLinkClosureCall;
+        return operationLinkPolymorphicCall;
     case MustPreserveRegisters:
-        return operationLinkClosureCallThatPreservesRegs;
+        return operationLinkPolymorphicCallThatPreservesRegs;
     }
     RELEASE_ASSERT_NOT_REACHED();

trunk/Source/JavaScriptCore/jit/JITStubRoutine.h

@@ -142 +142 @@
     }
     
+    // Return true if you are still valid after. Return false if you are now invalid. If you return
+    // false, you will usually not do any clearing because the idea is that you will simply be
+    // destroyed.
     virtual bool visitWeak(RepatchBuffer&);
 

trunk/Source/JavaScriptCore/jit/JITWriteBarrier.h

@@ -32 +32 @@
 #include "SlotVisitor.h"
 #include "UnusedPointer.h"
+#include "VM.h"
 #include "WriteBarrier.h"
 

trunk/Source/JavaScriptCore/jit/Repatch.cpp

@@ -30 +30 @@
 
 #include "AccessorCallJITStubRoutine.h"
+#include "BinarySwitch.h"
 #include "CCallHelpers.h"
 #include "DFGOperations.h"
@@ -1576 +1577 @@
 
 static void linkSlowFor(
+    RepatchBuffer& repatchBuffer, VM* vm, CallLinkInfo& callLinkInfo, ThunkGenerator generator)
+{
+    repatchBuffer.relink(
+        callLinkInfo.callReturnLocation, vm->getCTIStub(generator).code());
+}
+
+static void linkSlowFor(
     RepatchBuffer& repatchBuffer, VM* vm, CallLinkInfo& callLinkInfo,
     CodeSpecializationKind kind, RegisterPreservationMode registers)
 {
-    repatchBuffer.relink(
-        callLinkInfo.callReturnLocation,
-        vm->getCTIStub(virtualThunkGeneratorFor(kind, registers)).code());
+    linkSlowFor(repatchBuffer, vm, callLinkInfo, virtualThunkGeneratorFor(kind, registers));
 }
 
@@ -1593 +1599 @@
     CodeBlock* callerCodeBlock = exec->callerFrame()->codeBlock();
 
-    // If you're being call-linked from a DFG caller then you obviously didn't get inlined.
-    if (calleeCodeBlock && JITCode::isOptimizingJIT(callerCodeBlock->jitType()))
-        calleeCodeBlock->m_shouldAlwaysBeInlined = false;
-    
     VM* vm = callerCodeBlock->vm();
     
@@ -1612 +1614 @@
     
     if (kind == CodeForCall) {
-        repatchBuffer.relink(callLinkInfo.callReturnLocation, vm->getCTIStub(linkClosureCallThunkGeneratorFor(registers)).code());
+        linkSlowFor(
+            repatchBuffer, vm, callLinkInfo, linkPolymorphicCallThunkGeneratorFor(registers));
         return;
     }
@@ -1632 +1635 @@
 }
 
-void linkClosureCall(
-    ExecState* exec, CallLinkInfo& callLinkInfo, CodeBlock* calleeCodeBlock, 
-    ExecutableBase* executable, MacroAssemblerCodePtr codePtr,
-    RegisterPreservationMode registers)
-{
-    ASSERT(!callLinkInfo.stub);
+static void revertCall(
+    RepatchBuffer& repatchBuffer, VM* vm, CallLinkInfo& callLinkInfo, ThunkGenerator generator)
+{
+    repatchBuffer.revertJumpReplacementToBranchPtrWithPatch(
+        RepatchBuffer::startOfBranchPtrWithPatchOnRegister(callLinkInfo.hotPathBegin),
+        static_cast<MacroAssembler::RegisterID>(callLinkInfo.calleeGPR), 0);
+    linkSlowFor(repatchBuffer, vm, callLinkInfo, generator);
+    callLinkInfo.hasSeenShouldRepatch = false;
+    callLinkInfo.callee.clear();
+    callLinkInfo.stub.clear();
+    if (callLinkInfo.isOnList())
+        callLinkInfo.remove();
+}
+
+void unlinkFor(
+    RepatchBuffer& repatchBuffer, CallLinkInfo& callLinkInfo,
+    CodeSpecializationKind kind, RegisterPreservationMode registers)
+{
+    if (Options::showDisassembly())
+        dataLog("Unlinking call from ", callLinkInfo.callReturnLocation, " in request from ", pointerDump(repatchBuffer.codeBlock()), "\n");
+    
+    revertCall(
+        repatchBuffer, repatchBuffer.codeBlock()->vm(), callLinkInfo,
+        linkThunkGeneratorFor(kind, registers));
+}
+
+void linkVirtualFor(
+    ExecState* exec, CallLinkInfo& callLinkInfo,
+    CodeSpecializationKind kind, RegisterPreservationMode registers)
+{
+    // FIXME: We could generate a virtual call stub here. This would lead to faster virtual calls
+    // by eliminating the branch prediction bottleneck inside the shared virtual call thunk.
     
     CodeBlock* callerCodeBlock = exec->callerFrame()->codeBlock();
     VM* vm = callerCodeBlock->vm();
+    
+    if (shouldShowDisassemblyFor(callerCodeBlock))
+        dataLog("Linking virtual call at ", *callerCodeBlock, " ", exec->callerFrame()->codeOrigin(), "\n");
+    
+    RepatchBuffer repatchBuffer(callerCodeBlock);
+    revertCall(repatchBuffer, vm, callLinkInfo, virtualThunkGeneratorFor(kind, registers));
+}
+
+namespace {
+struct CallToCodePtr {
+    CCallHelpers::Call call;
+    MacroAssemblerCodePtr codePtr;
+};
+} // annonymous namespace
+
+void linkPolymorphicCall(
+    ExecState* exec, CallLinkInfo& callLinkInfo, CallVariant newVariant,
+    RegisterPreservationMode registers)
+{
+    // Currently we can't do anything for non-function callees.
+    // https://bugs.webkit.org/show_bug.cgi?id=140685
+    if (!newVariant || !newVariant.executable()) {
+        linkVirtualFor(exec, callLinkInfo, CodeForCall, registers);
+        return;
+    }
+    
+    CodeBlock* callerCodeBlock = exec->callerFrame()->codeBlock();
+    VM* vm = callerCodeBlock->vm();
+    
+    CallVariantList list;
+    if (PolymorphicCallStubRoutine* stub = callLinkInfo.stub.get())
+        list = stub->variants();
+    else if (JSFunction* oldCallee = callLinkInfo.callee.get())
+        list = CallVariantList{ CallVariant(oldCallee) };
+    
+    list = variantListWithVariant(list, newVariant);
+
+    // If there are any closure calls then it makes sense to treat all of them as closure calls.
+    // This makes switching on callee cheaper. It also produces profiling that's easier on the DFG;
+    // the DFG doesn't really want to deal with a combination of closure and non-closure callees.
+    bool isClosureCall = false;
+    for (CallVariant variant : list)  {
+        if (variant.isClosureCall()) {
+            list = despecifiedVariantList(list);
+            isClosureCall = true;
+            break;
+        }
+    }
+    
+    Vector<PolymorphicCallCase> callCases;
+    
+    // Figure out what our cases are.
+    for (CallVariant variant : list) {
+        CodeBlock* codeBlock;
+        if (variant.executable()->isHostFunction())
+            codeBlock = nullptr;
+        else {
+            codeBlock = jsCast<FunctionExecutable*>(variant.executable())->codeBlockForCall();
+            
+            // If we cannot handle a callee, assume that it's better for this whole thing to be a
+            // virtual call.
+            if (exec->argumentCountIncludingThis() < static_cast<size_t>(codeBlock->numParameters()) || callLinkInfo.callType == CallLinkInfo::CallVarargs || callLinkInfo.callType == CallLinkInfo::ConstructVarargs) {
+                linkVirtualFor(exec, callLinkInfo, CodeForCall, registers);
+                return;
+            }
+        }
+        
+        callCases.append(PolymorphicCallCase(variant, codeBlock));
+    }
+    
+    // If we are over the limit, just use a normal virtual call.
+    unsigned maxPolymorphicCallVariantListSize;
+    if (callerCodeBlock->jitType() == JITCode::topTierJIT())
+        maxPolymorphicCallVariantListSize = Options::maxPolymorphicCallVariantListSizeForTopTier();
+    else
+        maxPolymorphicCallVariantListSize = Options::maxPolymorphicCallVariantListSize();
+    if (list.size() > maxPolymorphicCallVariantListSize) {
+        linkVirtualFor(exec, callLinkInfo, CodeForCall, registers);
+        return;
+    }
     
     GPRReg calleeGPR = static_cast<GPRReg>(callLinkInfo.calleeGPR);
@@ -1656 +1765 @@
         okArgumentCount.link(&stubJit);
     }
+    
+    GPRReg scratch = AssemblyHelpers::selectScratchGPR(calleeGPR);
+    GPRReg comparisonValueGPR;
+    
+    if (isClosureCall) {
+        // Verify that we have a function and stash the executable in scratch.
 
 #if USE(JSVALUE64)
-    // We can safely clobber everything except the calleeGPR. We can't rely on tagMaskRegister
-    // being set. So we do this the hard way.
-    GPRReg scratch = AssemblyHelpers::selectScratchGPR(calleeGPR);
-    stubJit.move(MacroAssembler::TrustedImm64(TagMask), scratch);
-    slowPath.append(stubJit.branchTest64(CCallHelpers::NonZero, calleeGPR, scratch));
+        // We can safely clobber everything except the calleeGPR. We can't rely on tagMaskRegister
+        // being set. So we do this the hard way.
+        stubJit.move(MacroAssembler::TrustedImm64(TagMask), scratch);
+        slowPath.append(stubJit.branchTest64(CCallHelpers::NonZero, calleeGPR, scratch));
 #else
-    // We would have already checked that the callee is a cell.
-#endif
-    
-    slowPath.append(
-        stubJit.branch8(
-            CCallHelpers::NotEqual,
-            CCallHelpers::Address(calleeGPR, JSCell::typeInfoTypeOffset()),
-            CCallHelpers::TrustedImm32(JSFunctionType)));
-    
-    slowPath.append(
-        stubJit.branchPtr(
-            CCallHelpers::NotEqual,
+        // We would have already checked that the callee is a cell.
+#endif
+    
+        slowPath.append(
+            stubJit.branch8(
+                CCallHelpers::NotEqual,
+                CCallHelpers::Address(calleeGPR, JSCell::typeInfoTypeOffset()),
+                CCallHelpers::TrustedImm32(JSFunctionType)));
+    
+        stubJit.loadPtr(
             CCallHelpers::Address(calleeGPR, JSFunction::offsetOfExecutable()),
-            CCallHelpers::TrustedImmPtr(executable)));
-    
-    AssemblyHelpers::Call call = stubJit.nearCall();
-    AssemblyHelpers::Jump done = stubJit.jump();
+            scratch);
+        
+        comparisonValueGPR = scratch;
+    } else
+        comparisonValueGPR = calleeGPR;
+    
+    Vector<int64_t> caseValues(callCases.size());
+    Vector<CallToCodePtr> calls(callCases.size());
+    std::unique_ptr<uint32_t[]> fastCounts;
+    
+    if (callerCodeBlock->jitType() != JITCode::topTierJIT())
+        fastCounts = std::make_unique<uint32_t[]>(callCases.size());
+    
+    for (size_t i = callCases.size(); i--;) {
+        if (fastCounts)
+            fastCounts[i] = 0;
+        
+        CallVariant variant = callCases[i].variant();
+        if (isClosureCall)
+            caseValues[i] = bitwise_cast<intptr_t>(variant.executable());
+        else
+            caseValues[i] = bitwise_cast<intptr_t>(variant.function());
+    }
+    
+    GPRReg fastCountsBaseGPR =
+        AssemblyHelpers::selectScratchGPR(calleeGPR, comparisonValueGPR, GPRInfo::regT3);
+    stubJit.move(CCallHelpers::TrustedImmPtr(fastCounts.get()), fastCountsBaseGPR);
+    
+    BinarySwitch binarySwitch(comparisonValueGPR, caseValues, BinarySwitch::IntPtr);
+    CCallHelpers::JumpList done;
+    while (binarySwitch.advance(stubJit)) {
+        size_t caseIndex = binarySwitch.caseIndex();
+        
+        CallVariant variant = callCases[caseIndex].variant();
+        
+        ASSERT(variant.executable()->hasJITCodeForCall());
+        MacroAssemblerCodePtr codePtr =
+            variant.executable()->generatedJITCodeForCall()->addressForCall(
+                *vm, variant.executable(), ArityCheckNotRequired, registers);
+        
+        if (fastCounts) {
+            stubJit.add32(
+                CCallHelpers::TrustedImm32(1),
+                CCallHelpers::Address(fastCountsBaseGPR, caseIndex * sizeof(uint32_t)));
+        }
+        calls[caseIndex].call = stubJit.nearCall();
+        calls[caseIndex].codePtr = codePtr;
+        done.append(stubJit.jump());
+    }
     
     slowPath.link(&stubJit);
+    binarySwitch.fallThrough().link(&stubJit);
     stubJit.move(calleeGPR, GPRInfo::regT0);
 #if USE(JSVALUE32_64)
@@ -1692 +1850 @@
     stubJit.restoreReturnAddressBeforeReturn(GPRInfo::regT4);
     AssemblyHelpers::Jump slow = stubJit.jump();
-    
+        
     LinkBuffer patchBuffer(*vm, stubJit, callerCodeBlock);
     
-    patchBuffer.link(call, FunctionPtr(codePtr.executableAddress()));
+    RELEASE_ASSERT(callCases.size() == calls.size());
+    for (CallToCodePtr callToCodePtr : calls) {
+        patchBuffer.link(
+            callToCodePtr.call, FunctionPtr(callToCodePtr.codePtr.executableAddress()));
+    }
     if (JITCode::isOptimizingJIT(callerCodeBlock->jitType()))
         patchBuffer.link(done, callLinkInfo.callReturnLocation.labelAtOffset(0));
     else
         patchBuffer.link(done, callLinkInfo.hotPathOther.labelAtOffset(0));
-    patchBuffer.link(slow, CodeLocationLabel(vm->getCTIStub(virtualThunkGeneratorFor(CodeForCall, registers)).code()));
-    
-    RefPtr<ClosureCallStubRoutine> stubRoutine = adoptRef(new ClosureCallStubRoutine(
+    patchBuffer.link(slow, CodeLocationLabel(vm->getCTIStub(linkPolymorphicCallThunkGeneratorFor(registers)).code()));
+    
+    RefPtr<PolymorphicCallStubRoutine> stubRoutine = adoptRef(new PolymorphicCallStubRoutine(
         FINALIZE_CODE_FOR(
             callerCodeBlock, patchBuffer,
-            ("Closure call stub for %s, return point %p, target %p (%s)",
+            ("Polymorphic call stub for %s, return point %p, targets %s",
                 toCString(*callerCodeBlock).data(), callLinkInfo.callReturnLocation.labelAtOffset(0).executableAddress(),
-                codePtr.executableAddress(), toCString(pointerDump(calleeCodeBlock)).data())),
-        *vm, callerCodeBlock->ownerExecutable(), executable));
+                toCString(listDump(callCases)).data())),
+        *vm, callerCodeBlock->ownerExecutable(), exec->callerFrame(), callLinkInfo, callCases,
+        WTF::move(fastCounts)));
     
     RepatchBuffer repatchBuffer(callerCodeBlock);
@@ -1715 +1878 @@
         RepatchBuffer::startOfBranchPtrWithPatchOnRegister(callLinkInfo.hotPathBegin),
         CodeLocationLabel(stubRoutine->code().code()));
+    // This is weird. The original slow path should no longer be reachable.
     linkSlowFor(repatchBuffer, vm, callLinkInfo, CodeForCall, registers);
     
+    // If there had been a previous stub routine, that one will die as soon as the GC runs and sees
+    // that it's no longer on stack.
     callLinkInfo.stub = stubRoutine.release();
     
-    ASSERT(!calleeCodeBlock || calleeCodeBlock->isIncomingCallAlreadyLinked(&callLinkInfo));
+    // The call link info no longer has a call cache apart from the jump to the polymorphic call
+    // stub.
+    if (callLinkInfo.isOnList())
+        callLinkInfo.remove();
 }
 

trunk/Source/JavaScriptCore/jit/Repatch.h

@@ -1 +1 @@
 /*
- * Copyright (C) 2011 Apple Inc. All rights reserved.
+ * Copyright (C) 2011, 2015 Apple Inc. All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
@@ -30 +30 @@
 
 #include "CCallHelpers.h"
+#include "CallVariant.h"
 #include "JITOperations.h"
 
@@ -42 +43 @@
 void linkFor(ExecState*, CallLinkInfo&, CodeBlock*, JSFunction* callee, MacroAssemblerCodePtr, CodeSpecializationKind, RegisterPreservationMode);
 void linkSlowFor(ExecState*, CallLinkInfo&, CodeSpecializationKind, RegisterPreservationMode);
-void linkClosureCall(ExecState*, CallLinkInfo&, CodeBlock*, ExecutableBase*, MacroAssemblerCodePtr, RegisterPreservationMode);
+void unlinkFor(RepatchBuffer&, CallLinkInfo&, CodeSpecializationKind, RegisterPreservationMode);
+void linkVirtualFor(ExecState*, CallLinkInfo&, CodeSpecializationKind, RegisterPreservationMode);
+void linkPolymorphicCall(ExecState*, CallLinkInfo&, CallVariant, RegisterPreservationMode);
 void resetGetByID(RepatchBuffer&, StructureStubInfo&);
 void resetPutByID(RepatchBuffer&, StructureStubInfo&);

trunk/Source/JavaScriptCore/jit/ThunkGenerators.cpp

@@ -138 +138 @@
 }
 
-static MacroAssemblerCodeRef linkClosureCallForThunkGenerator(
+static MacroAssemblerCodeRef linkPolymorphicCallForThunkGenerator(
     VM* vm, RegisterPreservationMode registers)
 {
     CCallHelpers jit(vm);
     
-    slowPathFor(jit, vm, operationLinkClosureCallFor(registers));
+    slowPathFor(jit, vm, operationLinkPolymorphicCallFor(registers));
     
     LinkBuffer patchBuffer(*vm, jit, GLOBAL_THUNK_ID);
-    return FINALIZE_CODE(patchBuffer, ("Link closure call %s slow path thunk", registers == MustPreserveRegisters ? " that preserves registers" : ""));
+    return FINALIZE_CODE(patchBuffer, ("Link polymorphic call %s slow path thunk", registers == MustPreserveRegisters ? " that preserves registers" : ""));
 }
 
 // For closure optimizations, we only include calls, since if you're using closures for
 // object construction then you're going to lose big time anyway.
-MacroAssemblerCodeRef linkClosureCallThunkGenerator(VM* vm)
-{
-    return linkClosureCallForThunkGenerator(vm, RegisterPreservationNotRequired);
-}
-
-MacroAssemblerCodeRef linkClosureCallThatPreservesRegsThunkGenerator(VM* vm)
-{
-    return linkClosureCallForThunkGenerator(vm, MustPreserveRegisters);
+MacroAssemblerCodeRef linkPolymorphicCallThunkGenerator(VM* vm)
+{
+    return linkPolymorphicCallForThunkGenerator(vm, RegisterPreservationNotRequired);
+}
+
+MacroAssemblerCodeRef linkPolymorphicCallThatPreservesRegsThunkGenerator(VM* vm)
+{
+    return linkPolymorphicCallForThunkGenerator(vm, MustPreserveRegisters);
 }
 

trunk/Source/JavaScriptCore/jit/ThunkGenerators.h

@@ -66 +66 @@
 }
 
-MacroAssemblerCodeRef linkClosureCallThunkGenerator(VM*);
-MacroAssemblerCodeRef linkClosureCallThatPreservesRegsThunkGenerator(VM*);
+MacroAssemblerCodeRef linkPolymorphicCallThunkGenerator(VM*);
+MacroAssemblerCodeRef linkPolymorphicCallThatPreservesRegsThunkGenerator(VM*);
 
-inline ThunkGenerator linkClosureCallThunkGeneratorFor(RegisterPreservationMode registers)
+inline ThunkGenerator linkPolymorphicCallThunkGeneratorFor(RegisterPreservationMode registers)
 {
     switch (registers) {
     case RegisterPreservationNotRequired:
-        return linkClosureCallThunkGenerator;
+        return linkPolymorphicCallThunkGenerator;
     case MustPreserveRegisters:
-        return linkClosureCallThatPreservesRegsThunkGenerator;
+        return linkPolymorphicCallThatPreservesRegsThunkGenerator;
     }
     RELEASE_ASSERT_NOT_REACHED();

trunk/Source/JavaScriptCore/llint/LLIntSlowPaths.cpp

@@ -333 +333 @@
     }
     default:
+        dataLog("Unexpected code block in LLInt: ", *codeBlock, "\n");
         RELEASE_ASSERT_NOT_REACHED();
         return false;

trunk/Source/JavaScriptCore/runtime/Options.h

@@ -171 +171 @@
     v(bool, enablePolymorphicAccessInlining, true) \
     v(bool, enablePolymorphicCallInlining, true) \
-    v(bool, callStatusShouldUseCallEdgeProfile, true) \
-    v(bool, callEdgeProfileReallyProcessesLog, true) \
-    v(bool, baselineDoesCallEdgeProfiling, false) \
-    v(bool, dfgDoesCallEdgeProfiling, true) \
-    v(bool, enableCallEdgeProfiling, true) \
+    v(unsigned, maxPolymorphicCallVariantListSize, 15) \
+    v(unsigned, maxPolymorphicCallVariantListSizeForTopTier, 5) \
+    v(unsigned, maxPolymorphicCallVariantsForInlining, 5) \
     v(unsigned, frequentCallThreshold, 2) \
+    v(double, minimumCallToKnownRate, 0.51) \
     v(bool, optimizeNativeCalls, false) \
     v(bool, enableObjectAllocationSinking, true) \

trunk/Source/JavaScriptCore/runtime/VM.cpp

@@ -370 +370 @@
 }
 
-CallEdgeLog& VM::ensureCallEdgeLog()
-{
-    if (!callEdgeLog)
-        callEdgeLog = std::make_unique<CallEdgeLog>();
-    return *callEdgeLog;
-}
-
 #if ENABLE(JIT)
 static ThunkGenerator thunkGeneratorForIntrinsic(Intrinsic intrinsic)
@@ -460 +453 @@
 void VM::prepareToDiscardCode()
 {
-    if (callEdgeLog)
-        callEdgeLog->processLog();
-    
 #if ENABLE(DFG_JIT)
     for (unsigned i = DFG::numberOfWorklists(); i--;) {

trunk/Source/JavaScriptCore/runtime/VM.h

@@ -74 +74 @@
 class ArityCheckFailReturnThunks;
 class BuiltinExecutables;
-class CallEdgeLog;
 class CodeBlock;
 class CodeCache;
@@ -238 +237 @@
     std::unique_ptr<DFG::LongLivedState> dfgState;
 #endif // ENABLE(DFG_JIT)
-
-    std::unique_ptr<CallEdgeLog> callEdgeLog;
-    CallEdgeLog& ensureCallEdgeLog();
 
     VMType vmType;