Changeset 180279 in webkit for trunk/Source/JavaScriptCore/ftl/FTLCompile.cpp

Timestamp:

Feb 18, 2015, 11:55:47 AM (10 years ago)

Author:

[email protected]

Message:

DFG should really support varargs
​https://bugs.webkit.org/show_bug.cgi?id=141332

Reviewed by Oliver Hunt.

Source/JavaScriptCore:

This adds comprehensive vararg call support to the DFG and FTL compilers. Previously, if a
function had a varargs call, then it could only be compiled if that varargs call was just
forwarding arguments and we were inlining the function rather than compiling it directly. Also,
only varargs calls were dealt with; varargs constructs were not.

This lifts all of those restrictions. Every varargs call or construct can now be compiled by both
the DFG and the FTL. Those calls can also be inlined, too - provided that profiling gives us a
sensible bound on arguments list length. When we inline a varargs call, the act of loading the
varargs is now made explicit in IR. I believe that we have enough IR machinery in place that we
would be able to do the arguments forwarding optimization as an IR transformation. This patch
doesn't implement that yet, and keeps the old bytecode-based varargs argument forwarding
optimization for now.

There are three major IR features introduced in this patch:

CallVarargs/ConstructVarargs: these are like Call/Construct except that they take an arguments
array rather than a list of arguments. Currently, they splat this arguments array onto the stack
using the same basic technique as the baseline JIT has always done. Except, these nodes indicate
that we are not interested in doing the non-escaping "arguments" optimization.

CallForwardVarargs: this is a form of CallVarargs that just does the non-escaping "arguments"
optimization, aka forwarding arguments. It's somewhat lazy that this doesn't include
ConstructForwardVarargs, but the reason is that once we eliminate the lazy tear-off for
arguments, this whole thing will have to be tweaked - and for now forwarding on construct is just
not important in benchmarks. ConstructVarargs will still do forwarding, just not inlined.

LoadVarargs: loads all elements out of an array onto the stack in a manner suitable for a varargs
call. This is used only when a varargs call (or construct) was inlined. The bytecode parser will
make room on the stack for the arguments, and will use LoadVarars to put those arguments into
place.

In the future, we can consider adding strength reductions like:

• If CallVarargs/ConstructVarargs see an array of known size with known elements, turn them into Call/Construct.

• If CallVarargs/ConstructVarargs are passed an unmodified, unescaped Arguments object, then turn them into CallForwardVarargs/ConstructForwardVarargs.

• If LoadVarargs sees an array of known size, then turn it into a sequence of GetByVals and PutLocals.

• If LoadVarargs sees an unmodified, unescaped Arguments object, then turn it into something like LoadForwardVarargs.

• If CallVarargs/ConstructVarargs/LoadVarargs see the result of a splice (or other Array prototype function), then do the splice and varargs loading in one go (maybe via a new node type).

• CMakeLists.txt:
• JavaScriptCore.vcxproj/JavaScriptCore.vcxproj:
• JavaScriptCore.xcodeproj/project.pbxproj:
• assembler/MacroAssembler.h:

(JSC::MacroAssembler::rshiftPtr):
(JSC::MacroAssembler::urshiftPtr):

• assembler/MacroAssemblerARM64.h:

(JSC::MacroAssemblerARM64::urshift64):

• assembler/MacroAssemblerX86_64.h:

(JSC::MacroAssemblerX86_64::urshift64):

• assembler/X86Assembler.h:

(JSC::X86Assembler::shrq_i8r):

• bytecode/CallLinkInfo.h:

(JSC::CallLinkInfo::CallLinkInfo):

• bytecode/CallLinkStatus.cpp:

(JSC::CallLinkStatus::computeFor):
(JSC::CallLinkStatus::setProvenConstantCallee):
(JSC::CallLinkStatus::dump):

• bytecode/CallLinkStatus.h:

(JSC::CallLinkStatus::maxNumArguments):
(JSC::CallLinkStatus::setIsProved): Deleted.

• bytecode/CodeOrigin.cpp:

(WTF::printInternal):

• bytecode/CodeOrigin.h:

(JSC::InlineCallFrame::varargsKindFor):
(JSC::InlineCallFrame::specializationKindFor):
(JSC::InlineCallFrame::isVarargs):
(JSC::InlineCallFrame::isNormalCall): Deleted.

• bytecode/ExitKind.cpp:

(JSC::exitKindToString):

• bytecode/ExitKind.h:
• bytecode/ValueRecovery.cpp:

(JSC::ValueRecovery::dumpInContext):

• dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

• dfg/DFGArgumentsSimplificationPhase.cpp:

(JSC::DFG::ArgumentsSimplificationPhase::run):

• dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::flush):
(JSC::DFG::ByteCodeParser::addCall):
(JSC::DFG::ByteCodeParser::handleCall):
(JSC::DFG::ByteCodeParser::handleVarargsCall):
(JSC::DFG::ByteCodeParser::emitFunctionChecks):
(JSC::DFG::ByteCodeParser::inliningCost):
(JSC::DFG::ByteCodeParser::inlineCall):
(JSC::DFG::ByteCodeParser::attemptToInlineCall):
(JSC::DFG::ByteCodeParser::handleInlining):
(JSC::DFG::ByteCodeParser::handleMinMax):
(JSC::DFG::ByteCodeParser::handleIntrinsic):
(JSC::DFG::ByteCodeParser::handleTypedArrayConstructor):
(JSC::DFG::ByteCodeParser::handleConstantInternalFunction):
(JSC::DFG::ByteCodeParser::parseBlock):
(JSC::DFG::ByteCodeParser::removeLastNodeFromGraph): Deleted.
(JSC::DFG::ByteCodeParser::undoFunctionChecks): Deleted.

• dfg/DFGCapabilities.cpp:

(JSC::DFG::capabilityLevel):

• dfg/DFGCapabilities.h:

(JSC::DFG::functionCapabilityLevel):
(JSC::DFG::mightCompileFunctionFor):

• dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

• dfg/DFGCommon.cpp:

(WTF::printInternal):

• dfg/DFGCommon.h:

(JSC::DFG::canInline):
(JSC::DFG::leastUpperBound):

• dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

• dfg/DFGGraph.cpp:

(JSC::DFG::Graph::dump):
(JSC::DFG::Graph::dumpBlockHeader):
(JSC::DFG::Graph::isLiveInBytecode):
(JSC::DFG::Graph::valueProfileFor):
(JSC::DFG::Graph::methodOfGettingAValueProfileFor):

• dfg/DFGGraph.h:

(JSC::DFG::Graph::valueProfileFor): Deleted.
(JSC::DFG::Graph::methodOfGettingAValueProfileFor): Deleted.

• dfg/DFGJITCompiler.cpp:

(JSC::DFG::JITCompiler::compileExceptionHandlers):
(JSC::DFG::JITCompiler::link):

• dfg/DFGMayExit.cpp:

(JSC::DFG::mayExit):

• dfg/DFGNode.h:

(JSC::DFG::Node::hasCallVarargsData):
(JSC::DFG::Node::callVarargsData):
(JSC::DFG::Node::hasLoadVarargsData):
(JSC::DFG::Node::loadVarargsData):
(JSC::DFG::Node::hasHeapPrediction):

• dfg/DFGNodeType.h:
• dfg/DFGOSRAvailabilityAnalysisPhase.cpp:

(JSC::DFG::LocalOSRAvailabilityCalculator::executeNode):

• dfg/DFGOSRExitCompilerCommon.cpp:

(JSC::DFG::reifyInlinedCallFrames):

• dfg/DFGOperations.cpp:
• dfg/DFGOperations.h:
• dfg/DFGPlan.cpp:

(JSC::DFG::dumpAndVerifyGraph):
(JSC::DFG::Plan::compileInThreadImpl):

• dfg/DFGPreciseLocalClobberize.h:

(JSC::DFG::PreciseLocalClobberizeAdaptor::readTop):
(JSC::DFG::PreciseLocalClobberizeAdaptor::writeTop):

• dfg/DFGPredictionPropagationPhase.cpp:

(JSC::DFG::PredictionPropagationPhase::propagate):

• dfg/DFGSSAConversionPhase.cpp:
• dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

• dfg/DFGSpeculativeJIT.h:

(JSC::DFG::SpeculativeJIT::isFlushed):
(JSC::DFG::SpeculativeJIT::callOperation):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGStackLayoutPhase.cpp:

(JSC::DFG::StackLayoutPhase::run):
(JSC::DFG::StackLayoutPhase::assign):

• dfg/DFGStrengthReductionPhase.cpp:

(JSC::DFG::StrengthReductionPhase::handleNode):

• dfg/DFGTypeCheckHoistingPhase.cpp:

(JSC::DFG::TypeCheckHoistingPhase::run):

• dfg/DFGValidate.cpp:

(JSC::DFG::Validate::validateCPS):

• ftl/FTLAbbreviations.h:

(JSC::FTL::functionType):
(JSC::FTL::buildCall):

• ftl/FTLCapabilities.cpp:

(JSC::FTL::canCompile):

• ftl/FTLCompile.cpp:

(JSC::FTL::mmAllocateDataSection):

• ftl/FTLInlineCacheSize.cpp:

(JSC::FTL::sizeOfCall):
(JSC::FTL::sizeOfCallVarargs):
(JSC::FTL::sizeOfCallForwardVarargs):
(JSC::FTL::sizeOfConstructVarargs):
(JSC::FTL::sizeOfIn):
(JSC::FTL::sizeOfICFor):
(JSC::FTL::sizeOfCheckIn): Deleted.

• ftl/FTLInlineCacheSize.h:
• ftl/FTLIntrinsicRepository.h:
• ftl/FTLJSCall.cpp:

(JSC::FTL::JSCall::JSCall):

• ftl/FTLJSCallBase.cpp:
• ftl/FTLJSCallBase.h:
• ftl/FTLJSCallVarargs.cpp: Added.

(JSC::FTL::JSCallVarargs::JSCallVarargs):
(JSC::FTL::JSCallVarargs::numSpillSlotsNeeded):
(JSC::FTL::JSCallVarargs::emit):
(JSC::FTL::JSCallVarargs::link):

• ftl/FTLJSCallVarargs.h: Added.

(JSC::FTL::JSCallVarargs::node):
(JSC::FTL::JSCallVarargs::stackmapID):
(JSC::FTL::JSCallVarargs::operator<):

• ftl/FTLLowerDFGToLLVM.cpp:

(JSC::FTL::LowerDFGToLLVM::lower):
(JSC::FTL::LowerDFGToLLVM::compileNode):
(JSC::FTL::LowerDFGToLLVM::compileGetMyArgumentsLength):
(JSC::FTL::LowerDFGToLLVM::compileGetMyArgumentByVal):
(JSC::FTL::LowerDFGToLLVM::compileCallOrConstructVarargs):
(JSC::FTL::LowerDFGToLLVM::compileLoadVarargs):
(JSC::FTL::LowerDFGToLLVM::compileIn):
(JSC::FTL::LowerDFGToLLVM::emitStoreBarrier):
(JSC::FTL::LowerDFGToLLVM::vmCall):
(JSC::FTL::LowerDFGToLLVM::vmCallNoExceptions):
(JSC::FTL::LowerDFGToLLVM::callCheck):

• ftl/FTLOutput.h:

(JSC::FTL::Output::call):

• ftl/FTLState.cpp:

(JSC::FTL::State::State):

• ftl/FTLState.h:
• interpreter/Interpreter.cpp:

(JSC::sizeOfVarargs):
(JSC::sizeFrameForVarargs):

• interpreter/Interpreter.h:
• interpreter/StackVisitor.cpp:

(JSC::StackVisitor::readInlinedFrame):

• jit/AssemblyHelpers.cpp:

(JSC::AssemblyHelpers::emitExceptionCheck):

• jit/AssemblyHelpers.h:

(JSC::AssemblyHelpers::addressFor):
(JSC::AssemblyHelpers::calleeFrameSlot):
(JSC::AssemblyHelpers::calleeArgumentSlot):
(JSC::AssemblyHelpers::calleeFrameTagSlot):
(JSC::AssemblyHelpers::calleeFramePayloadSlot):
(JSC::AssemblyHelpers::calleeArgumentTagSlot):
(JSC::AssemblyHelpers::calleeArgumentPayloadSlot):
(JSC::AssemblyHelpers::calleeFrameCallerFrame):
(JSC::AssemblyHelpers::selectScratchGPR):

• jit/CCallHelpers.h:

(JSC::CCallHelpers::setupArgumentsWithExecState):

• jit/GPRInfo.h:
• jit/JIT.cpp:

(JSC::JIT::privateCompile):

• jit/JIT.h:
• jit/JITCall.cpp:

(JSC::JIT::compileSetupVarargsFrame):
(JSC::JIT::compileOpCall):

• jit/JITCall32_64.cpp:

(JSC::JIT::compileSetupVarargsFrame):
(JSC::JIT::compileOpCall):

• jit/JITOperations.h:
• jit/SetupVarargsFrame.cpp:

(JSC::emitSetupVarargsFrameFastCase):

• jit/SetupVarargsFrame.h:
• runtime/Arguments.h:

(JSC::Arguments::create):
(JSC::Arguments::registerArraySizeInBytes):
(JSC::Arguments::finishCreation):

• runtime/Options.h:
• tests/stress/construct-varargs-inline-smaller-Foo.js: Added.

(Foo):
(bar):
(checkEqual):
(test):

• tests/stress/construct-varargs-inline.js: Added.

(Foo):
(bar):
(checkEqual):
(test):

• tests/stress/construct-varargs-no-inline.js: Added.

(Foo):
(bar):
(checkEqual):
(test):

• tests/stress/get-argument-by-val-in-inlined-varargs-call-out-of-bounds.js: Added.

(foo):
(bar):

• tests/stress/get-argument-by-val-safe-in-inlined-varargs-call-out-of-bounds.js: Added.

(foo):
(bar):

• tests/stress/get-my-argument-by-val-creates-arguments.js: Added.

(blah):
(foo):
(bar):
(checkEqual):
(test):

• tests/stress/load-varargs-then-inlined-call-exit-in-foo.js: Added.

(foo):
(bar):
(checkEqual):

• tests/stress/load-varargs-then-inlined-call-inlined.js: Added.

(foo):
(bar):
(baz):
(checkEqual):
(test):

• tests/stress/load-varargs-then-inlined-call.js: Added.

(foo):
(bar):
(checkEqual):
(test):

LayoutTests:

Adds a version of deltablue that uses rest arguments profusely. This speeds up by 20% with this
patch. I believe that the machinery that this patch puts in place will allow us to ultimately
run deltablue-varargs at the same steady-state performance as normal deltablue.

• js/regress/deltablue-varargs-expected.txt: Added.
• js/regress/deltablue-varargs.html: Added.
• js/regress/script-tests/deltablue-varargs.js: Added.

(args):
(Object.prototype.inheritsFrom):
(OrderedCollection):
(OrderedCollection.prototype.add):
(OrderedCollection.prototype.at):
(OrderedCollection.prototype.size):
(OrderedCollection.prototype.removeFirst):
(OrderedCollection.prototype.remove):
(Strength):
(Strength.stronger):
(Strength.weaker):
(Strength.weakestOf):
(Strength.strongest):
(Strength.prototype.nextWeaker):
(Constraint):
(Constraint.prototype.addConstraint):
(Constraint.prototype.satisfy):
(Constraint.prototype.destroyConstraint):
(Constraint.prototype.isInput):
(UnaryConstraint):
(UnaryConstraint.prototype.addToGraph):
(UnaryConstraint.prototype.chooseMethod):
(UnaryConstraint.prototype.isSatisfied):
(UnaryConstraint.prototype.markInputs):
(UnaryConstraint.prototype.output):
(UnaryConstraint.prototype.recalculate):
(UnaryConstraint.prototype.markUnsatisfied):
(UnaryConstraint.prototype.inputsKnown):
(UnaryConstraint.prototype.removeFromGraph):
(StayConstraint):
(StayConstraint.prototype.execute):
(EditConstraint.prototype.isInput):
(EditConstraint.prototype.execute):
(BinaryConstraint):
(BinaryConstraint.prototype.chooseMethod):
(BinaryConstraint.prototype.addToGraph):
(BinaryConstraint.prototype.isSatisfied):
(BinaryConstraint.prototype.markInputs):
(BinaryConstraint.prototype.input):
(BinaryConstraint.prototype.output):
(BinaryConstraint.prototype.recalculate):
(BinaryConstraint.prototype.markUnsatisfied):
(BinaryConstraint.prototype.inputsKnown):
(BinaryConstraint.prototype.removeFromGraph):
(ScaleConstraint):
(ScaleConstraint.prototype.addToGraph):
(ScaleConstraint.prototype.removeFromGraph):
(ScaleConstraint.prototype.markInputs):
(ScaleConstraint.prototype.execute):
(ScaleConstraint.prototype.recalculate):
(EqualityConstraint):
(EqualityConstraint.prototype.execute):
(Variable):
(Variable.prototype.addConstraint):
(Variable.prototype.removeConstraint):
(Planner):
(Planner.prototype.incrementalAdd):
(Planner.prototype.incrementalRemove):
(Planner.prototype.newMark):
(Planner.prototype.makePlan):
(Planner.prototype.extractPlanFromConstraints):
(Planner.prototype.addPropagate):
(Planner.prototype.removePropagateFrom):
(Planner.prototype.addConstraintsConsumingTo):
(Plan):
(Plan.prototype.addConstraint):
(Plan.prototype.size):
(Plan.prototype.constraintAt):
(Plan.prototype.execute):
(chainTest):
(projectionTest):
(change):
(deltaBlue):

File:

• trunk/Source/JavaScriptCore/ftl/FTLCompile.cpp (modified) (14 diffs)

trunk/Source/JavaScriptCore/ftl/FTLCompile.cpp

@@ -1 +1 @@
 /*
- * Copyright (C) 2013, 2014 Apple Inc. All rights reserved.
+ * Copyright (C) 2013-2015 Apple Inc. All rights reserved.
  * Copyright (C) 2014 Samsung Electronics
  * Copyright (C) 2014 University of Szeged
@@ -129 +129 @@
 }
 
+static int offsetOfStackRegion(StackMaps::RecordMap& recordMap, uint32_t stackmapID)
+{
+    StackMaps::RecordMap::iterator iter = recordMap.find(stackmapID);
+    RELEASE_ASSERT(iter != recordMap.end());
+    RELEASE_ASSERT(iter->value.size() == 1);
+    RELEASE_ASSERT(iter->value[0].locations.size() == 1);
+    Location capturedLocation =
+        Location::forStackmaps(nullptr, iter->value[0].locations[0]);
+    RELEASE_ASSERT(capturedLocation.kind() == Location::Register);
+    RELEASE_ASSERT(capturedLocation.gpr() == GPRInfo::callFrameRegister);
+    RELEASE_ASSERT(!(capturedLocation.addend() % sizeof(Register)));
+    return capturedLocation.addend() / sizeof(Register);
+}
+
 template<typename DescriptorType>
 void generateICFastPath(
@@ -244 +258 @@
 }
 
+template<typename CallType>
+void adjustCallICsForStackmaps(Vector<CallType>& calls, StackMaps::RecordMap& recordMap)
+{
+    // Handling JS calls is weird: we need to ensure that we sort them by the PC in LLVM
+    // generated code. That implies first pruning the ones that LLVM didn't generate.
+
+    Vector<CallType> oldCalls;
+    oldCalls.swap(calls);
+    
+    for (unsigned i = 0; i < oldCalls.size(); ++i) {
+        CallType& call = oldCalls[i];
+        
+        StackMaps::RecordMap::iterator iter = recordMap.find(call.stackmapID());
+        if (iter == recordMap.end())
+            continue;
+        
+        for (unsigned j = 0; j < iter->value.size(); ++j) {
+            CallType copy = call;
+            copy.m_instructionOffset = iter->value[j].instructionOffset;
+            calls.append(copy);
+        }
+    }
+
+    std::sort(calls.begin(), calls.end());
+}
+
 static void fixFunctionBasedOnStackMaps(
     State& state, CodeBlock* codeBlock, JITCode* jitCode, GeneratedFunction generatedFunction,
@@ -252 +292 @@
     StackMaps stackmaps = jitCode->stackmaps;
     
-    StackMaps::RecordMap::iterator iter = recordMap.find(state.capturedStackmapID);
-    RELEASE_ASSERT(iter != recordMap.end());
-    RELEASE_ASSERT(iter->value.size() == 1);
-    RELEASE_ASSERT(iter->value[0].locations.size() == 1);
-    Location capturedLocation =
-        Location::forStackmaps(&jitCode->stackmaps, iter->value[0].locations[0]);
-    RELEASE_ASSERT(capturedLocation.kind() == Location::Register);
-    RELEASE_ASSERT(capturedLocation.gpr() == GPRInfo::callFrameRegister);
-    RELEASE_ASSERT(!(capturedLocation.addend() % sizeof(Register)));
-    int32_t localsOffset = capturedLocation.addend() / sizeof(Register) + graph.m_nextMachineLocal;
+    int localsOffset =
+        offsetOfStackRegion(recordMap, state.capturedStackmapID) + graph.m_nextMachineLocal;
+    
+    int varargsSpillSlotsOffset;
+    if (state.varargsSpillSlotsStackmapID != UINT_MAX)
+        varargsSpillSlotsOffset = offsetOfStackRegion(recordMap, state.varargsSpillSlotsStackmapID);
+    else
+        varargsSpillSlotsOffset = 0;
     
     for (unsigned i = graph.m_inlineVariableData.size(); i--;) {
@@ -294 +332 @@
         // At this point it's perfectly fair to just blow away all state and restore the
         // JS JIT view of the universe.
-        checkJIT.move(MacroAssembler::TrustedImm64(TagTypeNumber), GPRInfo::tagTypeNumberRegister);
-        checkJIT.move(MacroAssembler::TrustedImm64(TagMask), GPRInfo::tagMaskRegister);
-
         checkJIT.move(MacroAssembler::TrustedImmPtr(&vm), GPRInfo::argumentGPR0);
         checkJIT.move(GPRInfo::callFrameRegister, GPRInfo::argumentGPR1);
@@ -303 +338 @@
 
         stackOverflowException = checkJIT.label();
-        checkJIT.move(MacroAssembler::TrustedImm64(TagTypeNumber), GPRInfo::tagTypeNumberRegister);
-        checkJIT.move(MacroAssembler::TrustedImm64(TagMask), GPRInfo::tagMaskRegister);
-
         checkJIT.move(MacroAssembler::TrustedImmPtr(&vm), GPRInfo::argumentGPR0);
         checkJIT.move(GPRInfo::callFrameRegister, GPRInfo::argumentGPR1);
@@ -337 +369 @@
                 dataLog("Handling OSR stackmap #", exit.m_stackmapID, " for ", exit.m_codeOrigin, "\n");
 
-            iter = recordMap.find(exit.m_stackmapID);
+            auto iter = recordMap.find(exit.m_stackmapID);
             if (iter == recordMap.end()) {
                 // It was optimized out.
@@ -376 +408 @@
                 dataLog("Handling GetById stackmap #", getById.stackmapID(), "\n");
             
-            iter = recordMap.find(getById.stackmapID());
+            auto iter = recordMap.find(getById.stackmapID());
             if (iter == recordMap.end()) {
                 // It was optimized out.
@@ -413 +445 @@
                 dataLog("Handling PutById stackmap #", putById.stackmapID(), "\n");
             
-            iter = recordMap.find(putById.stackmapID());
+            auto iter = recordMap.find(putById.stackmapID());
             if (iter == recordMap.end()) {
                 // It was optimized out.
@@ -445 +477 @@
         }
 
-
         for (unsigned i = state.checkIns.size(); i--;) {
             CheckInDescriptor& checkIn = state.checkIns[i];
@@ -452 +483 @@
                 dataLog("Handling checkIn stackmap #", checkIn.stackmapID(), "\n");
             
-            iter = recordMap.find(checkIn.stackmapID());
+            auto iter = recordMap.find(checkIn.stackmapID());
             if (iter == recordMap.end()) {
                 // It was optimized out.
@@ -481 +512 @@
             }
         }
-
         
         exceptionTarget.link(&slowPathJIT);
@@ -504 +534 @@
             generateCheckInICFastPath(
                 state, codeBlock, generatedFunction, recordMap, state.checkIns[i],
-                sizeOfCheckIn()); 
+                sizeOfIn()); 
         } 
     }
     
-    // Handling JS calls is weird: we need to ensure that we sort them by the PC in LLVM
-    // generated code. That implies first pruning the ones that LLVM didn't generate.
-    Vector<JSCall> oldCalls = state.jsCalls;
-    state.jsCalls.resize(0);
-    for (unsigned i = 0; i < oldCalls.size(); ++i) {
-        JSCall& call = oldCalls[i];
-        
-        StackMaps::RecordMap::iterator iter = recordMap.find(call.stackmapID());
-        if (iter == recordMap.end())
-            continue;
-
-        for (unsigned j = 0; j < iter->value.size(); ++j) {
-            JSCall copy = call;
-            copy.m_instructionOffset = iter->value[j].instructionOffset;
-            state.jsCalls.append(copy);
-        }
-    }
-    
-    std::sort(state.jsCalls.begin(), state.jsCalls.end());
+    adjustCallICsForStackmaps(state.jsCalls, recordMap);
     
     for (unsigned i = state.jsCalls.size(); i--;) {
@@ -548 +560 @@
     }
     
+    adjustCallICsForStackmaps(state.jsCallVarargses, recordMap);
+    
+    for (unsigned i = state.jsCallVarargses.size(); i--;) {
+        JSCallVarargs& call = state.jsCallVarargses[i];
+        
+        CCallHelpers fastPathJIT(&vm, codeBlock);
+        call.emit(fastPathJIT, graph, varargsSpillSlotsOffset);
+        
+        char* startOfIC = bitwise_cast<char*>(generatedFunction) + call.m_instructionOffset;
+        size_t sizeOfIC = sizeOfICFor(call.node());
+
+        LinkBuffer linkBuffer(vm, fastPathJIT, startOfIC, sizeOfIC);
+        if (!linkBuffer.isValid()) {
+            dataLog("Failed to insert inline cache for varargs call (specifically, ", Graph::opName(call.node()->op()), ") because we thought the size would be ", sizeOfIC, " but it ended up being ", fastPathJIT.m_assembler.codeSize(), " prior to compaction.\n");
+            RELEASE_ASSERT_NOT_REACHED();
+        }
+        
+        MacroAssembler::AssemblerType_T::fillNops(
+            startOfIC + linkBuffer.size(), sizeOfIC - linkBuffer.size());
+        
+        call.link(vm, linkBuffer, state.finalizer->handleExceptionsLinkBuffer->entrypoint());
+    }
+    
     RepatchBuffer repatchBuffer(codeBlock);
 
-    iter = recordMap.find(state.handleStackOverflowExceptionStackmapID);
+    auto iter = recordMap.find(state.handleStackOverflowExceptionStackmapID);
     // It's sort of remotely possible that we won't have an in-band exception handling
     // path, for some kinds of functions.