Changeset 209653 in webkit for trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

Timestamp:

Dec 9, 2016, 11:32:38 PM (9 years ago)

Author:

[email protected]

Message:

JSVALUE64: Pass arguments in platform argument registers when making JavaScript calls
​https://bugs.webkit.org/show_bug.cgi?id=160355

Reviewed by Filip Pizlo.

JSTests:

New microbenchmarks to measure call type performance.

• microbenchmarks/calling-computed-args.js: Added.
• microbenchmarks/calling-many-callees.js: Added.
• microbenchmarks/calling-one-callee-fixed.js: Added.
• microbenchmarks/calling-one-callee.js: Added.
• microbenchmarks/calling-poly-callees.js: Added.
• microbenchmarks/calling-poly-extra-arity-callees.js: Added.
• microbenchmarks/calling-tailcall.js: Added.
• microbenchmarks/calling-virtual-arity-fixup-callees.js: Added.
• microbenchmarks/calling-virtual-arity-fixup-stackargs.js: Added.
• microbenchmarks/calling-virtual-callees.js: Added.
• microbenchmarks/calling-virtual-extra-arity-callees.js: Added.

Source/JavaScriptCore:

This patch implements passing JavaScript function arguments in registers for 64 bit platforms.

The implemented convention follows the ABI conventions for the associated platform.
The first two arguments are the callee and argument count, the rest of the argument registers
contain "this" and following argument until all platform argument registers are exhausted.
Arguments beyond what fit in registers are placed on the stack in the same location as
before this patch.

For X86-64 non-Windows platforms, there are 6 argument registers specified in the related ABI.
ARM64 has had argument registers. This allows for 4 or 6 parameter values to be placed in
registers on these respective platforms. This patch doesn't implement passing arguments in
registers for 32 bit platform, since most platforms have at most 4 argument registers
specified and 32 bit platforms use two 32 bit registers/memory locations to store one JSValue.

The call frame on the stack in unchanged in format and the arguments that are passed in
registers use the corresponding call frame location as a spill location. Arguments can
also be passed on the stack. The LLInt, baseline JIT'ed code as well as the initial entry
from C++ code base arguments on the stack. DFG s and FTL generated code pass arguments
via registers. All callees can accept arguments either in registers or on the stack.
The callee is responsible for moving argument to its preferred location.

The multiple entry points to JavaSCript code is now handled via the JITEntryPoints class and
related code. That class now has entries for StackArgsArityCheckNotRequired,
StackArgsMustCheckArity and for platforms that support registers arguments,
RegisterArgsArityCheckNotRequired, RegisterArgsMustCheckArity as well as and additional
RegisterArgsPossibleExtraArgs entry point when extra registers argument are passed.
This last case is needed to spill those extra arguments to the corresponding call frame
slots.

• JavaScriptCore.xcodeproj/project.pbxproj:
• b3/B3ArgumentRegValue.h:
• b3/B3Validate.cpp:
• bytecode/CallLinkInfo.cpp:

(JSC::CallLinkInfo::CallLinkInfo):

• bytecode/CallLinkInfo.h:

(JSC::CallLinkInfo::setUpCall):
(JSC::CallLinkInfo::argumentsLocation):
(JSC::CallLinkInfo::argumentsInRegisters):

• bytecode/PolymorphicAccess.cpp:

(JSC::AccessCase::generateImpl):

• dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

• dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::parseBlock):

• dfg/DFGCPSRethreadingPhase.cpp:

(JSC::DFG::CPSRethreadingPhase::canonicalizeLocalsInBlock):
(JSC::DFG::CPSRethreadingPhase::specialCaseArguments):
(JSC::DFG::CPSRethreadingPhase::computeIsFlushed):

• dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

• dfg/DFGCommon.h:
• dfg/DFGDCEPhase.cpp:

(JSC::DFG::DCEPhase::run):

• dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

• dfg/DFGDriver.cpp:

(JSC::DFG::compileImpl):

• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

• dfg/DFGGenerationInfo.h:

(JSC::DFG::GenerationInfo::initArgumentRegisterValue):

• dfg/DFGGraph.cpp:

(JSC::DFG::Graph::dump):
(JSC::DFG::Graph::methodOfGettingAValueProfileFor):

• dfg/DFGGraph.h:

(JSC::DFG::Graph::needsFlushedThis):
(JSC::DFG::Graph::addImmediateShouldSpeculateInt32):

• dfg/DFGInPlaceAbstractState.cpp:

(JSC::DFG::InPlaceAbstractState::initialize):

• dfg/DFGJITCompiler.cpp:

(JSC::DFG::JITCompiler::link):
(JSC::DFG::JITCompiler::compile):
(JSC::DFG::JITCompiler::compileFunction):
(JSC::DFG::JITCompiler::compileEntry): Deleted.

• dfg/DFGJITCompiler.h:

(JSC::DFG::JITCompiler::addJSDirectCall):
(JSC::DFG::JITCompiler::JSDirectCallRecord::JSDirectCallRecord):
(JSC::DFG::JITCompiler::JSDirectCallRecord::hasSlowCall):

• dfg/DFGJITFinalizer.cpp:

(JSC::DFG::JITFinalizer::JITFinalizer):
(JSC::DFG::JITFinalizer::finalize):
(JSC::DFG::JITFinalizer::finalizeFunction):

• dfg/DFGJITFinalizer.h:
• dfg/DFGLiveCatchVariablePreservationPhase.cpp:

(JSC::DFG::LiveCatchVariablePreservationPhase::handleBlock):

• dfg/DFGMaximalFlushInsertionPhase.cpp:

(JSC::DFG::MaximalFlushInsertionPhase::treatRegularBlock):
(JSC::DFG::MaximalFlushInsertionPhase::treatRootBlock):

• dfg/DFGMayExit.cpp:
• dfg/DFGMinifiedNode.cpp:

(JSC::DFG::MinifiedNode::fromNode):

• dfg/DFGMinifiedNode.h:

(JSC::DFG::belongsInMinifiedGraph):

• dfg/DFGNode.cpp:

(JSC::DFG::Node::hasVariableAccessData):

• dfg/DFGNode.h:

(JSC::DFG::Node::accessesStack):
(JSC::DFG::Node::setVariableAccessData):
(JSC::DFG::Node::hasArgumentRegisterIndex):
(JSC::DFG::Node::argumentRegisterIndex):

• dfg/DFGNodeType.h:
• dfg/DFGOSRAvailabilityAnalysisPhase.cpp:

(JSC::DFG::LocalOSRAvailabilityCalculator::executeNode):

• dfg/DFGOSREntrypointCreationPhase.cpp:

(JSC::DFG::OSREntrypointCreationPhase::run):

• dfg/DFGPlan.cpp:

(JSC::DFG::Plan::compileInThreadImpl):

• dfg/DFGPreciseLocalClobberize.h:

(JSC::DFG::PreciseLocalClobberizeAdaptor::readTop):

• dfg/DFGPredictionInjectionPhase.cpp:

(JSC::DFG::PredictionInjectionPhase::run):

• dfg/DFGPredictionPropagationPhase.cpp:
• dfg/DFGPutStackSinkingPhase.cpp:
• dfg/DFGRegisterBank.h:

(JSC::DFG::RegisterBank::iterator::unlock):
(JSC::DFG::RegisterBank::unlockAtIndex):

• dfg/DFGSSAConversionPhase.cpp:

(JSC::DFG::SSAConversionPhase::run):

• dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

• dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::SpeculativeJIT):
(JSC::DFG::SpeculativeJIT::clearGenerationInfo):
(JSC::DFG::dumpRegisterInfo):
(JSC::DFG::SpeculativeJIT::dump):
(JSC::DFG::SpeculativeJIT::compileCurrentBlock):
(JSC::DFG::SpeculativeJIT::checkArgumentTypes):
(JSC::DFG::SpeculativeJIT::setupArgumentRegistersForEntry):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT.h:

(JSC::DFG::SpeculativeJIT::allocate):
(JSC::DFG::SpeculativeJIT::spill):
(JSC::DFG::SpeculativeJIT::generationInfoFromVirtualRegister):
(JSC::DFG::JSValueOperand::JSValueOperand):
(JSC::DFG::JSValueOperand::gprUseSpecific):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::fillJSValue):
(JSC::DFG::SpeculativeJIT::emitCall):
(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGStrengthReductionPhase.cpp:

(JSC::DFG::StrengthReductionPhase::handleNode):

• dfg/DFGThunks.cpp:

(JSC::DFG::osrEntryThunkGenerator):

• dfg/DFGVariableEventStream.cpp:

(JSC::DFG::VariableEventStream::reconstruct):

• dfg/DFGVirtualRegisterAllocationPhase.cpp:

(JSC::DFG::VirtualRegisterAllocationPhase::allocateRegister):
(JSC::DFG::VirtualRegisterAllocationPhase::run):

• ftl/FTLCapabilities.cpp:

(JSC::FTL::canCompile):

• ftl/FTLJITCode.cpp:

(JSC::FTL::JITCode::~JITCode):
(JSC::FTL::JITCode::initializeEntrypointThunk):
(JSC::FTL::JITCode::setEntryFor):
(JSC::FTL::JITCode::addressForCall):
(JSC::FTL::JITCode::executableAddressAtOffset):
(JSC::FTL::JITCode::initializeAddressForCall): Deleted.
(JSC::FTL::JITCode::initializeArityCheckEntrypoint): Deleted.

• ftl/FTLJITCode.h:
• ftl/FTLJITFinalizer.cpp:

(JSC::FTL::JITFinalizer::finalizeFunction):

• ftl/FTLLink.cpp:

(JSC::FTL::link):

• ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::lower):
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetArgumentRegister):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstruct):
(JSC::FTL::DFG::LowerDFGToB3::compileDirectCallOrConstruct):
(JSC::FTL::DFG::LowerDFGToB3::compileTailCall):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargsSpread):
(JSC::FTL::DFG::LowerDFGToB3::compileCallOrConstructVarargs):
(JSC::FTL::DFG::LowerDFGToB3::compileCallEval):

• ftl/FTLOSREntry.cpp:

(JSC::FTL::prepareOSREntry):

• ftl/FTLOutput.cpp:

(JSC::FTL::Output::argumentRegister):
(JSC::FTL::Output::argumentRegisterInt32):

• ftl/FTLOutput.h:
• interpreter/ShadowChicken.cpp:

(JSC::ShadowChicken::update):

• jit/AssemblyHelpers.cpp:

(JSC::AssemblyHelpers::emitDumbVirtualCall):

• jit/AssemblyHelpers.h:

(JSC::AssemblyHelpers::spillArgumentRegistersToFrameBeforePrologue):
(JSC::AssemblyHelpers::spillArgumentRegistersToFrame):
(JSC::AssemblyHelpers::fillArgumentRegistersFromFrameBeforePrologue):
(JSC::AssemblyHelpers::emitPutArgumentToCallFrameBeforePrologue):
(JSC::AssemblyHelpers::emitPutArgumentToCallFrame):
(JSC::AssemblyHelpers::emitGetFromCallFrameHeaderBeforePrologue):
(JSC::AssemblyHelpers::emitGetFromCallFrameArgumentBeforePrologue):
(JSC::AssemblyHelpers::emitGetPayloadFromCallFrameHeaderBeforePrologue):
(JSC::AssemblyHelpers::incrementCounter):

• jit/CachedRecovery.cpp:

(JSC::CachedRecovery::addTargetJSValueRegs):

• jit/CachedRecovery.h:

(JSC::CachedRecovery::gprTargets):
(JSC::CachedRecovery::setWantedFPR):
(JSC::CachedRecovery::wantedJSValueRegs):
(JSC::CachedRecovery::setWantedJSValueRegs): Deleted.

• jit/CallFrameShuffleData.h:
• jit/CallFrameShuffler.cpp:

(JSC::CallFrameShuffler::CallFrameShuffler):
(JSC::CallFrameShuffler::dump):
(JSC::CallFrameShuffler::tryWrites):
(JSC::CallFrameShuffler::prepareAny):

• jit/CallFrameShuffler.h:

(JSC::CallFrameShuffler::snapshot):
(JSC::CallFrameShuffler::addNew):
(JSC::CallFrameShuffler::initDangerFrontier):
(JSC::CallFrameShuffler::updateDangerFrontier):
(JSC::CallFrameShuffler::findDangerFrontierFrom):

• jit/CallFrameShuffler64.cpp:

(JSC::CallFrameShuffler::emitDisplace):

• jit/GPRInfo.h:

(JSC::JSValueRegs::operator==):
(JSC::JSValueRegs::operator!=):
(JSC::GPRInfo::toArgumentIndex):
(JSC::argumentRegisterFor):
(JSC::argumentRegisterForCallee):
(JSC::argumentRegisterForArgumentCount):
(JSC::argumentRegisterIndexForJSFunctionArgument):
(JSC::jsFunctionArgumentForArgumentRegister):
(JSC::argumentRegisterForFunctionArgument):
(JSC::numberOfRegisterArgumentsFor):

• jit/JIT.cpp:

(JSC::JIT::compileWithoutLinking):
(JSC::JIT::link):
(JSC::JIT::compileCTINativeCall): Deleted.

• jit/JIT.h:

(JSC::JIT::compileNativeCallEntryPoints):

• jit/JITCall.cpp:

(JSC::JIT::compileSetupVarargsFrame):
(JSC::JIT::compileCallEval):
(JSC::JIT::compileCallEvalSlowCase):
(JSC::JIT::compileOpCall):
(JSC::JIT::compileOpCallSlowCase):

• jit/JITCall32_64.cpp:

(JSC::JIT::compileCallEvalSlowCase):
(JSC::JIT::compileOpCall):
(JSC::JIT::compileOpCallSlowCase):

• jit/JITCode.cpp:

(JSC::JITCode::execute):
(JSC::DirectJITCode::DirectJITCode):
(JSC::DirectJITCode::initializeEntryPoints):
(JSC::DirectJITCode::addressForCall):
(JSC::NativeJITCode::addressForCall):
(JSC::DirectJITCode::initializeCodeRef): Deleted.

• jit/JITCode.h:

(JSC::JITCode::executableAddress): Deleted.

• jit/JITEntryPoints.h: Added.

(JSC::JITEntryPoints::JITEntryPoints):
(JSC::JITEntryPoints::entryFor):
(JSC::JITEntryPoints::setEntryFor):
(JSC::JITEntryPoints::offsetOfEntryFor):
(JSC::JITEntryPoints::registerEntryTypeForArgumentCount):
(JSC::JITEntryPoints::registerEntryTypeForArgumentType):
(JSC::JITEntryPoints::clearEntries):
(JSC::JITEntryPoints::operator=):
(JSC::JITEntryPointsWithRef::JITEntryPointsWithRef):
(JSC::JITEntryPointsWithRef::codeRef):
(JSC::argumentsLocationFor):
(JSC::registerEntryPointTypeFor):
(JSC::entryPointTypeFor):
(JSC::thunkEntryPointTypeFor):
(JSC::JITJSCallThunkEntryPointsWithRef::JITJSCallThunkEntryPointsWithRef):
(JSC::JITJSCallThunkEntryPointsWithRef::entryFor):
(JSC::JITJSCallThunkEntryPointsWithRef::setEntryFor):
(JSC::JITJSCallThunkEntryPointsWithRef::offsetOfEntryFor):
(JSC::JITJSCallThunkEntryPointsWithRef::clearEntries):
(JSC::JITJSCallThunkEntryPointsWithRef::codeRef):
(JSC::JITJSCallThunkEntryPointsWithRef::operator=):

• jit/JITOpcodes.cpp:

(JSC::JIT::privateCompileJITEntryNativeCall):
(JSC::JIT::privateCompileCTINativeCall): Deleted.

• jit/JITOpcodes32_64.cpp:

(JSC::JIT::privateCompileJITEntryNativeCall):
(JSC::JIT::privateCompileCTINativeCall): Deleted.

• jit/JITOperations.cpp:
• jit/JITThunks.cpp:

(JSC::JITThunks::jitEntryNativeCall):
(JSC::JITThunks::jitEntryNativeConstruct):
(JSC::JITThunks::jitEntryStub):
(JSC::JITThunks::jitCallThunkEntryStub):
(JSC::JITThunks::hostFunctionStub):
(JSC::JITThunks::ctiNativeCall): Deleted.
(JSC::JITThunks::ctiNativeConstruct): Deleted.

• jit/JITThunks.h:
• jit/JSInterfaceJIT.h:

(JSC::JSInterfaceJIT::emitJumpIfNotInt32):
(JSC::JSInterfaceJIT::emitLoadInt32):

• jit/RegisterSet.cpp:

(JSC::RegisterSet::argumentRegisters):

• jit/RegisterSet.h:
• jit/Repatch.cpp:

(JSC::linkSlowFor):
(JSC::revertCall):
(JSC::unlinkFor):
(JSC::linkVirtualFor):
(JSC::linkPolymorphicCall):

• jit/SpecializedThunkJIT.h:

(JSC::SpecializedThunkJIT::SpecializedThunkJIT):
(JSC::SpecializedThunkJIT::checkJSStringArgument):
(JSC::SpecializedThunkJIT::linkFailureHere):
(JSC::SpecializedThunkJIT::finalize):

• jit/ThunkGenerator.h:
• jit/ThunkGenerators.cpp:

(JSC::createRegisterArgumentsSpillEntry):
(JSC::slowPathFor):
(JSC::linkCallThunkGenerator):
(JSC::linkDirectCallThunkGenerator):
(JSC::linkPolymorphicCallThunkGenerator):
(JSC::virtualThunkFor):
(JSC::nativeForGenerator):
(JSC::nativeCallGenerator):
(JSC::nativeTailCallGenerator):
(JSC::nativeTailCallWithoutSavedTagsGenerator):
(JSC::nativeConstructGenerator):
(JSC::stringCharLoadRegCall):
(JSC::charCodeAtThunkGenerator):
(JSC::charAtThunkGenerator):
(JSC::fromCharCodeThunkGenerator):
(JSC::clz32ThunkGenerator):
(JSC::sqrtThunkGenerator):
(JSC::floorThunkGenerator):
(JSC::ceilThunkGenerator):
(JSC::truncThunkGenerator):
(JSC::roundThunkGenerator):
(JSC::expThunkGenerator):
(JSC::logThunkGenerator):
(JSC::absThunkGenerator):
(JSC::imulThunkGenerator):
(JSC::randomThunkGenerator):
(JSC::boundThisNoArgsFunctionCallGenerator):

• jit/ThunkGenerators.h:
• jsc.cpp:

(jscmain):

• llint/LLIntEntrypoint.cpp:

(JSC::LLInt::setFunctionEntrypoint):
(JSC::LLInt::setEvalEntrypoint):
(JSC::LLInt::setProgramEntrypoint):
(JSC::LLInt::setModuleProgramEntrypoint):

• llint/LLIntSlowPaths.cpp:

(JSC::LLInt::entryOSR):
(JSC::LLInt::setUpCall):

• llint/LLIntThunks.cpp:

(JSC::LLInt::generateThunkWithJumpTo):
(JSC::LLInt::functionForRegisterCallEntryThunkGenerator):
(JSC::LLInt::functionForStackCallEntryThunkGenerator):
(JSC::LLInt::functionForRegisterConstructEntryThunkGenerator):
(JSC::LLInt::functionForStackConstructEntryThunkGenerator):
(JSC::LLInt::functionForRegisterCallArityCheckThunkGenerator):
(JSC::LLInt::functionForStackCallArityCheckThunkGenerator):
(JSC::LLInt::functionForRegisterConstructArityCheckThunkGenerator):
(JSC::LLInt::functionForStackConstructArityCheckThunkGenerator):
(JSC::LLInt::functionForCallEntryThunkGenerator): Deleted.
(JSC::LLInt::functionForConstructEntryThunkGenerator): Deleted.
(JSC::LLInt::functionForCallArityCheckThunkGenerator): Deleted.
(JSC::LLInt::functionForConstructArityCheckThunkGenerator): Deleted.

• llint/LLIntThunks.h:
• runtime/ArityCheckMode.h:
• runtime/ExecutableBase.cpp:

(JSC::ExecutableBase::clearCode):

• runtime/ExecutableBase.h:

(JSC::ExecutableBase::entrypointFor):
(JSC::ExecutableBase::offsetOfEntryFor):
(JSC::ExecutableBase::offsetOfJITCodeWithArityCheckFor): Deleted.

• runtime/JSBoundFunction.cpp:

(JSC::boundThisNoArgsFunctionCall):

• runtime/NativeExecutable.cpp:

(JSC::NativeExecutable::finishCreation):

• runtime/ScriptExecutable.cpp:

(JSC::ScriptExecutable::installCode):

• runtime/VM.cpp:

(JSC::VM::VM):
(JSC::thunkGeneratorForIntrinsic):
(JSC::VM::clearCounters):
(JSC::VM::dumpCounters):

• runtime/VM.h:

(JSC::VM::getJITEntryStub):
(JSC::VM::getJITCallThunkEntryStub):
(JSC::VM::addressOfCounter):
(JSC::VM::counterFor):

• wasm/WasmBinding.cpp:

(JSC::Wasm::importStubGenerator):

Source/WTF:

Added a new build option ENABLE_VM_COUNTERS to enable JIT'able counters.
The default is for the option to be off.

• wtf/Platform.h:

Added ENABLE_VM_COUNTERS

File:

• trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp (modified) (3 diffs)

trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

@@ -3698 +3698 @@
     // opposed to using a value we set explicitly.
     if (m_currentBlock == m_graph.block(0) && !inlineCallFrame()) {
-        m_graph.m_arguments.resize(m_numArguments);
-        // We will emit SetArgument nodes. They don't exit, but we're at the top of an op_enter so
-        // exitOK = true.
+        m_graph.m_argumentsOnStack.resize(m_numArguments);
+        m_graph.m_argumentsForChecking.resize(m_numArguments);
+        // Create all GetArgumentRegister nodes first and then the corresponding MovHint nodes,
+        // followed by the corresponding SetLocal nodes and finally any SetArgument nodes for
+        // the remaining arguments.
+        // We do this to make the exit processing correct. We start with m_exitOK = true since
+        // GetArgumentRegister nodes can exit, even though they don't. The MovHint's technically could
+        // exit but won't. The SetLocals can exit and therefore we want all the MovHints
+        // before the first SetLocal so that the register state is consistent.
+        // We do all this processing before creating any SetArgument nodes since they are
+        // morally equivalent to the SetLocals for GetArgumentRegister nodes.
         m_exitOK = true;
-        for (unsigned argument = 0; argument < m_numArguments; ++argument) {
+        
+        unsigned numRegisterArguments = std::min(m_numArguments, NUMBER_OF_JS_FUNCTION_ARGUMENT_REGISTERS);
+
+        Vector<Node*, NUMBER_OF_JS_FUNCTION_ARGUMENT_REGISTERS> getArgumentRegisterNodes;
+
+        // First create GetArgumentRegister nodes.
+        for (unsigned argument = 0; argument < numRegisterArguments; ++argument) {
+            getArgumentRegisterNodes.append(
+                addToGraph(GetArgumentRegister, OpInfo(0),
+                    OpInfo(argumentRegisterIndexForJSFunctionArgument(argument))));
+        }
+
+        // Create all the MovHint's for the GetArgumentRegister nodes created above.
+        for (unsigned i = 0; i < getArgumentRegisterNodes.size(); ++i) {
+            Node* getArgumentRegister = getArgumentRegisterNodes[i];
+            addToGraph(MovHint, OpInfo(virtualRegisterForArgument(i).offset()), getArgumentRegister);
+            // We can't exit anymore.
+            m_exitOK = false;
+        }
+
+        // Exit is now okay, but we need to fence with an ExitOK node.
+        m_exitOK = true;
+        addToGraph(ExitOK);
+
+        // Create all the SetLocals's for the GetArgumentRegister nodes created above.
+        for (unsigned i = 0; i < getArgumentRegisterNodes.size(); ++i) {
+            Node* getArgumentRegister = getArgumentRegisterNodes[i];
+            VariableAccessData* variableAccessData = newVariableAccessData(virtualRegisterForArgument(i));
+            variableAccessData->mergeStructureCheckHoistingFailed(
+                m_inlineStackTop->m_exitProfile.hasExitSite(m_currentIndex, BadCache));
+            variableAccessData->mergeCheckArrayHoistingFailed(
+                m_inlineStackTop->m_exitProfile.hasExitSite(m_currentIndex, BadIndexingType));
+            Node* setLocal = addToGraph(SetLocal, OpInfo(variableAccessData), getArgumentRegister);
+            m_currentBlock->variablesAtTail.argument(i) = setLocal;
+            getArgumentRegister->setVariableAccessData(setLocal->variableAccessData());
+            m_graph.m_argumentsOnStack[i] = setLocal;
+            m_graph.m_argumentsForChecking[i] = getArgumentRegister;
+        }
+
+        // Finally create any SetArgument nodes.
+        for (unsigned argument = NUMBER_OF_JS_FUNCTION_ARGUMENT_REGISTERS; argument < m_numArguments; ++argument) {
             VariableAccessData* variable = newVariableAccessData(
                 virtualRegisterForArgument(argument));
@@ -3711 +3759 @@
             
             Node* setArgument = addToGraph(SetArgument, OpInfo(variable));
-            m_graph.m_arguments[argument] = setArgument;
+            m_graph.m_argumentsOnStack[argument] = setArgument;
+            m_graph.m_argumentsForChecking[argument] = setArgument;
             m_currentBlock->variablesAtTail.setArgumentFirstTime(argument, setArgument);
         }
@@ -4821 +4870 @@
             // done by the arguments object creation node as that node may not exist.
             noticeArgumentsUse();
+            Terminality terminality = handleVarargsCall(currentInstruction, TailCallForwardVarargs, CallMode::Tail);
+            // We need to insert flush nodes for our arguments after the TailCallForwardVarargs
+            // node so that they will be flushed to the stack and kept alive.
             flushForReturn();
-            Terminality terminality = handleVarargsCall(currentInstruction, TailCallForwardVarargs, CallMode::Tail);
             ASSERT_WITH_MESSAGE(m_currentInstruction == currentInstruction, "handleVarargsCall, which may have inlined the callee, trashed m_currentInstruction");
             // If the call is terminal then we should not parse any further bytecodes as the TailCall will exit the function.