Changeset 253896 in webkit for trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

Timestamp:

Dec 23, 2019, 5:49:45 PM (5 years ago)

Author:

[email protected]

Message:

DFG/FTL should be able to exit to the middle of a bytecode
​https://bugs.webkit.org/show_bug.cgi?id=205232

Reviewed by Saam Barati.

JSTests:

• stress/apply-osr-exit-should-get-length-once-exceptions-occasionally.js: Added.

(expectedArgCount):
(callee):
(test):
(let.array.get length):

• stress/apply-osr-exit-should-get-length-once.js: Added.

(expectedArgCount):
(callee):
(test):
(let.array.get length):

• stress/load-varargs-then-inlined-call-and-exit-strict.js:

(checkEqual):

• stress/recursive-tail-call-with-different-argument-count.js:
• stress/rest-varargs-osr-exit-to-checkpoint.js: Added.

(foo):
(bar):

Source/JavaScriptCore:

It can be valuable to exit to the middle of a bytecode for a couple of reasons.
1) It can be used to combine bytecodes that share a majority of their operands, reducing bytecode steam size.
2) It enables creating bytecodes that are easier to reconstruct useful optimization information from.

To make exiting to the middle of a bytecode possible this patch
introduces the concept of a temporary operand. A temporary operand
is one that contains the result of effectful operations during the
process of executing a bytecode. tmp operands have no meaning when
executing in the LLInt or Baseline and are only used in the DFG to
preserve information for OSR exit. We use the term checkpoint to
refer to any point where an effectful component of a bytecode executes.
For example, in op_call_varargs there are two checkpoints the first is
before we have determined the number of variable arguments and the second
is the actual call.

When the DFG OSR exits if there are any active checkpoints inline
call stack we will emit a jit probe that allocates a side state
object keyed off the frame pointer of the bytecode whose
checkpoint needs to be finished. We need side state because we may
recursively inline several copies of the same
function. Alternatively, we could call back into ourselves after
OSR and exit again from optimized code before finishing the
checkpoint of our caller.

Another thing we need to be careful of is making sure we remove
side state as we unwind for an exception. To make sure we do this
correctly I've added an assertion to JSLock that there are no
pending checkpoint side states on the vm when releasing the lock.

A large amount of this patch is trying to remove as much code that
refers to virtual registers as an int as possible. Instead, this
patch replaces them with the VirtualRegister class. There are also
a couple of new classes/enums added to JSC:

1) There is now a class, Operand, that represents the combination
of a VirtualRegister and a temporary. This is handy in the DFG to
model OSR exit values all together. Additionally, Operands<T> has
been updated to work with respect to Operand values.

2) CallFrameSlot is now an enum class instead of a struct of
constexpr values. This lets us implicitly convert CallFrameSlots
to VirtualRegisters without allowing all ints to implicity
convert.

3) FTL::SelectPredictability is a new enum that describes to the
FTL whether or not we think a select is going to be
predictable. SelectPredictability has four options: Unpredictable,
Predictable, LeftLikely, and RightLikely. Unpredictable means we
think a branch predictor won't do a good job guessing this value
so we should compile the select to a cmov. The other options mean
we either think we are going to pick the same value every time or
there's a reasonable chance the branch predictor will be able to
guess the value.

In order to validate the correctness of this patch the various
varargs call opcodes have been reworked to use checkpoints. This
also fixed a long-standing issue where we could call length
getters twice if we OSR exit during LoadVarargs but before the
actually call.

Lastly, we have not enabled the probe-based OSR exit for a long
time in production, thus this patch removes that code since it
would be a non-trivial amount of work to get checkpoints working
with probe OSR.

• CMakeLists.txt:
• DerivedSources-input.xcfilelist:
• JavaScriptCore.xcodeproj/project.pbxproj:
• assembler/MacroAssemblerCodeRef.h:
• assembler/ProbeFrame.h:

(JSC::Probe::Frame::operand):
(JSC::Probe::Frame::setOperand):

• b3/testb3.h:

(populateWithInterestingValues):
(floatingPointOperands):

• bytecode/AccessCase.cpp:

(JSC::AccessCase::generateImpl):

• bytecode/AccessCaseSnippetParams.cpp:

(JSC::SlowPathCallGeneratorWithArguments::generateImpl):

• bytecode/BytecodeDumper.cpp:

(JSC::BytecodeDumperBase::dumpValue):
(JSC::BytecodeDumper<Block>::registerName const):
(JSC::BytecodeDumper<Block>::constantName const):
(JSC::Wasm::BytecodeDumper::constantName const):

• bytecode/BytecodeDumper.h:
• bytecode/BytecodeIndex.cpp:

(JSC::BytecodeIndex::dump const):

• bytecode/BytecodeIndex.h:

(JSC::BytecodeIndex::BytecodeIndex):
(JSC::BytecodeIndex::offset const):
(JSC::BytecodeIndex::checkpoint const):
(JSC::BytecodeIndex::asBits const):
(JSC::BytecodeIndex::hash const):
(JSC::BytecodeIndex::operator bool const):
(JSC::BytecodeIndex::pack):
(JSC::BytecodeIndex::fromBits):

• bytecode/BytecodeList.rb:
• bytecode/BytecodeLivenessAnalysis.cpp:

(JSC::enumValuesEqualAsIntegral):
(JSC::tmpLivenessForCheckpoint):

• bytecode/BytecodeLivenessAnalysis.h:
• bytecode/BytecodeLivenessAnalysisInlines.h:

(JSC::virtualRegisterIsAlwaysLive):
(JSC::virtualRegisterThatIsNotAlwaysLiveIsLive):
(JSC::virtualRegisterIsLive):
(JSC::operandIsAlwaysLive): Deleted.
(JSC::operandThatIsNotAlwaysLiveIsLive): Deleted.
(JSC::operandIsLive): Deleted.

• bytecode/CodeBlock.cpp:

(JSC::CodeBlock::finishCreation):
(JSC::CodeBlock::bytecodeIndexForExit const):
(JSC::CodeBlock::ensureCatchLivenessIsComputedForBytecodeIndexSlow):
(JSC::CodeBlock::updateAllValueProfilePredictionsAndCountLiveness):

• bytecode/CodeBlock.h:

(JSC::CodeBlock::numTmps const):
(JSC::CodeBlock::isKnownNotImmediate):
(JSC::CodeBlock::isTemporaryRegister):
(JSC::CodeBlock::constantRegister):
(JSC::CodeBlock::getConstant const):
(JSC::CodeBlock::constantSourceCodeRepresentation const):
(JSC::CodeBlock::replaceConstant):
(JSC::CodeBlock::isTemporaryRegisterIndex): Deleted.
(JSC::CodeBlock::isConstantRegisterIndex): Deleted.

• bytecode/CodeOrigin.h:
• bytecode/FullBytecodeLiveness.h:

(JSC::FullBytecodeLiveness::virtualRegisterIsLive const):
(JSC::FullBytecodeLiveness::operandIsLive const): Deleted.

• bytecode/InlineCallFrame.h:

(JSC::InlineCallFrame::InlineCallFrame):
(JSC::InlineCallFrame::setTmpOffset):
(JSC::CodeOrigin::walkUpInlineStack const):
(JSC::CodeOrigin::inlineStackContainsActiveCheckpoint const):
(JSC::remapOperand):
(JSC::unmapOperand):
(JSC::CodeOrigin::walkUpInlineStack): Deleted.

• bytecode/LazyOperandValueProfile.h:

(JSC::LazyOperandValueProfileKey::LazyOperandValueProfileKey):
(JSC::LazyOperandValueProfileKey::hash const):
(JSC::LazyOperandValueProfileKey::operand const):

• bytecode/MethodOfGettingAValueProfile.cpp:

(JSC::MethodOfGettingAValueProfile::fromLazyOperand):
(JSC::MethodOfGettingAValueProfile::emitReportValue const):
(JSC::MethodOfGettingAValueProfile::reportValue):

• bytecode/MethodOfGettingAValueProfile.h:
• bytecode/Operands.h:

(JSC::Operand::Operand):
(JSC::Operand::tmp):
(JSC::Operand::kind const):
(JSC::Operand::value const):
(JSC::Operand::virtualRegister const):
(JSC::Operand::asBits const):
(JSC::Operand::isTmp const):
(JSC::Operand::isArgument const):
(JSC::Operand::isLocal const):
(JSC::Operand::isHeader const):
(JSC::Operand::isConstant const):
(JSC::Operand::toArgument const):
(JSC::Operand::toLocal const):
(JSC::Operand::operator== const):
(JSC::Operand::isValid const):
(JSC::Operand::fromBits):
(JSC::Operands::Operands):
(JSC::Operands::numberOfLocals const):
(JSC::Operands::numberOfTmps const):
(JSC::Operands::tmpIndex const):
(JSC::Operands::argumentIndex const):
(JSC::Operands::localIndex const):
(JSC::Operands::tmp):
(JSC::Operands::tmp const):
(JSC::Operands::argument):
(JSC::Operands::argument const):
(JSC::Operands::local):
(JSC::Operands::local const):
(JSC::Operands::sizeFor const):
(JSC::Operands::atFor):
(JSC::Operands::atFor const):
(JSC::Operands::ensureLocals):
(JSC::Operands::ensureTmps):
(JSC::Operands::getForOperandIndex):
(JSC::Operands::getForOperandIndex const):
(JSC::Operands::operandIndex const):
(JSC::Operands::operand):
(JSC::Operands::operand const):
(JSC::Operands::hasOperand const):
(JSC::Operands::setOperand):
(JSC::Operands::at const):
(JSC::Operands::at):
(JSC::Operands::operator[] const):
(JSC::Operands::operator[]):
(JSC::Operands::operandForIndex const):
(JSC::Operands::operator== const):
(JSC::Operands::isArgument const): Deleted.
(JSC::Operands::isLocal const): Deleted.
(JSC::Operands::virtualRegisterForIndex const): Deleted.
(JSC::Operands::setOperandFirstTime): Deleted.

• bytecode/OperandsInlines.h:

(JSC::Operand::dump const):
(JSC::Operands<T>::dumpInContext const):
(JSC::Operands<T>::dump const):

• bytecode/UnlinkedCodeBlock.cpp:

(JSC::UnlinkedCodeBlock::UnlinkedCodeBlock):

• bytecode/UnlinkedCodeBlock.h:

(JSC::UnlinkedCodeBlock::hasCheckpoints const):
(JSC::UnlinkedCodeBlock::setHasCheckpoints):
(JSC::UnlinkedCodeBlock::constantRegister const):
(JSC::UnlinkedCodeBlock::getConstant const):
(JSC::UnlinkedCodeBlock::isConstantRegisterIndex const): Deleted.

• bytecode/ValueProfile.h:

(JSC::ValueProfileAndVirtualRegisterBuffer::ValueProfileAndVirtualRegisterBuffer):
(JSC::ValueProfileAndVirtualRegisterBuffer::~ValueProfileAndVirtualRegisterBuffer):
(JSC::ValueProfileAndOperandBuffer::ValueProfileAndOperandBuffer): Deleted.
(JSC::ValueProfileAndOperandBuffer::~ValueProfileAndOperandBuffer): Deleted.
(JSC::ValueProfileAndOperandBuffer::forEach): Deleted.

• bytecode/ValueRecovery.cpp:

(JSC::ValueRecovery::recover const):

• bytecode/ValueRecovery.h:
• bytecode/VirtualRegister.h:

(JSC::virtualRegisterIsLocal):
(JSC::virtualRegisterIsArgument):
(JSC::VirtualRegister::VirtualRegister):
(JSC::VirtualRegister::isValid const):
(JSC::VirtualRegister::isLocal const):
(JSC::VirtualRegister::isArgument const):
(JSC::VirtualRegister::isConstant const):
(JSC::VirtualRegister::toConstantIndex const):
(JSC::operandIsLocal): Deleted.
(JSC::operandIsArgument): Deleted.

• bytecompiler/BytecodeGenerator.cpp:

(JSC::BytecodeGenerator::initializeNextParameter):
(JSC::BytecodeGenerator::initializeParameters):
(JSC::BytecodeGenerator::emitEqualityOpImpl):
(JSC::BytecodeGenerator::emitCallVarargs):

• bytecompiler/BytecodeGenerator.h:

(JSC::BytecodeGenerator::setUsesCheckpoints):

• bytecompiler/RegisterID.h:

(JSC::RegisterID::setIndex):

• dfg/DFGAbstractHeap.cpp:

(JSC::DFG::AbstractHeap::Payload::dumpAsOperand const):
(JSC::DFG::AbstractHeap::dump const):

• dfg/DFGAbstractHeap.h:

(JSC::DFG::AbstractHeap::Payload::Payload):
(JSC::DFG::AbstractHeap::AbstractHeap):
(JSC::DFG::AbstractHeap::operand const):

• dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

• dfg/DFGArgumentPosition.h:

(JSC::DFG::ArgumentPosition::dump):

• dfg/DFGArgumentsEliminationPhase.cpp:
• dfg/DFGArgumentsUtilities.cpp:

(JSC::DFG::argumentsInvolveStackSlot):
(JSC::DFG::emitCodeToGetArgumentsArrayLength):

• dfg/DFGArgumentsUtilities.h:
• dfg/DFGAtTailAbstractState.h:

(JSC::DFG::AtTailAbstractState::operand):

• dfg/DFGAvailabilityMap.cpp:

(JSC::DFG::AvailabilityMap::pruneByLiveness):

• dfg/DFGAvailabilityMap.h:

(JSC::DFG::AvailabilityMap::closeStartingWithLocal):

• dfg/DFGBasicBlock.cpp:

(JSC::DFG::BasicBlock::BasicBlock):
(JSC::DFG::BasicBlock::ensureTmps):

• dfg/DFGBasicBlock.h:
• dfg/DFGBlockInsertionSet.cpp:

(JSC::DFG::BlockInsertionSet::insert):

• dfg/DFGByteCodeParser.cpp:

(JSC::DFG::ByteCodeParser::ByteCodeParser):
(JSC::DFG::ByteCodeParser::ensureTmps):
(JSC::DFG::ByteCodeParser::progressToNextCheckpoint):
(JSC::DFG::ByteCodeParser::newVariableAccessData):
(JSC::DFG::ByteCodeParser::getDirect):
(JSC::DFG::ByteCodeParser::get):
(JSC::DFG::ByteCodeParser::setDirect):
(JSC::DFG::ByteCodeParser::injectLazyOperandSpeculation):
(JSC::DFG::ByteCodeParser::getLocalOrTmp):
(JSC::DFG::ByteCodeParser::setLocalOrTmp):
(JSC::DFG::ByteCodeParser::setArgument):
(JSC::DFG::ByteCodeParser::findArgumentPositionForLocal):
(JSC::DFG::ByteCodeParser::findArgumentPosition):
(JSC::DFG::ByteCodeParser::flushImpl):
(JSC::DFG::ByteCodeParser::flushForTerminalImpl):
(JSC::DFG::ByteCodeParser::flush):
(JSC::DFG::ByteCodeParser::flushDirect):
(JSC::DFG::ByteCodeParser::addFlushOrPhantomLocal):
(JSC::DFG::ByteCodeParser::phantomLocalDirect):
(JSC::DFG::ByteCodeParser::flushForTerminal):
(JSC::DFG::ByteCodeParser::addToGraph):
(JSC::DFG::ByteCodeParser::InlineStackEntry::remapOperand const):
(JSC::DFG::ByteCodeParser::DelayedSetLocal::DelayedSetLocal):
(JSC::DFG::ByteCodeParser::DelayedSetLocal::execute):
(JSC::DFG::ByteCodeParser::allocateTargetableBlock):
(JSC::DFG::ByteCodeParser::allocateUntargetableBlock):
(JSC::DFG::ByteCodeParser::handleRecursiveTailCall):
(JSC::DFG::ByteCodeParser::inlineCall):
(JSC::DFG::ByteCodeParser::handleVarargsInlining):
(JSC::DFG::ByteCodeParser::handleInlining):
(JSC::DFG::ByteCodeParser::parseBlock):
(JSC::DFG::ByteCodeParser::InlineStackEntry::InlineStackEntry):
(JSC::DFG::ByteCodeParser::parse):
(JSC::DFG::ByteCodeParser::getLocal): Deleted.
(JSC::DFG::ByteCodeParser::setLocal): Deleted.

• dfg/DFGCFAPhase.cpp:

(JSC::DFG::CFAPhase::injectOSR):

• dfg/DFGCPSRethreadingPhase.cpp:

(JSC::DFG::CPSRethreadingPhase::run):
(JSC::DFG::CPSRethreadingPhase::canonicalizeGetLocal):
(JSC::DFG::CPSRethreadingPhase::canonicalizeFlushOrPhantomLocalFor):
(JSC::DFG::CPSRethreadingPhase::canonicalizeFlushOrPhantomLocal):
(JSC::DFG::CPSRethreadingPhase::canonicalizeSet):
(JSC::DFG::CPSRethreadingPhase::canonicalizeLocalsInBlock):
(JSC::DFG::CPSRethreadingPhase::propagatePhis):
(JSC::DFG::CPSRethreadingPhase::phiStackFor):

• dfg/DFGCSEPhase.cpp:
• dfg/DFGCapabilities.cpp:

(JSC::DFG::capabilityLevel):

• dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

• dfg/DFGCombinedLiveness.cpp:

(JSC::DFG::addBytecodeLiveness):

• dfg/DFGCommonData.cpp:

(JSC::DFG::CommonData::addCodeOrigin):
(JSC::DFG::CommonData::addUniqueCallSiteIndex):
(JSC::DFG::CommonData::lastCallSite const):

• dfg/DFGConstantFoldingPhase.cpp:

(JSC::DFG::ConstantFoldingPhase::foldConstants):

• dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

• dfg/DFGDriver.cpp:

(JSC::DFG::compileImpl):

• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

• dfg/DFGForAllKills.h:

(JSC::DFG::forAllKilledOperands):
(JSC::DFG::forAllKilledNodesAtNodeIndex):
(JSC::DFG::forAllKillsInBlock):

• dfg/DFGGraph.cpp:

(JSC::DFG::Graph::dump):
(JSC::DFG::Graph::dumpBlockHeader):
(JSC::DFG::Graph::substituteGetLocal):
(JSC::DFG::Graph::isLiveInBytecode):
(JSC::DFG::Graph::localsAndTmpsLiveInBytecode):
(JSC::DFG::Graph::methodOfGettingAValueProfileFor):
(JSC::DFG::Graph::localsLiveInBytecode): Deleted.

• dfg/DFGGraph.h:

(JSC::DFG::Graph::forAllLocalsAndTmpsLiveInBytecode):
(JSC::DFG::Graph::forAllLiveInBytecode):
(JSC::DFG::Graph::forAllLocalsLiveInBytecode): Deleted.

• dfg/DFGInPlaceAbstractState.cpp:

(JSC::DFG::InPlaceAbstractState::InPlaceAbstractState):

• dfg/DFGInPlaceAbstractState.h:

(JSC::DFG::InPlaceAbstractState::operand):

• dfg/DFGJITCompiler.cpp:

(JSC::DFG::JITCompiler::linkOSRExits):
(JSC::DFG::JITCompiler::noticeOSREntry):

• dfg/DFGJITCompiler.h:

(JSC::DFG::JITCompiler::emitStoreCallSiteIndex):

• dfg/DFGLiveCatchVariablePreservationPhase.cpp:

(JSC::DFG::LiveCatchVariablePreservationPhase::isValidFlushLocation):
(JSC::DFG::LiveCatchVariablePreservationPhase::handleBlockForTryCatch):
(JSC::DFG::LiveCatchVariablePreservationPhase::newVariableAccessData):

• dfg/DFGMovHintRemovalPhase.cpp:
• dfg/DFGNode.h:

(JSC::DFG::StackAccessData::StackAccessData):
(JSC::DFG::Node::hasArgumentsChild):
(JSC::DFG::Node::argumentsChild):
(JSC::DFG::Node::operand):
(JSC::DFG::Node::hasUnlinkedOperand):
(JSC::DFG::Node::unlinkedOperand):
(JSC::DFG::Node::hasLoadVarargsData):
(JSC::DFG::Node::local): Deleted.
(JSC::DFG::Node::hasUnlinkedLocal): Deleted.
(JSC::DFG::Node::unlinkedLocal): Deleted.

• dfg/DFGNodeType.h:
• dfg/DFGOSRAvailabilityAnalysisPhase.cpp:

(JSC::DFG::OSRAvailabilityAnalysisPhase::run):
(JSC::DFG::LocalOSRAvailabilityCalculator::executeNode):

• dfg/DFGOSREntry.cpp:

(JSC::DFG::prepareOSREntry):
(JSC::DFG::prepareCatchOSREntry):

• dfg/DFGOSREntrypointCreationPhase.cpp:

(JSC::DFG::OSREntrypointCreationPhase::run):

• dfg/DFGOSRExit.cpp:

(JSC::DFG::OSRExit::emitRestoreArguments):
(JSC::DFG::OSRExit::compileExit):
(JSC::DFG::jsValueFor): Deleted.
(JSC::DFG::restoreCalleeSavesFor): Deleted.
(JSC::DFG::saveCalleeSavesFor): Deleted.
(JSC::DFG::restoreCalleeSavesFromVMEntryFrameCalleeSavesBuffer): Deleted.
(JSC::DFG::copyCalleeSavesToVMEntryFrameCalleeSavesBuffer): Deleted.
(JSC::DFG::saveOrCopyCalleeSavesFor): Deleted.
(JSC::DFG::createDirectArgumentsDuringExit): Deleted.
(JSC::DFG::createClonedArgumentsDuringExit): Deleted.
(JSC::DFG::emitRestoreArguments): Deleted.
(JSC::DFG::OSRExit::executeOSRExit): Deleted.
(JSC::DFG::reifyInlinedCallFrames): Deleted.
(JSC::DFG::adjustAndJumpToTarget): Deleted.
(JSC::DFG::printOSRExit): Deleted.

• dfg/DFGOSRExit.h:
• dfg/DFGOSRExitBase.h:

(JSC::DFG::OSRExitBase::isExitingToCheckpointHandler const):

• dfg/DFGOSRExitCompilerCommon.cpp:

(JSC::DFG::callerReturnPC):
(JSC::DFG::reifyInlinedCallFrames):
(JSC::DFG::adjustAndJumpToTarget):

• dfg/DFGObjectAllocationSinkingPhase.cpp:
• dfg/DFGOpInfo.h:

(JSC::DFG::OpInfo::OpInfo):

• dfg/DFGOperations.cpp:
• dfg/DFGPhantomInsertionPhase.cpp:
• dfg/DFGPreciseLocalClobberize.h:

(JSC::DFG::PreciseLocalClobberizeAdaptor::read):
(JSC::DFG::PreciseLocalClobberizeAdaptor::write):
(JSC::DFG::PreciseLocalClobberizeAdaptor::def):
(JSC::DFG::PreciseLocalClobberizeAdaptor::callIfAppropriate):

• dfg/DFGPredictionInjectionPhase.cpp:

(JSC::DFG::PredictionInjectionPhase::run):

• dfg/DFGPredictionPropagationPhase.cpp:
• dfg/DFGPutStackSinkingPhase.cpp:
• dfg/DFGSSAConversionPhase.cpp:

(JSC::DFG::SSAConversionPhase::run):

• dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

• dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::compileMovHint):
(JSC::DFG::SpeculativeJIT::compileCurrentBlock):
(JSC::DFG::SpeculativeJIT::checkArgumentTypes):
(JSC::DFG::SpeculativeJIT::compileVarargsLength):
(JSC::DFG::SpeculativeJIT::compileLoadVarargs):
(JSC::DFG::SpeculativeJIT::compileForwardVarargs):
(JSC::DFG::SpeculativeJIT::compileCreateDirectArguments):
(JSC::DFG::SpeculativeJIT::compileGetArgumentCountIncludingThis):

• dfg/DFGSpeculativeJIT.h:

(JSC::DFG::SpeculativeJIT::recordSetLocal):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGStackLayoutPhase.cpp:

(JSC::DFG::StackLayoutPhase::run):
(JSC::DFG::StackLayoutPhase::assign):

• dfg/DFGStrengthReductionPhase.cpp:

(JSC::DFG::StrengthReductionPhase::handleNode):

• dfg/DFGThunks.cpp:

(JSC::DFG::osrExitThunkGenerator): Deleted.

• dfg/DFGThunks.h:
• dfg/DFGTypeCheckHoistingPhase.cpp:

(JSC::DFG::TypeCheckHoistingPhase::run):
(JSC::DFG::TypeCheckHoistingPhase::disableHoistingAcrossOSREntries):

• dfg/DFGValidate.cpp:
• dfg/DFGVarargsForwardingPhase.cpp:
• dfg/DFGVariableAccessData.cpp:

(JSC::DFG::VariableAccessData::VariableAccessData):
(JSC::DFG::VariableAccessData::shouldUseDoubleFormatAccordingToVote):
(JSC::DFG::VariableAccessData::tallyVotesForShouldUseDoubleFormat):
(JSC::DFG::VariableAccessData::couldRepresentInt52Impl):

• dfg/DFGVariableAccessData.h:

(JSC::DFG::VariableAccessData::operand):
(JSC::DFG::VariableAccessData::local): Deleted.

• dfg/DFGVariableEvent.cpp:

(JSC::DFG::VariableEvent::dump const):

• dfg/DFGVariableEvent.h:

(JSC::DFG::VariableEvent::spill):
(JSC::DFG::VariableEvent::setLocal):
(JSC::DFG::VariableEvent::movHint):
(JSC::DFG::VariableEvent::spillRegister const):
(JSC::DFG::VariableEvent::operand const):
(JSC::DFG::VariableEvent::bytecodeRegister const): Deleted.

• dfg/DFGVariableEventStream.cpp:

(JSC::DFG::VariableEventStream::logEvent):
(JSC::DFG::VariableEventStream::reconstruct const):

• dfg/DFGVariableEventStream.h:

(JSC::DFG::VariableEventStream::appendAndLog):

• ftl/FTLCapabilities.cpp:

(JSC::FTL::canCompile):

• ftl/FTLForOSREntryJITCode.cpp:

(JSC::FTL::ForOSREntryJITCode::ForOSREntryJITCode):

• ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::lower):
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileExtractOSREntryLocal):
(JSC::FTL::DFG::LowerDFGToB3::compileGetStack):
(JSC::FTL::DFG::LowerDFGToB3::compileGetCallee):
(JSC::FTL::DFG::LowerDFGToB3::compileSetCallee):
(JSC::FTL::DFG::LowerDFGToB3::compileSetArgumentCountIncludingThis):
(JSC::FTL::DFG::LowerDFGToB3::compileVarargsLength):
(JSC::FTL::DFG::LowerDFGToB3::compileLoadVarargs):
(JSC::FTL::DFG::LowerDFGToB3::compileForwardVarargs):
(JSC::FTL::DFG::LowerDFGToB3::getSpreadLengthFromInlineCallFrame):
(JSC::FTL::DFG::LowerDFGToB3::compileForwardVarargsWithSpread):
(JSC::FTL::DFG::LowerDFGToB3::compileLogShadowChickenPrologue):
(JSC::FTL::DFG::LowerDFGToB3::getArgumentsLength):
(JSC::FTL::DFG::LowerDFGToB3::getCurrentCallee):
(JSC::FTL::DFG::LowerDFGToB3::callPreflight):
(JSC::FTL::DFG::LowerDFGToB3::appendOSRExitDescriptor):
(JSC::FTL::DFG::LowerDFGToB3::buildExitArguments):
(JSC::FTL::DFG::LowerDFGToB3::addressFor):
(JSC::FTL::DFG::LowerDFGToB3::payloadFor):
(JSC::FTL::DFG::LowerDFGToB3::tagFor):

• ftl/FTLOSREntry.cpp:

(JSC::FTL::prepareOSREntry):

• ftl/FTLOSRExit.cpp:

(JSC::FTL::OSRExitDescriptor::OSRExitDescriptor):

• ftl/FTLOSRExit.h:
• ftl/FTLOSRExitCompiler.cpp:

(JSC::FTL::compileStub):

• ftl/FTLOperations.cpp:

(JSC::FTL::operationMaterializeObjectInOSR):

• ftl/FTLOutput.cpp:

(JSC::FTL::Output::select):

• ftl/FTLOutput.h:
• ftl/FTLSelectPredictability.h: Copied from Source/JavaScriptCore/ftl/FTLForOSREntryJITCode.cpp.
• ftl/FTLSlowPathCall.h:

(JSC::FTL::callOperation):

• generator/Checkpoints.rb: Added.
• generator/Opcode.rb:
• generator/Section.rb:
• heap/Heap.cpp:

(JSC::Heap::gatherStackRoots):

• interpreter/CallFrame.cpp:

(JSC::CallFrame::callSiteAsRawBits const):
(JSC::CallFrame::unsafeCallSiteAsRawBits const):
(JSC::CallFrame::callSiteIndex const):
(JSC::CallFrame::unsafeCallSiteIndex const):
(JSC::CallFrame::setCurrentVPC):
(JSC::CallFrame::bytecodeIndex):
(JSC::CallFrame::codeOrigin):

• interpreter/CallFrame.h:

(JSC::CallSiteIndex::CallSiteIndex):
(JSC::CallSiteIndex::operator bool const):
(JSC::CallSiteIndex::operator== const):
(JSC::CallSiteIndex::bits const):
(JSC::CallSiteIndex::fromBits):
(JSC::CallSiteIndex::bytecodeIndex const):
(JSC::DisposableCallSiteIndex::DisposableCallSiteIndex):
(JSC::CallFrame::callee const):
(JSC::CallFrame::unsafeCallee const):
(JSC::CallFrame::addressOfCodeBlock const):
(JSC::CallFrame::argumentCountIncludingThis const):
(JSC::CallFrame::offsetFor):
(JSC::CallFrame::setArgumentCountIncludingThis):
(JSC::CallFrame::setReturnPC):

• interpreter/CallFrameInlines.h:

(JSC::CallFrame::r):
(JSC::CallFrame::uncheckedR):
(JSC::CallFrame::guaranteedJSValueCallee const):
(JSC::CallFrame::jsCallee const):
(JSC::CallFrame::codeBlock const):
(JSC::CallFrame::unsafeCodeBlock const):
(JSC::CallFrame::setCallee):
(JSC::CallFrame::setCodeBlock):

• interpreter/CheckpointOSRExitSideState.h: Copied from Source/JavaScriptCore/dfg/DFGThunks.h.
• interpreter/Interpreter.cpp:

(JSC::eval):
(JSC::sizeOfVarargs):
(JSC::loadVarargs):
(JSC::setupVarargsFrame):
(JSC::UnwindFunctor::operator() const):
(JSC::Interpreter::executeCall):
(JSC::Interpreter::executeConstruct):

• interpreter/Interpreter.h:
• interpreter/StackVisitor.cpp:

(JSC::StackVisitor::readInlinedFrame):

• jit/AssemblyHelpers.h:

(JSC::AssemblyHelpers::emitGetFromCallFrameHeaderPtr):
(JSC::AssemblyHelpers::emitGetFromCallFrameHeader32):
(JSC::AssemblyHelpers::emitGetFromCallFrameHeader64):
(JSC::AssemblyHelpers::emitPutToCallFrameHeader):
(JSC::AssemblyHelpers::emitPutToCallFrameHeaderBeforePrologue):
(JSC::AssemblyHelpers::emitPutPayloadToCallFrameHeaderBeforePrologue):
(JSC::AssemblyHelpers::emitPutTagToCallFrameHeaderBeforePrologue):
(JSC::AssemblyHelpers::addressFor):
(JSC::AssemblyHelpers::tagFor):
(JSC::AssemblyHelpers::payloadFor):
(JSC::AssemblyHelpers::calleeFrameSlot):
(JSC::AssemblyHelpers::calleeArgumentSlot):
(JSC::AssemblyHelpers::calleeFrameTagSlot):
(JSC::AssemblyHelpers::calleeFramePayloadSlot):
(JSC::AssemblyHelpers::calleeFrameCallerFrame):
(JSC::AssemblyHelpers::argumentCount):

• jit/CallFrameShuffler.cpp:

(JSC::CallFrameShuffler::CallFrameShuffler):

• jit/CallFrameShuffler.h:

(JSC::CallFrameShuffler::setCalleeJSValueRegs):
(JSC::CallFrameShuffler::assumeCalleeIsCell):

• jit/JIT.h:
• jit/JITArithmetic.cpp:

(JSC::JIT::emit_op_unsigned):
(JSC::JIT::emit_compareAndJump):
(JSC::JIT::emit_compareAndJumpImpl):
(JSC::JIT::emit_compareUnsignedAndJump):
(JSC::JIT::emit_compareUnsignedAndJumpImpl):
(JSC::JIT::emit_compareUnsigned):
(JSC::JIT::emit_compareUnsignedImpl):
(JSC::JIT::emit_compareAndJumpSlow):
(JSC::JIT::emit_compareAndJumpSlowImpl):
(JSC::JIT::emit_op_inc):
(JSC::JIT::emit_op_dec):
(JSC::JIT::emit_op_mod):
(JSC::JIT::emitBitBinaryOpFastPath):
(JSC::JIT::emit_op_bitnot):
(JSC::JIT::emitRightShiftFastPath):
(JSC::JIT::emitMathICFast):
(JSC::JIT::emitMathICSlow):
(JSC::JIT::emit_op_div):

• jit/JITCall.cpp:

(JSC::JIT::emitPutCallResult):
(JSC::JIT::compileSetupFrame):
(JSC::JIT::compileOpCall):

• jit/JITExceptions.cpp:

(JSC::genericUnwind):

• jit/JITInlines.h:

(JSC::JIT::isOperandConstantDouble):
(JSC::JIT::getConstantOperand):
(JSC::JIT::emitPutIntToCallFrameHeader):
(JSC::JIT::appendCallWithExceptionCheckSetJSValueResult):
(JSC::JIT::appendCallWithExceptionCheckSetJSValueResultWithProfile):
(JSC::JIT::linkSlowCaseIfNotJSCell):
(JSC::JIT::isOperandConstantChar):
(JSC::JIT::getOperandConstantInt):
(JSC::JIT::getOperandConstantDouble):
(JSC::JIT::emitInitRegister):
(JSC::JIT::emitLoadTag):
(JSC::JIT::emitLoadPayload):
(JSC::JIT::emitGet):
(JSC::JIT::emitPutVirtualRegister):
(JSC::JIT::emitLoad):
(JSC::JIT::emitLoad2):
(JSC::JIT::emitLoadDouble):
(JSC::JIT::emitLoadInt32ToDouble):
(JSC::JIT::emitStore):
(JSC::JIT::emitStoreInt32):
(JSC::JIT::emitStoreCell):
(JSC::JIT::emitStoreBool):
(JSC::JIT::emitStoreDouble):
(JSC::JIT::emitJumpSlowCaseIfNotJSCell):
(JSC::JIT::isOperandConstantInt):
(JSC::JIT::emitGetVirtualRegister):
(JSC::JIT::emitGetVirtualRegisters):

• jit/JITOpcodes.cpp:

(JSC::JIT::emit_op_mov):
(JSC::JIT::emit_op_end):
(JSC::JIT::emit_op_new_object):
(JSC::JIT::emitSlow_op_new_object):
(JSC::JIT::emit_op_overrides_has_instance):
(JSC::JIT::emit_op_instanceof):
(JSC::JIT::emitSlow_op_instanceof):
(JSC::JIT::emit_op_is_empty):
(JSC::JIT::emit_op_is_undefined):
(JSC::JIT::emit_op_is_undefined_or_null):
(JSC::JIT::emit_op_is_boolean):
(JSC::JIT::emit_op_is_number):
(JSC::JIT::emit_op_is_cell_with_type):
(JSC::JIT::emit_op_is_object):
(JSC::JIT::emit_op_ret):
(JSC::JIT::emit_op_to_primitive):
(JSC::JIT::emit_op_set_function_name):
(JSC::JIT::emit_op_not):
(JSC::JIT::emit_op_jfalse):
(JSC::JIT::emit_op_jeq_null):
(JSC::JIT::emit_op_jneq_null):
(JSC::JIT::emit_op_jundefined_or_null):
(JSC::JIT::emit_op_jnundefined_or_null):
(JSC::JIT::emit_op_jneq_ptr):
(JSC::JIT::emit_op_eq):
(JSC::JIT::emit_op_jeq):
(JSC::JIT::emit_op_jtrue):
(JSC::JIT::emit_op_neq):
(JSC::JIT::emit_op_jneq):
(JSC::JIT::emit_op_throw):
(JSC::JIT::compileOpStrictEq):
(JSC::JIT::compileOpStrictEqJump):
(JSC::JIT::emit_op_to_number):
(JSC::JIT::emit_op_to_numeric):
(JSC::JIT::emit_op_to_string):
(JSC::JIT::emit_op_to_object):
(JSC::JIT::emit_op_catch):
(JSC::JIT::emit_op_get_parent_scope):
(JSC::JIT::emit_op_switch_imm):
(JSC::JIT::emit_op_switch_char):
(JSC::JIT::emit_op_switch_string):
(JSC::JIT::emit_op_eq_null):
(JSC::JIT::emit_op_neq_null):
(JSC::JIT::emit_op_enter):
(JSC::JIT::emit_op_get_scope):
(JSC::JIT::emit_op_to_this):
(JSC::JIT::emit_op_create_this):
(JSC::JIT::emit_op_check_tdz):
(JSC::JIT::emitSlow_op_eq):
(JSC::JIT::emitSlow_op_neq):
(JSC::JIT::emitSlow_op_instanceof_custom):
(JSC::JIT::emit_op_new_regexp):
(JSC::JIT::emitNewFuncCommon):
(JSC::JIT::emitNewFuncExprCommon):
(JSC::JIT::emit_op_new_array):
(JSC::JIT::emit_op_new_array_with_size):
(JSC::JIT::emit_op_has_structure_property):
(JSC::JIT::emit_op_has_indexed_property):
(JSC::JIT::emitSlow_op_has_indexed_property):
(JSC::JIT::emit_op_get_direct_pname):
(JSC::JIT::emit_op_enumerator_structure_pname):
(JSC::JIT::emit_op_enumerator_generic_pname):
(JSC::JIT::emit_op_profile_type):
(JSC::JIT::emit_op_log_shadow_chicken_prologue):
(JSC::JIT::emit_op_log_shadow_chicken_tail):
(JSC::JIT::emit_op_argument_count):
(JSC::JIT::emit_op_get_rest_length):
(JSC::JIT::emit_op_get_argument):

• jit/JITOpcodes32_64.cpp:

(JSC::JIT::emit_op_catch):

• jit/JITOperations.cpp:
• jit/JITPropertyAccess.cpp:

(JSC::JIT::emit_op_get_by_val):
(JSC::JIT::emitSlow_op_get_by_val):
(JSC::JIT::emit_op_put_by_val):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emitPutByValWithCachedId):
(JSC::JIT::emitSlow_op_put_by_val):
(JSC::JIT::emit_op_put_getter_by_id):
(JSC::JIT::emit_op_put_setter_by_id):
(JSC::JIT::emit_op_put_getter_setter_by_id):
(JSC::JIT::emit_op_put_getter_by_val):
(JSC::JIT::emit_op_put_setter_by_val):
(JSC::JIT::emit_op_del_by_id):
(JSC::JIT::emit_op_del_by_val):
(JSC::JIT::emit_op_try_get_by_id):
(JSC::JIT::emitSlow_op_try_get_by_id):
(JSC::JIT::emit_op_get_by_id_direct):
(JSC::JIT::emitSlow_op_get_by_id_direct):
(JSC::JIT::emit_op_get_by_id):
(JSC::JIT::emit_op_get_by_id_with_this):
(JSC::JIT::emitSlow_op_get_by_id):
(JSC::JIT::emitSlow_op_get_by_id_with_this):
(JSC::JIT::emit_op_put_by_id):
(JSC::JIT::emit_op_in_by_id):
(JSC::JIT::emitSlow_op_in_by_id):
(JSC::JIT::emitResolveClosure):
(JSC::JIT::emit_op_resolve_scope):
(JSC::JIT::emitLoadWithStructureCheck):
(JSC::JIT::emitGetClosureVar):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emitSlow_op_get_from_scope):
(JSC::JIT::emitPutGlobalVariable):
(JSC::JIT::emitPutGlobalVariableIndirect):
(JSC::JIT::emitPutClosureVar):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emit_op_get_from_arguments):
(JSC::JIT::emit_op_put_to_arguments):
(JSC::JIT::emitWriteBarrier):
(JSC::JIT::emit_op_get_internal_field):
(JSC::JIT::emit_op_put_internal_field):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):

• jit/JSInterfaceJIT.h:

(JSC::JSInterfaceJIT::emitLoadJSCell):
(JSC::JSInterfaceJIT::emitJumpIfNotJSCell):
(JSC::JSInterfaceJIT::emitLoadInt32):
(JSC::JSInterfaceJIT::emitLoadDouble):
(JSC::JSInterfaceJIT::emitGetFromCallFrameHeaderPtr):
(JSC::JSInterfaceJIT::emitPutToCallFrameHeader):
(JSC::JSInterfaceJIT::emitPutCellToCallFrameHeader):

• jit/SetupVarargsFrame.cpp:

(JSC::emitSetupVarargsFrameFastCase):

• jit/SpecializedThunkJIT.h:

(JSC::SpecializedThunkJIT::loadDoubleArgument):
(JSC::SpecializedThunkJIT::loadCellArgument):
(JSC::SpecializedThunkJIT::loadInt32Argument):

• jit/ThunkGenerators.cpp:

(JSC::absThunkGenerator):

• llint/LLIntSlowPaths.cpp:

(JSC::LLInt::getNonConstantOperand):
(JSC::LLInt::getOperand):
(JSC::LLInt::genericCall):
(JSC::LLInt::varargsSetup):
(JSC::LLInt::commonCallEval):
(JSC::LLInt::LLINT_SLOW_PATH_DECL):
(JSC::LLInt::handleVarargsCheckpoint):
(JSC::LLInt::dispatchToNextInstruction):
(JSC::LLInt::slow_path_checkpoint_osr_exit_from_inlined_call):
(JSC::LLInt::slow_path_checkpoint_osr_exit):
(JSC::LLInt::llint_throw_stack_overflow_error):

• llint/LLIntSlowPaths.h:
• llint/LowLevelInterpreter.asm:
• llint/LowLevelInterpreter32_64.asm:
• llint/LowLevelInterpreter64.asm:
• runtime/ArgList.h:

(JSC::MarkedArgumentBuffer::fill):

• runtime/CachedTypes.cpp:

(JSC::CachedCodeBlock::hasCheckpoints const):
(JSC::UnlinkedCodeBlock::UnlinkedCodeBlock):
(JSC::CachedCodeBlock<CodeBlockType>::encode):

• runtime/CommonSlowPaths.cpp:

(JSC::SLOW_PATH_DECL):

• runtime/ConstructData.cpp:

(JSC::construct):

• runtime/ConstructData.h:
• runtime/DirectArguments.cpp:

(JSC::DirectArguments::copyToArguments):

• runtime/DirectArguments.h:
• runtime/GenericArguments.h:
• runtime/GenericArgumentsInlines.h:

(JSC::GenericArguments<Type>::copyToArguments):

• runtime/JSArray.cpp:

(JSC::JSArray::copyToArguments):

• runtime/JSArray.h:
• runtime/JSImmutableButterfly.cpp:

(JSC::JSImmutableButterfly::copyToArguments):

• runtime/JSImmutableButterfly.h:
• runtime/JSLock.cpp:

(JSC::JSLock::willReleaseLock):

• runtime/ModuleProgramExecutable.cpp:

(JSC::ModuleProgramExecutable::create):

• runtime/Options.cpp:

(JSC::recomputeDependentOptions):

• runtime/ScopedArguments.cpp:

(JSC::ScopedArguments::copyToArguments):

• runtime/ScopedArguments.h:
• runtime/VM.cpp:

(JSC::VM::addCheckpointOSRSideState):
(JSC::VM::findCheckpointOSRSideState):
(JSC::VM::scanSideState const):

• runtime/VM.h:

(JSC::VM::hasCheckpointOSRSideState const):

• tools/VMInspector.cpp:

(JSC::VMInspector::dumpRegisters):

• wasm/WasmFunctionCodeBlock.h:

(JSC::Wasm::FunctionCodeBlock::getConstant const):
(JSC::Wasm::FunctionCodeBlock::getConstantType const):

• wasm/WasmLLIntGenerator.cpp:

(JSC::Wasm::LLIntGenerator::setUsesCheckpoints const):

• wasm/WasmOperations.cpp:

(JSC::Wasm::operationWasmToJSException):

• wasm/WasmSlowPaths.cpp:

Source/WTF:

• WTF.xcodeproj/project.pbxproj:
• wtf/Bitmap.h:

(WTF::WordType>::invert):
(WTF::WordType>::operator):
(WTF::WordType>::operator const const):

• wtf/CMakeLists.txt:
• wtf/EnumClassOperatorOverloads.h: Added.
• wtf/FastBitVector.h:

(WTF::FastBitReference::operator bool const):
(WTF::FastBitReference::operator|=):
(WTF::FastBitReference::operator&=):
(WTF::FastBitVector::fill):
(WTF::FastBitVector::grow):

• wtf/UnalignedAccess.h:

(WTF::unalignedLoad):
(WTF::unalignedStore):

Tools:

• Scripts/run-jsc-stress-tests:

File:

• trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp (modified) (54 diffs)

trunk/Source/JavaScriptCore/dfg/DFGByteCodeParser.cpp

@@ -112 +112 @@
         , m_numArguments(m_codeBlock->numParameters())
         , m_numLocals(m_codeBlock->numCalleeLocals())
+        , m_numTmps(m_codeBlock->numTmps())
         , m_parameterSlots(0)
         , m_numPassedVarArgs(0)
@@ -139 +140 @@
             m_graph.block(i)->ensureLocals(newNumLocals);
     }
+
+    void ensureTmps(unsigned newNumTmps)
+    {
+        VERBOSE_LOG("   ensureTmps: trying to raise m_numTmps from ", m_numTmps, " to ", newNumTmps, "\n");
+        if (newNumTmps <= m_numTmps)
+            return;
+        m_numTmps = newNumTmps;
+        for (size_t i = 0; i < m_graph.numBlocks(); ++i)
+            m_graph.block(i)->ensureTmps(newNumTmps);
+    }
+
 
     // Helper for min and max.
@@ -271 +283 @@
     void linkBlocks(Vector<BasicBlock*>& unlinkedBlocks, Vector<BasicBlock*>& possibleTargets);
     
-    VariableAccessData* newVariableAccessData(VirtualRegister operand)
+    void progressToNextCheckpoint()
+    {
+        m_currentIndex = BytecodeIndex(m_currentIndex.offset(), m_currentIndex.checkpoint() + 1);
+        // At this point, it's again OK to OSR exit.
+        m_exitOK = true;
+        addToGraph(ExitOK);
+
+        processSetLocalQueue();
+    }
+
+    VariableAccessData* newVariableAccessData(Operand operand)
     {
         ASSERT(!operand.isConstant());
@@ -280 +302 @@
     
     // Get/Set the operands/result of a bytecode instruction.
-    Node* getDirect(VirtualRegister operand)
+    Node* getDirect(Operand operand)
     {
         ASSERT(!operand.isConstant());
 
-        // Is this an argument?
         if (operand.isArgument())
-            return getArgument(operand);
-
-        // Must be a local.
-        return getLocal(operand);
+            return getArgument(operand.virtualRegister());
+
+        return getLocalOrTmp(operand);
     }
 
@@ -299 +319 @@
             if (constantIndex >= oldSize || !m_constants[constantIndex]) {
                 const CodeBlock& codeBlock = *m_inlineStackTop->m_codeBlock;
-                JSValue value = codeBlock.getConstant(operand.offset());
-                SourceCodeRepresentation sourceCodeRepresentation = codeBlock.constantSourceCodeRepresentation(operand.offset());
+                JSValue value = codeBlock.getConstant(operand);
+                SourceCodeRepresentation sourceCodeRepresentation = codeBlock.constantSourceCodeRepresentation(operand);
                 if (constantIndex >= oldSize) {
                     m_constants.grow(constantIndex + 1);
@@ -359 +379 @@
         ImmediateNakedSet
     };
-    Node* setDirect(VirtualRegister operand, Node* value, SetMode setMode = NormalSet)
+
+    Node* setDirect(Operand operand, Node* value, SetMode setMode = NormalSet)
     {
-        addToGraph(MovHint, OpInfo(operand.offset()), value);
+        addToGraph(MovHint, OpInfo(operand), value);
 
         // We can't exit anymore because our OSR exit state has changed.
@@ -375 +396 @@
         return delayed.execute(this);
     }
-    
+
     void processSetLocalQueue()
     {
@@ -393 +414 @@
         ASSERT(node->origin.semantic.bytecodeIndex() == m_currentIndex);
         ConcurrentJSLocker locker(m_inlineStackTop->m_profiledBlock->m_lock);
-        LazyOperandValueProfileKey key(m_currentIndex, node->local());
+        LazyOperandValueProfileKey key(m_currentIndex, node->operand());
         SpeculatedType prediction = m_inlineStackTop->m_lazyOperands.prediction(locker, key);
         node->variableAccessData()->predict(prediction);
@@ -400 +421 @@
 
     // Used in implementing get/set, above, where the operand is a local variable.
-    Node* getLocal(VirtualRegister operand)
+    Node* getLocalOrTmp(Operand operand)
     {
-        unsigned local = operand.toLocal();
-
-        Node* node = m_currentBlock->variablesAtTail.local(local);
+        ASSERT(operand.isTmp() || operand.isLocal());
+        Node*& node = m_currentBlock->variablesAtTail.operand(operand);
         
         // This has two goals: 1) link together variable access datas, and 2)
@@ -429 +449 @@
         
         node = injectLazyOperandSpeculation(addToGraph(GetLocal, OpInfo(variable)));
-        m_currentBlock->variablesAtTail.local(local) = node;
         return node;
     }
-    Node* setLocal(const CodeOrigin& semanticOrigin, VirtualRegister operand, Node* value, SetMode setMode = NormalSet)
+    Node* setLocalOrTmp(const CodeOrigin& semanticOrigin, Operand operand, Node* value, SetMode setMode = NormalSet)
     {
+        ASSERT(operand.isTmp() || operand.isLocal());
         SetForScope<CodeOrigin> originChange(m_currentSemanticOrigin, semanticOrigin);
 
-        unsigned local = operand.toLocal();
-        
-        if (setMode != ImmediateNakedSet) {
-            ArgumentPosition* argumentPosition = findArgumentPositionForLocal(operand);
+        if (operand.isTmp() && static_cast<unsigned>(operand.value()) >= m_numTmps) {
+            if (inlineCallFrame())
+                dataLogLn(*inlineCallFrame());
+            dataLogLn("Bad operand: ", operand, " but current number of tmps is: ", m_numTmps, " code block has: ", m_profiledBlock->numTmps(), " tmps.");
+            CRASH();
+        }
+
+        if (setMode != ImmediateNakedSet && !operand.isTmp()) {
+            VirtualRegister reg = operand.virtualRegister();
+            ArgumentPosition* argumentPosition = findArgumentPositionForLocal(reg);
             if (argumentPosition)
                 flushDirect(operand, argumentPosition);
-            else if (m_graph.needsScopeRegister() && operand == m_codeBlock->scopeRegister())
+            else if (m_graph.needsScopeRegister() && reg == m_codeBlock->scopeRegister())
                 flush(operand);
         }
@@ -452 +478 @@
             m_inlineStackTop->m_exitProfile.hasExitSite(semanticOrigin.bytecodeIndex(), BadIndexingType));
         Node* node = addToGraph(SetLocal, OpInfo(variableAccessData), value);
-        m_currentBlock->variablesAtTail.local(local) = node;
+        m_currentBlock->variablesAtTail.operand(operand) = node;
         return node;
     }
@@ -484 +510 @@
         return node;
     }
-    Node* setArgument(const CodeOrigin& semanticOrigin, VirtualRegister operand, Node* value, SetMode setMode = NormalSet)
+    Node* setArgument(const CodeOrigin& semanticOrigin, Operand operand, Node* value, SetMode setMode = NormalSet)
     {
         SetForScope<CodeOrigin> originChange(m_currentSemanticOrigin, semanticOrigin);
 
-        unsigned argument = operand.toArgument();
+        VirtualRegister reg = operand.virtualRegister();
+        unsigned argument = reg.toArgument();
         ASSERT(argument < m_numArguments);
         
-        VariableAccessData* variableAccessData = newVariableAccessData(operand);
+        VariableAccessData* variableAccessData = newVariableAccessData(reg);
 
         // Always flush arguments, except for 'this'. If 'this' is created by us,
@@ -497 +524 @@
         if (argument || m_graph.needsFlushedThis()) {
             if (setMode != ImmediateNakedSet)
-                flushDirect(operand);
+                flushDirect(reg);
         }
         
@@ -533 +560 @@
             return stack->m_argumentPositions[argument];
         }
-        return 0;
-    }
-    
-    ArgumentPosition* findArgumentPosition(VirtualRegister operand)
+        return nullptr;
+    }
+    
+    ArgumentPosition* findArgumentPosition(Operand operand)
     {
+        if (operand.isTmp())
+            return nullptr;
         if (operand.isArgument())
             return findArgumentPositionForArgument(operand.toArgument());
-        return findArgumentPositionForLocal(operand);
+        return findArgumentPositionForLocal(operand.virtualRegister());
     }
 
@@ -551 +580 @@
             numArguments = inlineCallFrame->argumentsWithFixup.size();
             if (inlineCallFrame->isClosureCall)
-                addFlushDirect(inlineCallFrame, remapOperand(inlineCallFrame, VirtualRegister(CallFrameSlot::callee)));
+                addFlushDirect(inlineCallFrame, remapOperand(inlineCallFrame, CallFrameSlot::callee));
             if (inlineCallFrame->isVarargs())
-                addFlushDirect(inlineCallFrame, remapOperand(inlineCallFrame, VirtualRegister(CallFrameSlot::argumentCountIncludingThis)));
+                addFlushDirect(inlineCallFrame, remapOperand(inlineCallFrame, CallFrameSlot::argumentCountIncludingThis));
         } else
             numArguments = m_graph.baselineCodeBlockFor(inlineCallFrame)->numParameters();
@@ -576 +605 @@
                 CodeBlock* codeBlock = m_graph.baselineCodeBlockFor(inlineCallFrame);
                 FullBytecodeLiveness& fullLiveness = m_graph.livenessFor(codeBlock);
+                // Note: We don't need to handle tmps here because tmps are not required to be flushed to the stack.
                 const auto& livenessAtBytecode = fullLiveness.getLiveness(bytecodeIndex, m_graph.appropriateLivenessCalculationPoint(origin, isCallerOrigin));
                 for (unsigned local = codeBlock->numCalleeLocals(); local--;) {
@@ -585 +615 @@
     }
 
-    void flush(VirtualRegister operand)
+    void flush(Operand operand)
     {
         flushDirect(m_inlineStackTop->remapOperand(operand));
     }
     
-    void flushDirect(VirtualRegister operand)
+    void flushDirect(Operand operand)
     {
         flushDirect(operand, findArgumentPosition(operand));
     }
 
-    void flushDirect(VirtualRegister operand, ArgumentPosition* argumentPosition)
+    void flushDirect(Operand operand, ArgumentPosition* argumentPosition)
     {
         addFlushOrPhantomLocal<Flush>(operand, argumentPosition);
@@ -601 +631 @@
 
     template<NodeType nodeType>
-    void addFlushOrPhantomLocal(VirtualRegister operand, ArgumentPosition* argumentPosition)
+    void addFlushOrPhantomLocal(Operand operand, ArgumentPosition* argumentPosition)
     {
         ASSERT(!operand.isConstant());
         
-        Node* node = m_currentBlock->variablesAtTail.operand(operand);
+        Node*& node = m_currentBlock->variablesAtTail.operand(operand);
         
         VariableAccessData* variable;
@@ -615 +645 @@
         
         node = addToGraph(nodeType, OpInfo(variable));
-        m_currentBlock->variablesAtTail.operand(operand) = node;
         if (argumentPosition)
             argumentPosition->addVariable(variable);
     }
 
-    void phantomLocalDirect(VirtualRegister operand)
+    void phantomLocalDirect(Operand operand)
     {
         addFlushOrPhantomLocal<PhantomLocal>(operand, findArgumentPosition(operand));
@@ -627 +656 @@
     void flush(InlineStackEntry* inlineStackEntry)
     {
-        auto addFlushDirect = [&] (InlineCallFrame*, VirtualRegister reg) { flushDirect(reg); };
+        auto addFlushDirect = [&] (InlineCallFrame*, Operand operand) { flushDirect(operand); };
         flushImpl(inlineStackEntry->m_inlineCallFrame, addFlushDirect);
     }
@@ -633 +662 @@
     void flushForTerminal()
     {
-        auto addFlushDirect = [&] (InlineCallFrame*, VirtualRegister reg) { flushDirect(reg); };
-        auto addPhantomLocalDirect = [&] (InlineCallFrame*, VirtualRegister reg) { phantomLocalDirect(reg); };
+        auto addFlushDirect = [&] (InlineCallFrame*, Operand operand) { flushDirect(operand); };
+        auto addPhantomLocalDirect = [&] (InlineCallFrame*, Operand operand) { phantomLocalDirect(operand); };
         flushForTerminalImpl(currentCodeOrigin(), addFlushDirect, addPhantomLocalDirect);
     }
@@ -761 +790 @@
             Edge(child1), Edge(child2), Edge(child3));
         return addToGraph(result);
+    }
+    Node* addToGraph(NodeType op, Operand operand, Node* child1)
+    {
+        ASSERT(op == MovHint);
+        return addToGraph(op, OpInfo(operand.kind()), OpInfo(operand.value()), child1);
     }
     Node* addToGraph(NodeType op, OpInfo info1, OpInfo info2, Edge child1, Edge child2 = Edge(), Edge child3 = Edge())
@@ -1092 +1126 @@
     // The number of arguments passed to the function.
     unsigned m_numArguments;
-    // The number of locals (vars + temporaries) used in the function.
+    // The number of locals (vars + temporaries) used by the bytecode for the function.
     unsigned m_numLocals;
+    // The max number of temps used for forwarding data to an OSR exit checkpoint.
+    unsigned m_numTmps;
     // The number of slots (in units of sizeof(Register)) that we need to
     // preallocate for arguments to outgoing calls from this frame. This
@@ -1160 +1196 @@
         ~InlineStackEntry();
         
-        VirtualRegister remapOperand(VirtualRegister operand) const
+        Operand remapOperand(Operand operand) const
         {
             if (!m_inlineCallFrame)
                 return operand;
+
+            if (operand.isTmp())
+                return Operand::tmp(operand.value() + m_inlineCallFrame->tmpOffset);
             
-            ASSERT(!operand.isConstant());
-
-            return VirtualRegister(operand.offset() + m_inlineCallFrame->stackOffset);
+            ASSERT(!operand.virtualRegister().isConstant());
+
+            return operand.virtualRegister() + m_inlineCallFrame->stackOffset;
         }
     };
@@ -1176 +1215 @@
     
     struct DelayedSetLocal {
-        CodeOrigin m_origin;
-        VirtualRegister m_operand;
-        Node* m_value;
-        SetMode m_setMode;
-        
         DelayedSetLocal() { }
-        DelayedSetLocal(const CodeOrigin& origin, VirtualRegister operand, Node* value, SetMode setMode)
+        DelayedSetLocal(const CodeOrigin& origin, Operand operand, Node* value, SetMode setMode)
             : m_origin(origin)
             , m_operand(operand)
@@ -1195 +1229 @@
             if (m_operand.isArgument())
                 return parser->setArgument(m_origin, m_operand, m_value, m_setMode);
-            return parser->setLocal(m_origin, m_operand, m_value, m_setMode);
-        }
+            return parser->setLocalOrTmp(m_origin, m_operand, m_value, m_setMode);
+        }
+
+        CodeOrigin m_origin;
+        Operand m_operand;
+        Node* m_value { nullptr };
+        SetMode m_setMode;
     };
     
@@ -1209 +1248 @@
 {
     ASSERT(bytecodeIndex);
-    Ref<BasicBlock> block = adoptRef(*new BasicBlock(bytecodeIndex, m_numArguments, m_numLocals, 1));
+    Ref<BasicBlock> block = adoptRef(*new BasicBlock(bytecodeIndex, m_numArguments, m_numLocals, m_numTmps, 1));
     BasicBlock* blockPtr = block.ptr();
     // m_blockLinkingTargets must always be sorted in increasing order of bytecodeBegin
@@ -1221 +1260 @@
 BasicBlock* ByteCodeParser::allocateUntargetableBlock()
 {
-    Ref<BasicBlock> block = adoptRef(*new BasicBlock(BytecodeIndex(), m_numArguments, m_numLocals, 1));
+    Ref<BasicBlock> block = adoptRef(*new BasicBlock(BytecodeIndex(), m_numArguments, m_numLocals, m_numTmps, 1));
     BasicBlock* blockPtr = block.ptr();
     m_graph.appendBlock(WTFMove(block));
@@ -1424 +1463 @@
                 continue;
             // If the target InlineCallFrame is Varargs, we do not know how many arguments are actually filled by LoadVarargs. Varargs InlineCallFrame's
-            // argumentCountIncludingThis is maximum number of potentially filled arguments by LoadVarargs. We "continue" to the upper frame which may be
+            // argumentCountIncludingThis is maximum number of potentially filled arguments by xkLoadVarargs. We "continue" to the upper frame which may be
             // a good target to jump into.
             if (callFrame->isVarargs())
@@ -1450 +1489 @@
         if (stackEntry->m_inlineCallFrame) {
             if (stackEntry->m_inlineCallFrame->isClosureCall)
-                setDirect(stackEntry->remapOperand(VirtualRegister(CallFrameSlot::callee)), callTargetNode, NormalSet);
+                setDirect(remapOperand(stackEntry->m_inlineCallFrame, CallFrameSlot::callee), callTargetNode, NormalSet);
         } else
             addToGraph(SetCallee, callTargetNode);
@@ -1615 +1654 @@
     int registerOffsetAfterFixup = registerOffset - numberOfStackPaddingSlots;
     
-    int inlineCallFrameStart = m_inlineStackTop->remapOperand(VirtualRegister(registerOffsetAfterFixup)).offset() + CallFrame::headerSizeInRegisters;
+    Operand inlineCallFrameStart = VirtualRegister(m_inlineStackTop->remapOperand(VirtualRegister(registerOffsetAfterFixup)).value() + CallFrame::headerSizeInRegisters);
     
     ensureLocals(
-        VirtualRegister(inlineCallFrameStart).toLocal() + 1 +
+        inlineCallFrameStart.toLocal() + 1 +
         CallFrame::headerSizeInRegisters + codeBlock->numCalleeLocals());
     
+    ensureTmps((m_inlineStackTop->m_inlineCallFrame ? m_inlineStackTop->m_inlineCallFrame->tmpOffset : 0) + m_inlineStackTop->m_codeBlock->numTmps() + codeBlock->numTmps());
+
     size_t argumentPositionStart = m_graph.m_argumentPositions.size();
 
     if (result.isValid())
-        result = m_inlineStackTop->remapOperand(result);
+        result = m_inlineStackTop->remapOperand(result).virtualRegister();
 
     VariableAccessData* calleeVariable = nullptr;
@@ -1637 +1678 @@
     InlineStackEntry* callerStackTop = m_inlineStackTop;
     InlineStackEntry inlineStackEntry(this, codeBlock, codeBlock, callee.function(), result,
-        (VirtualRegister)inlineCallFrameStart, argumentCountIncludingThis, kind, continuationBlock);
+        inlineCallFrameStart.virtualRegister(), argumentCountIncludingThis, kind, continuationBlock);
 
     // This is where the actual inlining really happens.
@@ -1672 +1713 @@
         // callee make it so that if we exit at <HERE>, we can recover loc9 and loc10.
         for (int index = 0; index < argumentCountIncludingThis; ++index) {
-            VirtualRegister argumentToGet = callerStackTop->remapOperand(virtualRegisterForArgument(index, registerOffset));
+            Operand argumentToGet = callerStackTop->remapOperand(virtualRegisterForArgument(index, registerOffset));
             Node* value = getDirect(argumentToGet);
-            addToGraph(MovHint, OpInfo(argumentToGet.offset()), value);
+            addToGraph(MovHint, OpInfo(argumentToGet), value);
             m_setLocalQueue.append(DelayedSetLocal { currentCodeOrigin(), argumentToGet, value, ImmediateNakedSet });
         }
@@ -1718 +1759 @@
         if (registerOffsetAfterFixup != registerOffset) {
             for (int index = 0; index < argumentCountIncludingThis; ++index) {
-                VirtualRegister argumentToGet = callerStackTop->remapOperand(virtualRegisterForArgument(index, registerOffset));
+                Operand argumentToGet = callerStackTop->remapOperand(virtualRegisterForArgument(index, registerOffset));
                 Node* value = getDirect(argumentToGet);
-                VirtualRegister argumentToSet = m_inlineStackTop->remapOperand(virtualRegisterForArgument(index));
-                addToGraph(MovHint, OpInfo(argumentToSet.offset()), value);
+                Operand argumentToSet = m_inlineStackTop->remapOperand(virtualRegisterForArgument(index));
+                addToGraph(MovHint, OpInfo(argumentToSet), value);
                 m_setLocalQueue.append(DelayedSetLocal { currentCodeOrigin(), argumentToSet, value, ImmediateNakedSet });
             }
         }
         for (int index = 0; index < arityFixupCount; ++index) {
-            VirtualRegister argumentToSet = m_inlineStackTop->remapOperand(virtualRegisterForArgument(argumentCountIncludingThis + index));
-            addToGraph(MovHint, OpInfo(argumentToSet.offset()), undefined);
+            Operand argumentToSet = m_inlineStackTop->remapOperand(virtualRegisterForArgument(argumentCountIncludingThis + index));
+            addToGraph(MovHint, OpInfo(argumentToSet), undefined);
             m_setLocalQueue.append(DelayedSetLocal { currentCodeOrigin(), argumentToSet, undefined, ImmediateNakedSet });
         }
@@ -1894 +1935 @@
         
         int remappedRegisterOffset =
-        m_inlineStackTop->remapOperand(VirtualRegister(registerOffset)).offset();
+        m_inlineStackTop->remapOperand(VirtualRegister(registerOffset)).virtualRegister().offset();
         
         ensureLocals(VirtualRegister(remappedRegisterOffset).toLocal());
         
         int argumentStart = registerOffset + CallFrame::headerSizeInRegisters;
-        int remappedArgumentStart = m_inlineStackTop->remapOperand(VirtualRegister(argumentStart)).offset();
+        int remappedArgumentStart = m_inlineStackTop->remapOperand(VirtualRegister(argumentStart)).virtualRegister().offset();
         
         LoadVarargsData* data = m_graph.m_loadVarargsData.add();
@@ -1907 +1948 @@
         data->limit = maxArgumentCountIncludingThis;
         data->mandatoryMinimum = mandatoryMinimum;
-        
-        if (callOp == TailCallForwardVarargs)
-            addToGraph(ForwardVarargs, OpInfo(data));
-        else
-            addToGraph(LoadVarargs, OpInfo(data), get(argumentsArgument));
+
+        if (callOp == TailCallForwardVarargs) {
+            Node* argumentCount;
+            if (!inlineCallFrame())
+                argumentCount = addToGraph(GetArgumentCountIncludingThis);
+            else if (inlineCallFrame()->isVarargs())
+                argumentCount = getDirect(remapOperand(inlineCallFrame(), CallFrameSlot::argumentCountIncludingThis));
+            else 
+                argumentCount = addToGraph(JSConstant, OpInfo(m_graph.freeze(jsNumber(inlineCallFrame()->argumentCountIncludingThis))));
+            addToGraph(ForwardVarargs, OpInfo(data), argumentCount);
+        } else {
+            Node* arguments = get(argumentsArgument);
+            auto argCountTmp = m_inlineStackTop->remapOperand(Operand::tmp(OpCallVarargs::argCountIncludingThis));
+            setDirect(argCountTmp, addToGraph(VarargsLength, OpInfo(data), arguments));
+            progressToNextCheckpoint();
+
+            addToGraph(LoadVarargs, OpInfo(data), getLocalOrTmp(argCountTmp), arguments);
+        }
         
         // LoadVarargs may OSR exit. Hence, we need to keep alive callTargetNode, thisArgument
@@ -1923 +1977 @@
         // before SSA.
         
-        VariableAccessData* countVariable = newVariableAccessData(VirtualRegister(remappedRegisterOffset + CallFrameSlot::argumentCountIncludingThis));
+        VariableAccessData* countVariable = newVariableAccessData(data->count);
         // This is pretty lame, but it will force the count to be flushed as an int. This doesn't
         // matter very much, since our use of a SetArgumentDefinitely and Flushes for this local slot is
@@ -1930 +1984 @@
         countVariable->mergeIsProfitableToUnbox(true);
         Node* setArgumentCount = addToGraph(SetArgumentDefinitely, OpInfo(countVariable));
-        m_currentBlock->variablesAtTail.setOperand(countVariable->local(), setArgumentCount);
+        m_currentBlock->variablesAtTail.setOperand(countVariable->operand(), setArgumentCount);
         
         set(VirtualRegister(argumentStart), get(thisArgument), ImmediateNakedSet);
@@ -1954 +2008 @@
             
             Node* setArgument = addToGraph(numSetArguments >= mandatoryMinimum ? SetArgumentMaybe : SetArgumentDefinitely, OpInfo(variable));
-            m_currentBlock->variablesAtTail.setOperand(variable->local(), setArgument);
+            m_currentBlock->variablesAtTail.setOperand(variable->operand(), setArgument);
             ++numSetArguments;
         }
@@ -2054 +2108 @@
     VERBOSE_LOG("Register offset: ", registerOffset);
     VirtualRegister calleeReg(registerOffset + CallFrameSlot::callee);
-    calleeReg = m_inlineStackTop->remapOperand(calleeReg);
+    calleeReg = m_inlineStackTop->remapOperand(calleeReg).virtualRegister();
     VERBOSE_LOG("Callee is going to be ", calleeReg, "\n");
     setDirect(calleeReg, callTargetNode, ImmediateSetWithFlush);
@@ -5103 +5157 @@
             auto bytecode = currentInstruction->as<OpNewRegexp>();
             ASSERT(bytecode.m_regexp.isConstant());
-            FrozenValue* frozenRegExp = m_graph.freezeStrong(m_inlineStackTop->m_codeBlock->getConstant(bytecode.m_regexp.offset()));
+            FrozenValue* frozenRegExp = m_graph.freezeStrong(m_inlineStackTop->m_codeBlock->getConstant(bytecode.m_regexp));
             set(bytecode.m_dst, addToGraph(NewRegexp, OpInfo(frozenRegExp), jsConstant(jsNumber(0))));
             NEXT_OPCODE(op_new_regexp);
@@ -6179 +6233 @@
             RELEASE_ASSERT(!m_currentBlock->size() || (m_graph.compilation() && m_currentBlock->size() == 1 && m_currentBlock->at(0)->op() == CountExecution));
 
-            ValueProfileAndOperandBuffer* buffer = bytecode.metadata(codeBlock).m_buffer;
+            ValueProfileAndVirtualRegisterBuffer* buffer = bytecode.metadata(codeBlock).m_buffer;
 
             if (!buffer) {
@@ -6196 +6250 @@
                 ConcurrentJSLocker locker(m_inlineStackTop->m_profiledBlock->m_lock);
 
-                buffer->forEach([&] (ValueProfileAndOperand& profile) {
+                buffer->forEach([&] (ValueProfileAndVirtualRegister& profile) {
                     VirtualRegister operand(profile.m_operand);
                     SpeculatedType prediction = profile.computeUpdatedPrediction(locker);
@@ -6224 +6278 @@
 
             unsigned numberOfLocals = 0;
-            buffer->forEach([&] (ValueProfileAndOperand& profile) {
+            buffer->forEach([&] (ValueProfileAndVirtualRegister& profile) {
                 VirtualRegister operand(profile.m_operand);
                 if (operand.isArgument())
@@ -6231 +6285 @@
                 Node* value = addToGraph(ExtractCatchLocal, OpInfo(numberOfLocals), OpInfo(localPredictions[numberOfLocals]));
                 ++numberOfLocals;
-                addToGraph(MovHint, OpInfo(profile.m_operand), value);
+                addToGraph(MovHint, OpInfo(operand), value);
                 localsToSet.uncheckedAppend(std::make_pair(operand, value));
             });
@@ -6359 +6413 @@
         case op_jneq_ptr: {
             auto bytecode = currentInstruction->as<OpJneqPtr>();
-            FrozenValue* frozenPointer = m_graph.freezeStrong(m_inlineStackTop->m_codeBlock->getConstant(bytecode.m_specialPointer.offset()));
+            FrozenValue* frozenPointer = m_graph.freezeStrong(m_inlineStackTop->m_codeBlock->getConstant(bytecode.m_specialPointer));
             unsigned relativeOffset = jumpTarget(bytecode.m_targetLabel);
             Node* child = get(bytecode.m_value);
@@ -6827 +6881 @@
             auto bytecode = currentInstruction->as<OpCreateLexicalEnvironment>();
             ASSERT(bytecode.m_symbolTable.isConstant() && bytecode.m_initialValue.isConstant());
-            FrozenValue* symbolTable = m_graph.freezeStrong(m_inlineStackTop->m_codeBlock->getConstant(bytecode.m_symbolTable.offset()));
-            FrozenValue* initialValue = m_graph.freezeStrong(m_inlineStackTop->m_codeBlock->getConstant(bytecode.m_initialValue.offset()));
+            FrozenValue* symbolTable = m_graph.freezeStrong(m_inlineStackTop->m_codeBlock->getConstant(bytecode.m_symbolTable));
+            FrozenValue* initialValue = m_graph.freezeStrong(m_inlineStackTop->m_codeBlock->getConstant(bytecode.m_initialValue));
             Node* scope = get(bytecode.m_scope);
             Node* lexicalEnvironment = addToGraph(CreateActivation, OpInfo(symbolTable), OpInfo(initialValue), scope);
@@ -6858 +6912 @@
             // bytecode-level liveness of the scope register.
             auto bytecode = currentInstruction->as<OpGetScope>();
-            Node* callee = get(VirtualRegister(CallFrameSlot::callee));
+            Node* callee = get(CallFrameSlot::callee);
             Node* result;
             if (JSFunction* function = callee->dynamicCastConstant<JSFunction*>(*m_vm))
@@ -7283 +7337 @@
         // plan finishes.
         m_inlineCallFrame->baselineCodeBlock.setWithoutWriteBarrier(codeBlock->baselineVersion());
+        m_inlineCallFrame->setTmpOffset((m_caller->m_inlineCallFrame ? m_caller->m_inlineCallFrame->tmpOffset : 0) + m_caller->m_codeBlock->numTmps());
         m_inlineCallFrame->setStackOffset(inlineCallFrameStart.offset() - CallFrame::headerSizeInRegisters);
         m_inlineCallFrame->argumentCountIncludingThis = argumentCountIncludingThis;
+        RELEASE_ASSERT(m_inlineCallFrame->argumentCountIncludingThis == argumentCountIncludingThis);
         if (callee) {
             m_inlineCallFrame->calleeRecovery = ValueRecovery::constant(callee);
@@ -7646 +7702 @@
 
                     if (node->hasVariableAccessData(m_graph))
-                        mapping.operand(node->local()) = node->variableAccessData();
+                        mapping.operand(node->operand()) = node->variableAccessData();
 
                     if (node->op() != ForceOSRExit)
@@ -7666 +7722 @@
                     }
 
-                    auto insertLivenessPreservingOp = [&] (InlineCallFrame* inlineCallFrame, NodeType op, VirtualRegister operand) {
+                    auto insertLivenessPreservingOp = [&] (InlineCallFrame* inlineCallFrame, NodeType op, Operand operand) {
                         VariableAccessData* variable = mapping.operand(operand);
                         if (!variable) {
@@ -7673 +7729 @@
                         }
 
-                        VirtualRegister argument = operand - (inlineCallFrame ? inlineCallFrame->stackOffset : 0);
+                        Operand argument = unmapOperand(inlineCallFrame, operand);
                         if (argument.isArgument() && !argument.isHeader()) {
                             const Vector<ArgumentPosition*>& arguments = m_inlineCallFrameToArgumentPositions.get(inlineCallFrame);
@@ -7680 +7736 @@
                         insertionSet.insertNode(nodeIndex, SpecNone, op, origin, OpInfo(variable));
                     };
-                    auto addFlushDirect = [&] (InlineCallFrame* inlineCallFrame, VirtualRegister operand) {
+                    auto addFlushDirect = [&] (InlineCallFrame* inlineCallFrame, Operand operand) {
                         insertLivenessPreservingOp(inlineCallFrame, Flush, operand);
                     };
-                    auto addPhantomLocalDirect = [&] (InlineCallFrame* inlineCallFrame, VirtualRegister operand) {
+                    auto addPhantomLocalDirect = [&] (InlineCallFrame* inlineCallFrame, Operand operand) {
                         insertLivenessPreservingOp(inlineCallFrame, PhantomLocal, operand);
                     };
@@ -7748 +7804 @@
     }
 
+    m_graph.m_tmps = m_numTmps;
     m_graph.m_localVars = m_numLocals;
     m_graph.m_parameterSlots = m_parameterSlots;