Ignore:
Timestamp:
Oct 31, 2019, 3:32:52 PM (6 years ago)
Author:
Tadeu Zagallo
Message:

[WebAssembly] Create a Wasm interpreter
​https://bugs.webkit.org/show_bug.cgi?id=194257
<rdar://problem/44186794>

Reviewed by Saam Barati.

Source/JavaScriptCore:

Add an interpreter tier to WebAssembly which reuses the LLInt infrastructure. The interpreter
currently tiers up straight to OMG and can OSR enter at the prologue and from loops. The initial
implementation of the interpreter is very naive, but despite the lack of optimizations it still
shows a 2x improvement on the WebAssembly subtests in JetStream2 and 2x improvement on the
PSPDFKit benchmark. It reduces "compilation" times by ~3x and it's neutral on throughput.

The interpreter follows the same calling conventions as the BBQ/OMG, this means that:

  • We have to allocate locals for all argument registers and write all arguments registers to the stack in the prologue.
  • Calls have to allocate space for at least as many arguments as the number of argument registers. Before each call, all argument registers must be loaded from the stack, and after we return from the call, all registers must be stored back to the stack, in case they contain return values. We carefully layout the stack so that the arguments that would already have to be passed in the stack end up in the right place. The stack layout for calls is:

[ gprs ][ fprs ][ optional stack arguments ][ callee frame ]

sp

  • The return opcode has to load all registers from the stack, since they might need to contain results of the function.
  • The calling convention requires that the callee should store itself in the callee slot of the call frame, which is impossible in the interpreter, since the code we execute is the same for all callees. In order to work around that, we generate an entry thunk to the wasm interpreter for each function. All this thunk does is store the callee in the call frame and tail call the interpreter.
  • CMakeLists.txt:
  • DerivedSources-input.xcfilelist:
  • DerivedSources-output.xcfilelist:
  • DerivedSources.make:
  • JavaScriptCore.xcodeproj/project.pbxproj:
  • Sources.txt:
  • bytecode/BytecodeDumper.cpp:

(JSC::BytecodeDumper<Block>::constantName const):
(JSC::BytecodeDumper<Block>::dumpValue):
(JSC::BytecodeDumper<Block>::dumpBytecode):
(JSC::CodeBlockBytecodeDumper<Block>::vm const):
(JSC::CodeBlockBytecodeDumper<Block>::identifier const):
(JSC::CodeBlockBytecodeDumper<Block>::dumpIdentifiers):
(JSC::CodeBlockBytecodeDumper<Block>::dumpConstants):
(JSC::CodeBlockBytecodeDumper<Block>::dumpExceptionHandlers):
(JSC::CodeBlockBytecodeDumper<Block>::dumpSwitchJumpTables):
(JSC::CodeBlockBytecodeDumper<Block>::dumpStringSwitchJumpTables):
(JSC::CodeBlockBytecodeDumper<Block>::dumpBlock):

  • bytecode/BytecodeDumper.h:

(JSC::BytecodeDumper::dumpValue):
(JSC::BytecodeDumper::BytecodeDumper):

  • bytecode/BytecodeGeneratorification.cpp:

(JSC::performGeneratorification):

  • bytecode/BytecodeList.rb:
  • bytecode/CodeBlock.cpp:

(JSC::CodeBlock::dumpBytecode):

  • bytecode/Fits.h:
  • bytecode/Instruction.h:

(JSC::BaseInstruction::BaseInstruction):
(JSC::BaseInstruction::Impl::opcodeID const):
(JSC::BaseInstruction::opcodeID const):
(JSC::BaseInstruction::name const):
(JSC::BaseInstruction::isWide16 const):
(JSC::BaseInstruction::isWide32 const):
(JSC::BaseInstruction::hasMetadata const):
(JSC::BaseInstruction::sizeShiftAmount const):
(JSC::BaseInstruction::size const):
(JSC::BaseInstruction::is const):
(JSC::BaseInstruction::as const):
(JSC::BaseInstruction::cast):
(JSC::BaseInstruction::cast const):
(JSC::BaseInstruction::wide16 const):
(JSC::BaseInstruction::wide32 const):

  • bytecode/InstructionStream.h:

(JSC::InstructionStream::iterator::operator+=):
(JSC::InstructionStream::iterator::operator++):
(JSC::InstructionStreamWriter::iterator::operator+=):
(JSC::InstructionStreamWriter::iterator::operator++):

  • bytecode/Opcode.cpp:
  • bytecode/Opcode.h:
  • bytecode/PreciseJumpTargetsInlines.h:
  • bytecode/UnlinkedCodeBlock.h:
  • bytecode/VirtualRegister.cpp:

(JSC::VirtualRegister::VirtualRegister):

  • bytecode/VirtualRegister.h:
  • bytecompiler/BytecodeGenerator.cpp:

(JSC::GenericLabel<JSGeneratorTraits>::setLocation):
(JSC::BytecodeGenerator::BytecodeGenerator):

  • bytecompiler/BytecodeGenerator.h:
  • bytecompiler/BytecodeGeneratorBase.h: Added.
  • bytecompiler/BytecodeGeneratorBaseInlines.h: Added.

(JSC::shrinkToFit):
(JSC::BytecodeGeneratorBase<Traits>::BytecodeGeneratorBase):
(JSC::BytecodeGeneratorBase<Traits>::newLabel):
(JSC::BytecodeGeneratorBase<Traits>::newEmittedLabel):
(JSC::BytecodeGeneratorBase<Traits>::reclaimFreeRegisters):
(JSC::BytecodeGeneratorBase<Traits>::emitLabel):
(JSC::BytecodeGeneratorBase<Traits>::recordOpcode):
(JSC::BytecodeGeneratorBase<Traits>::alignWideOpcode16):
(JSC::BytecodeGeneratorBase<Traits>::alignWideOpcode32):
(JSC::BytecodeGeneratorBase<Traits>::write):
(JSC::BytecodeGeneratorBase<Traits>::newRegister):
(JSC::BytecodeGeneratorBase<Traits>::newTemporary):
(JSC::BytecodeGeneratorBase<Traits>::addVar):
(JSC::BytecodeGeneratorBase<Traits>::allocateCalleeSaveSpace):

  • bytecompiler/Label.h:

(JSC::GenericBoundLabel::GenericBoundLabel):
(JSC::GenericBoundLabel::target):
(JSC::GenericBoundLabel::saveTarget):
(JSC::GenericBoundLabel::commitTarget):

  • dfg/DFGByteCodeParser.cpp:
  • dfg/DFGCapabilities.cpp:

(JSC::DFG::capabilityLevel):

  • dfg/DFGOperations.cpp:
  • generator/Argument.rb:
  • generator/DSL.rb:
  • generator/GeneratedFile.rb:
  • generator/Opcode.rb:
  • generator/Options.rb:
  • generator/Section.rb:
  • generator/Wasm.rb: Added.
  • interpreter/Register.h:
  • interpreter/RegisterInlines.h:

(JSC::Register::operator=):

  • jit/JITArithmetic.cpp:
  • jit/JITOpcodes.cpp:
  • llint/LLIntData.cpp:

(JSC::LLInt::initialize):

  • llint/LLIntData.h:

(JSC::LLInt::wasmExceptionInstructions):

  • llint/LLIntOfflineAsmConfig.h:
  • llint/LLIntOffsetsExtractor.cpp:
  • llint/LLIntSlowPaths.cpp:
  • llint/LLIntThunks.cpp:

(JSC::LLInt::generateThunkWithJumpTo):
(JSC::LLInt::wasmFunctionEntryThunk):

  • llint/LLIntThunks.h:
  • llint/LowLevelInterpreter.asm:
  • llint/LowLevelInterpreter32_64.asm:
  • llint/LowLevelInterpreter64.asm:
  • llint/WebAssembly.asm: Added.
  • offlineasm/arm64.rb:
  • offlineasm/instructions.rb:
  • offlineasm/parser.rb:
  • offlineasm/registers.rb:
  • offlineasm/transform.rb:
  • offlineasm/x86.rb:
  • parser/Nodes.h:
  • runtime/Error.cpp:

(JSC::FindFirstCallerFrameWithCodeblockFunctor::operator() const):

  • runtime/ErrorInstance.cpp:

(JSC::ErrorInstance::finishCreation):

  • runtime/Options.cpp:

(JSC::overrideDefaults):

  • runtime/OptionsList.h:
  • runtime/SamplingProfiler.cpp:

(JSC::FrameWalker::recordJITFrame):
(JSC::FrameWalker::resetAtMachineFrame):

  • wasm/WasmAirIRGenerator.cpp:

(JSC::Wasm::AirIRGenerator::isControlTypeIf):
(JSC::Wasm::AirIRGenerator::emitLoopTierUpCheck):

  • wasm/WasmB3IRGenerator.cpp:

(JSC::Wasm::B3IRGenerator::isControlTypeIf):

  • wasm/WasmBBQPlan.cpp:

(JSC::Wasm::BBQPlan::prepareImpl):
(JSC::Wasm::BBQPlan::work):
(JSC::Wasm::BBQPlan::compileFunction):
(JSC::Wasm::BBQPlan::didCompleteCompilation):
(JSC::Wasm::BBQPlan::initializeCallees):

  • wasm/WasmBBQPlan.h:
  • wasm/WasmBBQPlanInlines.h: Removed.
  • wasm/WasmCallee.cpp:

(JSC::Wasm::Callee::Callee):
(JSC::Wasm::Callee::dump const):
(JSC::Wasm::JITCallee::JITCallee):
(JSC::Wasm::LLIntCallee::setEntrypoint):
(JSC::Wasm::LLIntCallee::entrypoint const):
(JSC::Wasm::LLIntCallee::calleeSaveRegisters):
(JSC::Wasm:: const):

  • wasm/WasmCallee.h:

(JSC::Wasm::Callee::setOSREntryCallee):
(JSC::Wasm::JITCallee::wasmToWasmCallsites):
(JSC::Wasm::JITCallee:: const):

  • wasm/WasmCallingConvention.h:
  • wasm/WasmCodeBlock.cpp:

(JSC::Wasm::CodeBlock::CodeBlock):

  • wasm/WasmCodeBlock.h:

(JSC::Wasm::CodeBlock::wasmEntrypointCalleeFromFunctionIndexSpace):
(JSC::Wasm::CodeBlock::wasmBBQCalleeFromFunctionIndexSpace):
(JSC::Wasm::CodeBlock::wasmToWasmExitStub):

  • wasm/WasmCompilationMode.cpp:

(JSC::Wasm::makeString):

  • wasm/WasmCompilationMode.h:
  • wasm/WasmEmbedder.h:
  • wasm/WasmEntryPlan.cpp: Added.

(JSC::Wasm::EntryPlan::EntryPlan):
(JSC::Wasm::EntryPlan::stateString):
(JSC::Wasm::EntryPlan::moveToState):
(JSC::Wasm::EntryPlan::didReceiveFunctionData):
(JSC::Wasm::EntryPlan::parseAndValidateModule):
(JSC::Wasm::EntryPlan::prepare):
(JSC::Wasm::EntryPlan::ThreadCountHolder::ThreadCountHolder):
(JSC::Wasm::EntryPlan::ThreadCountHolder::~ThreadCountHolder):
(JSC::Wasm::EntryPlan::complete):
(JSC::Wasm::EntryPlan::compileFunctions):
(JSC::Wasm::EntryPlan::work):

  • wasm/WasmEntryPlan.h: Copied from Source/JavaScriptCore/wasm/WasmBBQPlan.h.

(JSC::Wasm::EntryPlan::parseAndValidateModule):
(JSC::Wasm::EntryPlan::exports const):
(JSC::Wasm::EntryPlan::internalFunctionCount const):
(JSC::Wasm::EntryPlan::takeModuleInformation):
(JSC::Wasm::EntryPlan::takeWasmToWasmExitStubs):
(JSC::Wasm::EntryPlan::takeWasmToWasmCallsites):
(JSC::Wasm::EntryPlan::hasBeenPrepared const):
(JSC::Wasm::EntryPlan::tryReserveCapacity):

  • wasm/WasmFunctionCodeBlock.cpp: Added.

(JSC::Wasm::FunctionCodeBlock::setInstructions):
(JSC::Wasm::FunctionCodeBlock::dumpBytecode):
(JSC::Wasm::FunctionCodeBlock::addOutOfLineJumpTarget):
(JSC::Wasm::FunctionCodeBlock::outOfLineJumpOffset):
(JSC::Wasm::FunctionCodeBlock::outOfLineJumpTarget):
(JSC::Wasm::FunctionCodeBlock::addSignature):
(JSC::Wasm::FunctionCodeBlock::signature const):
(JSC::Wasm::FunctionCodeBlock::addJumpTable):
(JSC::Wasm::FunctionCodeBlock::jumpTable const const):
(JSC::Wasm::FunctionCodeBlock::numberOfJumpTables const):

  • wasm/WasmFunctionCodeBlock.h: Added.

(JSC::Wasm::FunctionCodeBlock::FunctionCodeBlock):
(JSC::Wasm::FunctionCodeBlock::getConstant const):
(JSC::Wasm::FunctionCodeBlock::functionIndex const):
(JSC::Wasm::FunctionCodeBlock::addJumpTarget):
(JSC::Wasm::FunctionCodeBlock::numberOfJumpTargets):
(JSC::Wasm::FunctionCodeBlock::lastJumpTarget):
(JSC::Wasm::FunctionCodeBlock::outOfLineJumpOffset):
(JSC::Wasm::FunctionCodeBlock::bytecodeOffset):
(JSC::Wasm::FunctionCodeBlock::tierUpCounter):

  • wasm/WasmFunctionParser.h:

(JSC::Wasm::FunctionParser<Context>::parseExpression):
(JSC::Wasm::FunctionParser<Context>::parseUnreachableExpression):

  • wasm/WasmInstance.h:
  • wasm/WasmLLIntGenerator.cpp: Added.

(JSC::Wasm::LLIntGenerator::ControlType::ControlType):
(JSC::Wasm::LLIntGenerator::ControlType::loop):
(JSC::Wasm::LLIntGenerator::ControlType::topLevel):
(JSC::Wasm::LLIntGenerator::ControlType::block):
(JSC::Wasm::LLIntGenerator::ControlType::if_):
(JSC::Wasm::LLIntGenerator::ControlType::targetLabelForBranch const):
(JSC::Wasm::LLIntGenerator::fail const):
(JSC::Wasm::LLIntGenerator::unifyValuesWithBlock):
(JSC::Wasm::LLIntGenerator::emptyExpression):
(JSC::Wasm::LLIntGenerator::createStack):
(JSC::Wasm::LLIntGenerator::isControlTypeIf):
(JSC::Wasm::LLIntGenerator::addEndToUnreachable):
(JSC::Wasm::LLIntGenerator::setParser):
(JSC::Wasm::LLIntGenerator::dump):
(JSC::Wasm::LLIntGenerator::virtualRegisterForLocal):
(JSC::Wasm::LLIntGenerator::tmpsForSignature):
(JSC::Wasm::LLIntGenerator::jsNullConstant):
(JSC::Wasm::LLIntGenerator::isConstant):
(JSC::Wasm::parseAndCompileBytecode):
(JSC::Wasm::LLIntGenerator::LLIntGenerator):
(JSC::Wasm::LLIntGenerator::finalize):
(JSC::Wasm::LLIntGenerator::callInformationFor):
(JSC::Wasm::LLIntGenerator::addArguments):
(JSC::Wasm::LLIntGenerator::addLocal):
(JSC::Wasm::LLIntGenerator::addConstant):
(JSC::Wasm::LLIntGenerator::getLocal):
(JSC::Wasm::LLIntGenerator::setLocal):
(JSC::Wasm::LLIntGenerator::getGlobal):
(JSC::Wasm::LLIntGenerator::setGlobal):
(JSC::Wasm::LLIntGenerator::addLoop):
(JSC::Wasm::LLIntGenerator::addTopLevel):
(JSC::Wasm::LLIntGenerator::addBlock):
(JSC::Wasm::LLIntGenerator::addIf):
(JSC::Wasm::LLIntGenerator::addElse):
(JSC::Wasm::LLIntGenerator::addElseToUnreachable):
(JSC::Wasm::LLIntGenerator::addReturn):
(JSC::Wasm::LLIntGenerator::addBranch):
(JSC::Wasm::LLIntGenerator::addSwitch):
(JSC::Wasm::LLIntGenerator::endBlock):
(JSC::Wasm::LLIntGenerator::addCall):
(JSC::Wasm::LLIntGenerator::addCallIndirect):
(JSC::Wasm::LLIntGenerator::addRefIsNull):
(JSC::Wasm::LLIntGenerator::addRefFunc):
(JSC::Wasm::LLIntGenerator::addTableGet):
(JSC::Wasm::LLIntGenerator::addTableSet):
(JSC::Wasm::LLIntGenerator::addTableSize):
(JSC::Wasm::LLIntGenerator::addTableGrow):
(JSC::Wasm::LLIntGenerator::addTableFill):
(JSC::Wasm::LLIntGenerator::addUnreachable):
(JSC::Wasm::LLIntGenerator::addCurrentMemory):
(JSC::Wasm::LLIntGenerator::addGrowMemory):
(JSC::Wasm::LLIntGenerator::addSelect):
(JSC::Wasm::LLIntGenerator::load):
(JSC::Wasm::LLIntGenerator::store):
(JSC::GenericLabel<Wasm::GeneratorTraits>::setLocation):

  • wasm/WasmLLIntGenerator.h: Copied from Source/JavaScriptCore/wasm/WasmCompilationMode.h.
  • wasm/WasmLLIntPlan.cpp: Added.

(JSC::Wasm::LLIntPlan::prepareImpl):
(JSC::Wasm::LLIntPlan::compileFunction):
(JSC::Wasm::LLIntPlan::didCompleteCompilation):
(JSC::Wasm::LLIntPlan::initializeCallees):

  • wasm/WasmLLIntPlan.h: Copied from Source/JavaScriptCore/wasm/WasmOMGForOSREntryPlan.h.
  • wasm/WasmLLIntTierUpCounter.cpp: Copied from Source/JavaScriptCore/wasm/WasmCompilationMode.cpp.

(JSC::Wasm::LLIntTierUpCounter::addOSREntryDataForLoop):
(JSC::Wasm::LLIntTierUpCounter::osrEntryDataForLoop const const):

  • wasm/WasmLLIntTierUpCounter.h: Copied from Source/JavaScriptCore/wasm/WasmOMGForOSREntryPlan.h.

(JSC::Wasm::LLIntTierUpCounter::LLIntTierUpCounter):
(JSC::Wasm::LLIntTierUpCounter::optimizeAfterWarmUp):
(JSC::Wasm::LLIntTierUpCounter::checkIfOptimizationThresholdReached):
(JSC::Wasm::LLIntTierUpCounter::optimizeSoon):

  • wasm/WasmMemoryInformation.cpp:

(JSC::Wasm::PinnedRegisterInfo::get):

  • wasm/WasmModule.cpp:

(JSC::Wasm::makeValidationResult):
(JSC::Wasm::makeValidationCallback):
(JSC::Wasm::Module::validateSync):
(JSC::Wasm::Module::validateAsync):

  • wasm/WasmOMGForOSREntryPlan.cpp:

(JSC::Wasm::OMGForOSREntryPlan::OMGForOSREntryPlan):
(JSC::Wasm::OMGForOSREntryPlan::work):

  • wasm/WasmOMGForOSREntryPlan.h:
  • wasm/WasmOMGPlan.cpp:

(JSC::Wasm::OMGPlan::work):

  • wasm/WasmSlowPaths.cpp: Added.

(JSC::LLInt::jitCompileAndSetHeuristics):
(JSC::LLInt::WASM_SLOW_PATH_DECL):
(JSC::LLInt::doWasmCall):
(JSC::LLInt::doWasmCallIndirect):
(JSC::LLInt::slow_path_wasm_throw_exception):
(JSC::LLInt::slow_path_wasm_popcount):
(JSC::LLInt::slow_path_wasm_popcountll):

  • wasm/WasmSlowPaths.h: Added.
  • wasm/WasmTable.cpp:

(JSC::Wasm::FuncRefTable::function const):
(JSC::Wasm::FuncRefTable::instance const):

  • wasm/WasmTable.h:
  • wasm/WasmTierUpCount.h:
  • wasm/WasmValidate.cpp:

(JSC::Wasm::Validate::isControlTypeIf):

  • wasm/js/JSToWasm.cpp:

(JSC::Wasm::createJSToWasmWrapper):

  • wasm/js/JSToWasm.h:
  • wasm/js/WebAssemblyFunction.cpp:

(JSC::WebAssemblyFunction::calleeSaves const):

Tools:

Add a mode that runs WebAssembly tests without the LLInt (i.e. only Air)
and update the no-air mode to also disable the LLInt tier.

  • Scripts/run-jsc-stress-tests:
File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/offlineasm/arm64.rb

    r250094 r251886  
    8989
    9090def arm64FPRName(name, kind)
    91     raise "bad FPR kind #{kind}" unless kind == :double
    9291    raise "bad FPR name #{name}" unless name =~ /^q/
    93     "d" + name[1..-1]
     92    case kind
     93    when :double
     94        "d" + name[1..-1]
     95    when :float
     96        "s" + name[1..-1]
     97    when :vector
     98        "v" + name[1..-1]
     99    else
     100        raise "bad FPR kind #{kind}"
     101    end
    94102end
    95103
    … …  
    113121    def arm64Operand(kind)
    114122        case @name
    115         when 't0', 'a0', 'r0'
     123        when 't0', 'a0', 'r0', 'wa0'
    116124            arm64GPRName('x0', kind)
    117         when 't1', 'a1', 'r1'
     125        when 't1', 'a1', 'r1', 'wa1'
    118126            arm64GPRName('x1', kind)
    119         when 't2', 'a2'
     127        when 't2', 'a2', 'wa2'
    120128            arm64GPRName('x2', kind)
    121         when 't3', 'a3'
     129        when 't3', 'a3', 'wa3'
    122130            arm64GPRName('x3', kind)
    123         when 't4'
     131        when 't4', 'wa4'
    124132            arm64GPRName('x4', kind)
    125         when 't5'
     133        when 't5', 'wa5'
    126134          arm64GPRName('x5', kind)
    127         when 't6'
     135        when 't6', 'wa6'
    128136          arm64GPRName('x6', kind)
    129         when 't7'
     137        when 't7', 'wa7'
    130138          arm64GPRName('x7', kind)
     139        when 'ws0'
     140          arm64GPRName('x9', kind)
     141        when 'ws1'
     142          arm64GPRName('x10', kind)
    131143        when 'cfr'
    132144            arm64GPRName('x29', kind)
    … …  
    164176    def arm64Operand(kind)
    165177        case @name
    166         when 'ft0', 'fr', 'fa0'
     178        when 'ft0', 'fr', 'fa0', 'wfa0'
    167179            arm64FPRName('q0', kind)
    168         when 'ft1', 'fa1'
     180        when 'ft1', 'fa1', 'wfa1'
    169181            arm64FPRName('q1', kind)
    170         when 'ft2', 'fa2'
     182        when 'ft2', 'fa2', 'wfa2'
    171183            arm64FPRName('q2', kind)
    172         when 'ft3', 'fa3'
     184        when 'ft3', 'fa3', 'wfa3'
    173185            arm64FPRName('q3', kind)
    174         when 'ft4'
     186        when 'ft4', 'wfa4'
    175187            arm64FPRName('q4', kind)
    176         when 'ft5'
     188        when 'ft5', 'wfa5'
    177189            arm64FPRName('q5', kind)
     190        when 'wfa6'
     191            arm64FPRName('q6', kind)
     192        when 'wfa7'
     193            arm64FPRName('q7', kind)
    178194        when 'csfr0'
    179195            arm64FPRName('q8', kind)
    … …  
    379395            when "loadb", "loadbsi", "loadbsq", "storeb", /^bb/, /^btb/, /^cb/, /^tb/
    380396                size = 1
    381             when "loadh", "loadhsi", "loadhsq"
     397            when "loadh", "loadhsi", "loadhsq", "storeh"
    382398                size = 2
    383399            when "loadi", "loadis", "storei", "addi", "andi", "lshifti", "muli", "negi",
    384400                "noti", "ori", "rshifti", "urshifti", "subi", "xori", /^bi/, /^bti/,
    385                 /^ci/, /^ti/, "addis", "subis", "mulis", "smulli", "leai"
     401                /^ci/, /^ti/, "addis", "subis", "mulis", "smulli", "leai", "loadf", "storef"
    386402                size = 4
    387403            when "loadp", "storep", "loadq", "storeq", "loadd", "stored", "lshiftp", "lshiftq", "negp", "negq", "rshiftp", "rshiftq",
    … …  
    532548end
    533549
     550def emitARM64Div(opcode, operands, kind)
     551    if operands.size == 2
     552        operands = [operands[1], operands[1], operands[0]]
     553    elsif operands.size == 3
     554        operands = [operands[2], operands[1], operands[0]]
     555    else
     556        raise
     557    end
     558    $asm.puts "#{opcode} #{arm64Operands(operands, kind)}"
     559end
     560
     561def emitARM64TACWithOperandSuffix(opcode, operands, kind)
     562    raise unless [:float, :double].include? kind
     563    size = kind == :float ? 8 : 16
     564    operands = operands.map { |operand|
     565        raise unless operand.is_a? FPRegisterID
     566        "#{operand.arm64Operand(:vector)}.#{size}b"
     567    }
     568    if operands.length == 2
     569      operands = [operands[1], operands[1], operands[0]]
     570    else
     571      raise unless operands.length == 3
     572      operands = [operands[2], operands[0], operands[1]]
     573    end
     574    $asm.puts "#{opcode} #{operands.join(", ")}"
     575end
     576
    534577def emitARM64(opcode, operands, kind)
    535578    $asm.puts "#{opcode} #{arm64FlippedOperands(operands, kind)}"
    … …  
    580623    emitARM64Unflipped(opcode, operands[0..-2], kind)
    581624    $asm.puts "#{branchOpcode} #{operands[-1].asmLabel}"
     625end
     626
     627def emitARM64CompareFP(operands, kind, compareCode)
     628    emitARM64Unflipped("fcmp", operands[0..-2], kind)
     629    $asm.puts "cset #{operands[-1].arm64Operand(:word)}, #{compareCode}"
    582630end
    583631
    … …  
    638686        when "xorq"
    639687            emitARM64TAC("eor", operands, :quad)
     688        when 'divi'
     689            emitARM64Div("udiv", operands, :word)
     690        when 'divis'
     691            emitARM64Div("sdiv", operands, :word)
     692        when 'divq'
     693            emitARM64Div("udiv", operands, :quad)
     694        when 'divqs'
     695            emitARM64Div("sdiv", operands, :quad)
    640696        when "lshifti"
    641697            emitARM64LShift(operands, :word)
    … …  
    740796        when "sqrtd"
    741797            emitARM64("fsqrt", operands, :double)
    742         when "ci2d"
    743             emitARM64("scvtf", operands, [:word, :double])
    744798        when "bdeq"
    745799            emitARM64Branch("fcmp", operands, :double, "b.eq")
    … …  
    749803            $asm.puts "b.vs #{LocalLabelReference.new(codeOrigin, isUnordered).asmLabel}"
    750804            $asm.puts "b.ne #{operands[2].asmLabel}"
    751             isUnordered.lower("ARM64")
     805            isUnordered.lower($activeBackend)
    752806        when "bdgt"
    753807            emitARM64Branch("fcmp", operands, :double, "b.gt")
    … …  
    810864                emitARM64("mov", operands, :quad)
    811865            end
     866        when "moved"
     867            emitARM64("fmov", operands, :double)
    812868        when "sxi2p"
    813869            emitARM64("sxtw", operands, [:word, :ptr])
    … …  
    10571113                $asm.putStr("#endif")
    10581114            }
     1115
     1116        when "andf", "andd"
     1117            emitARM64TACWithOperandSuffix("and", operands, :double)
     1118        when "orf", "ord"
     1119            emitARM64TACWithOperandSuffix("orr", operands, :double)
     1120        when "lrotatei"
     1121            tmp = Tmp.new(codeOrigin, :gpr)
     1122            Sequence.new(codeOrigin, [
     1123                Instruction.new(codeOrigin, "move", [operands[0], tmp]),
     1124                Instruction.new(codeOrigin, "negq", [tmp]),
     1125                Instruction.new(codeOrigin, "rrotatei", [tmp, operands[1]]),
     1126            ]).lower($activeBackend)
     1127        when "lrotateq"
     1128            tmp = Tmp.new(codeOrigin, :gpr)
     1129            Sequence.new(codeOrigin, [
     1130                Instruction.new(codeOrigin, "move", [operands[0], tmp]),
     1131                Instruction.new(codeOrigin, "negq", [tmp]),
     1132                Instruction.new(codeOrigin, "rrotateq", [tmp, operands[1]]),
     1133            ]).lower($activeBackend)
     1134        when "rrotatei"
     1135            emitARM64TAC("ror", operands, :word)
     1136        when "rrotateq"
     1137            emitARM64TAC("ror", operands, :quad)
     1138        when "loadf"
     1139            emitARM64Access("ldr", "ldur", operands[1], operands[0], :float)
     1140        when "storef"
     1141            emitARM64Unflipped("str", operands, :float)
     1142        when "addf"
     1143            emitARM64TAC("fadd", operands, :float)
     1144        when "divf"
     1145            emitARM64TAC("fdiv", operands, :float)
     1146        when "subf"
     1147            emitARM64TAC("fsub", operands, :float)
     1148        when "mulf"
     1149            emitARM64TAC("fmul", operands, :float)
     1150        when "sqrtf"
     1151            emitARM64("fsqrt", operands, :float)
     1152        when "floorf"
     1153            emitARM64("frintm", operands, :float)
     1154        when "floord"
     1155            emitARM64("frintm", operands, :double)
     1156        when "roundf"
     1157            emitARM64("frintn", operands, :float)
     1158        when "roundd"
     1159            emitARM64("frintn", operands, :double)
     1160        when "truncatef"
     1161            emitARM64("frintz", operands, :float)
     1162        when "truncated"
     1163            emitARM64("frintz", operands, :double)
     1164        when "truncatef2i"
     1165            emitARM64("fcvtzu", operands, [:float, :word])
     1166        when "truncatef2q"
     1167            emitARM64("fcvtzu", operands, [:float, :quad])
     1168        when "truncated2q"
     1169            emitARM64("fcvtzu", operands, [:double, :quad])
     1170        when "truncated2i"
     1171            emitARM64("fcvtzu", operands, [:double, :word])
     1172        when "truncatef2is"
     1173            emitARM64("fcvtzs", operands, [:float, :word])
     1174        when "truncated2is"
     1175            emitARM64("fcvtzs", operands, [:double, :word])
     1176        when "truncatef2qs"
     1177            emitARM64("fcvtzs", operands, [:float, :quad])
     1178        when "truncated2qs"
     1179            emitARM64("fcvtzs", operands, [:double, :quad])
     1180        when "ci2d"
     1181            emitARM64("ucvtf", operands, [:word, :double])
     1182        when "ci2ds"
     1183            emitARM64("scvtf", operands, [:word, :double])
     1184        when "ci2f"
     1185            emitARM64("ucvtf", operands, [:word, :float])
     1186        when "ci2fs"
     1187            emitARM64("scvtf", operands, [:word, :float])
     1188        when "cq2f"
     1189            emitARM64("ucvtf", operands, [:quad, :float])
     1190        when "cq2fs"
     1191            emitARM64("scvtf", operands, [:quad, :float])
     1192        when "cq2d"
     1193            emitARM64("ucvtf", operands, [:quad, :double])
     1194        when "cq2ds"
     1195            emitARM64("scvtf", operands, [:quad, :double])
     1196        when "cd2f"
     1197            emitARM64("fcvt", operands, [:double, :float])
     1198        when "cf2d"
     1199            emitARM64("fcvt", operands, [:float, :double])
     1200        when "bfeq"
     1201            emitARM64Branch("fcmp", operands, :float, "b.eq")
     1202        when "bfgt"
     1203            emitARM64Branch("fcmp", operands, :float, "b.gt")
     1204        when "bflt"
     1205            emitARM64Branch("fcmp", operands, :float, "b.mi")
     1206        when "bfgtun"
     1207            emitARM64Branch("fcmp", operands, :float, "b.hi")
     1208        when "bfgtequn"
     1209            emitARM64Branch("fcmp", operands, :float, "b.pl")
     1210        when "bfltun"
     1211            emitARM64Branch("fcmp", operands, :float, "b.lt")
     1212        when "bfltequn"
     1213            emitARM64Branch("fcmp", operands, :float, "b.le")
     1214        when "tzcnti"
     1215            emitARM64("rbit", operands, :word)
     1216            emitARM64("clz", [operands[1], operands[1]], :word)
     1217        when "tzcntq"
     1218            emitARM64("rbit", operands, :quad)
     1219            emitARM64("clz", [operands[1], operands[1]], :quad)
     1220        when "lzcnti"
     1221            emitARM64("clz", operands, :word)
     1222        when "lzcntq"
     1223            emitARM64("clz", operands, :quad)
     1224        when "absf"
     1225            emitARM64("fabs", operands, :float)
     1226        when "absd"
     1227            emitARM64("fabs", operands, :double)
     1228        when "negf"
     1229            emitARM64("fneg", operands, :float)
     1230        when "negd"
     1231            emitARM64("fneg", operands, :double)
     1232        when "ceilf"
     1233            emitARM64("frintp", operands, :float)
     1234        when "ceild"
     1235            emitARM64("frintp", operands, :double)
     1236        when "cfeq"
     1237            emitARM64CompareFP(operands, :float, "eq")
     1238        when "cdeq"
     1239            emitARM64CompareFP(operands, :double, "eq")
     1240        when "cfneq"
     1241            $asm.puts "move $0, #{operands[2].arm64Operand(:word)}"
     1242            emitARM64Unflipped("fcmp", operands[0..1], :float)
     1243            isUnordered = LocalLabel.unique("cdneq")
     1244            $asm.puts "b.vs #{LocalLabelReference.new(codeOrigin, isUnordered).asmLabel}"
     1245            $asm.puts "cset #{operands[2].arm64Operand(:word)}, ne"
     1246            isUnordered.lower($activeBackend)
     1247        when "cdneq"
     1248            $asm.puts "move $0, #{operands[2].arm64Operand(:word)}"
     1249            emitARM64Unflipped("fcmp", operands[0..1], :double)
     1250            isUnordered = LocalLabel.unique("cdneq")
     1251            $asm.puts "b.vs #{LocalLabelReference.new(codeOrigin, isUnordered).asmLabel}"
     1252            $asm.puts "cset #{operands[2].arm64Operand(:word)}, ne"
     1253            isUnordered.lower($activeBackend)
     1254        when "cfnequn"
     1255            emitARM64CompareFP(operands, :float, "ne")
     1256        when "cdnequn"
     1257            emitARM64CompareFP(operands, :double, "ne")
     1258        when "cflt"
     1259            emitARM64CompareFP(operands, :float, "mi")
     1260        when "cdlt"
     1261            emitARM64CompareFP(operands, :double, "mi")
     1262        when "cflteq"
     1263            emitARM64CompareFP(operands, :float, "ls")
     1264        when "cdlteq"
     1265            emitARM64CompareFP(operands, :double, "ls")
     1266        when "cfgt"
     1267            emitARM64CompareFP(operands, :float, "gt")
     1268        when "cdgt"
     1269            emitARM64CompareFP(operands, :double, "gt")
     1270        when "cfgteq"
     1271            emitARM64CompareFP(operands, :float, "ge")
     1272        when "cdgteq"
     1273            emitARM64CompareFP(operands, :double, "ge")
     1274        when "fi2f"
     1275            emitARM64("fmov", operands, [:word, :float])
     1276        when "ff2i"
     1277            emitARM64("fmov", operands, [:float, :word])
     1278        when "tls_loadp"
     1279            tmp = ARM64_EXTRA_GPRS[0].arm64Operand(:ptr)
     1280            if operands[0].immediate?
     1281              offset = "\##{operands[0].value * 8}"
     1282            else
     1283              offset = operands[0].arm64Operand(:word)
     1284            end
     1285            $asm.puts "mrs #{tmp}, tpidrro_el0"
     1286            $asm.puts "bic #{tmp}, #{tmp}, #7"
     1287            $asm.puts "ldr #{operands[1].arm64Operand(:ptr)}, [#{tmp}, #{offset}]"
     1288        when "tls_storep"
     1289            tmp = ARM64_EXTRA_GPRS[0].arm64Operand(:ptr)
     1290            if operands[1].immediate?
     1291              offset = "\##{operands[1].value * 8}"
     1292            else
     1293              offset = operands[1].arm64Operand(:word)
     1294            end
     1295            $asm.puts "mrs #{tmp}, tpidrro_el0"
     1296            $asm.puts "bic #{tmp}, #{tmp}, #7"
     1297            $asm.puts "str #{operands[0].arm64Operand(:ptr)}, [#{tmp}, #{offset}]"
    10591298        else
    10601299            lowerDefault
Note: See TracChangeset for help on using the changeset viewer.