Ignore:
Timestamp:
Aug 20, 2017, 9:26:40 PM (8 years ago)
Author:
[email protected]
Message:

Enhance MacroAssembler::probe() to allow the probe function to resize the stack frame and alter stack data in one pass.
​https://bugs.webkit.org/show_bug.cgi?id=175688
<rdar://problem/33436870>

Reviewed by JF Bastien.

With this patch, the clients of the MacroAssembler::probe() can now change
stack values without having to worry about whether there is enough room in the
current stack frame for it or not. This is done using the Probe::Context's stack
member like so:

jit.probe([] (Probe::Context& context) {

auto cpu = context.cpu;
auto stack = context.stack();
uintptr_t* currentSP = cpu.sp<uintptr_t*>();

Get a value at the current stack pointer location.
auto value = stack.get<uintptr_t>(currentSP);

Set a value above the current stack pointer (within current frame).
stack.set<uintptr_t>(currentSP + 10, value);

Set a value below the current stack pointer (out of current frame).
stack.set<uintptr_t>(currentSP - 10, value);

Set the new stack pointer.
cpu.sp() = currentSP - 20;

});

What happens behind the scene:

  1. the generated JIT probe code will now call Probe::executeProbe(), and Probe::executeProbe() will in turn call the client's probe function.

Probe::executeProbe() receives the Probe::State on the machine stack passed
to it by the probe trampoline. Probe::executeProbe() will instantiate a
Probe::Context to be passed to the client's probe function. The client will
no longer see the Probe::State directly.

  1. The Probe::Context comes with a Probe::Stack which serves as a manager of stack pages. Currently, each page is 1K in size. Probe::Context::stack() returns a reference to an instance of Probe::Stack.
  1. Invoking get() of set() on Probe::Stack with an address will lead to the following:
  1. the address will be decoded to a baseAddress that points to the 1K page that contains that address.
  1. the Probe::Stack will check if it already has a cached 1K page for that baseAddress. If so, go to step (f). Else, continue with step (c).
  1. the Probe::Stack will malloc a 1K mirror page, and memcpy the 1K stack page for that specified baseAddress to this mirror page.
  1. the mirror page will be added to the ProbeStack's m_pages HashMap, keyed on the baseAddress.
  1. the ProbeStack will also cache the last baseAddress and its corresponding mirror page in use. With memory accesses tending to be localized, this will save us from having to look up the page in the HashMap.
  1. get() will map the requested address to a physical address in the mirror page, and return the value at that location.
  1. set() will map the requested address to a physical address in the mirror page, and set the value at that location in the mirror page.

set() will also set a dirty bit corresponding to the "cache line" that
was modified in the mirror page.

  1. When the client's probe function returns, Probe::executeProbe() will check if there are stack changes that need to be applied. If stack changes are needed:
  1. Probe::executeProbe() will adjust the stack pointer to ensure enough stack space is available to flush the dirty stack pages. It will also register a flushStackDirtyPages callback function in the Probe::State. Thereafter, Probe::executeProbe() returns to the probe trampoline.
  1. the probe trampoline adjusts the stack pointer, moves the Probe::State to a safe place if needed, and then calls the flushStackDirtyPages callback if needed.
  1. the flushStackDirtyPages() callback iterates the Probe::Stack's m_pages HashMap and flush all dirty "cache lines" to the machine stack. Thereafter, flushStackDirtyPages() returns to the probe trampoline.
  1. lastly, the probe trampoline will restore all register values and return to the pc set in the Probe::State.

To make this patch work, I also had to do the following work:

  1. Refactor MacroAssembler::CPUState into Probe::CPUState. Mainly, this means moving the code over to ProbeContext.h. I also added some convenience accessor methods for spr registers.

Moved Probe::Context over to its own file ProbeContext.h/cpp.

  1. Fix all probe trampolines to pass the address of Probe::executeProbe in addition to the client's probe function and arg.

I also took this opportunity to optimize the generated JIT probe code to
minimize the amount of memory stores needed.

  1. Simplified the ARM64 probe trampoline. The ARM64 probe only supports changing either lr or pc (or neither), but not both at in the same probe invocation. The ARM64 probe trampoline used to have to check for this invariant in the assembly trampoline code. With the introduction of Probe::executeProbe(), we can now do it there and simplify the trampoline.
  1. Fix a bug in the old ARM64 probe trampoline for the case where the client changes lr. That code path never worked before, but has now been fixed.
  1. Removed trustedImm32FromPtr() helper functions in MacroAssemblerARM and MacroAssemblerARMv7.

We can now use move() with TrustedImmPtr, and it does the same thing but in a
more generic way.

  1. ARMv7's move() emitter may encode a T1 move instruction, which happens to have

the same semantics as movs (according to the Thumb spec). This means these
instructions may trash the APSR flags before we have a chance to preserve them.

This patch changes MacroAssemblerARMv7's probe() to preserve the APSR register
early on. This entails adding support for the mrs instruction in the
ARMv7Assembler.

  1. Change testmasm's testProbeModifiesStackValues() to now modify stack values

the easy way.

Also fixed testmasm tests which check flag registers to only compare the
portions that are modifiable by the client i.e. some masking is applied.

This patch has passed the testmasm tests on x86, x86_64, arm64, and armv7.

  • CMakeLists.txt:
  • JavaScriptCore.xcodeproj/project.pbxproj:
  • assembler/ARMv7Assembler.h:

(JSC::ARMv7Assembler::mrs):

  • assembler/AbstractMacroAssembler.h:
  • assembler/MacroAssembler.cpp:

(JSC::stdFunctionCallback):
(JSC::MacroAssembler::probe):

  • assembler/MacroAssembler.h:

(JSC::MacroAssembler::CPUState::gprName): Deleted.
(JSC::MacroAssembler::CPUState::sprName): Deleted.
(JSC::MacroAssembler::CPUState::fprName): Deleted.
(JSC::MacroAssembler::CPUState::gpr): Deleted.
(JSC::MacroAssembler::CPUState::spr): Deleted.
(JSC::MacroAssembler::CPUState::fpr): Deleted.
(JSC:: const): Deleted.
(JSC::MacroAssembler::CPUState::fpr const): Deleted.
(JSC::MacroAssembler::CPUState::pc): Deleted.
(JSC::MacroAssembler::CPUState::fp): Deleted.
(JSC::MacroAssembler::CPUState::sp): Deleted.
(JSC::MacroAssembler::CPUState::pc const): Deleted.
(JSC::MacroAssembler::CPUState::fp const): Deleted.
(JSC::MacroAssembler::CPUState::sp const): Deleted.
(JSC::Probe::State::gpr): Deleted.
(JSC::Probe::State::spr): Deleted.
(JSC::Probe::State::fpr): Deleted.
(JSC::Probe::State::gprName): Deleted.
(JSC::Probe::State::sprName): Deleted.
(JSC::Probe::State::fprName): Deleted.
(JSC::Probe::State::pc): Deleted.
(JSC::Probe::State::fp): Deleted.
(JSC::Probe::State::sp): Deleted.

  • assembler/MacroAssemblerARM.cpp:

(JSC::MacroAssembler::probe):

  • assembler/MacroAssemblerARM.h:

(JSC::MacroAssemblerARM::trustedImm32FromPtr): Deleted.

  • assembler/MacroAssemblerARM64.cpp:

(JSC::MacroAssembler::probe):
(JSC::arm64ProbeError): Deleted.

  • assembler/MacroAssemblerARMv7.cpp:

(JSC::MacroAssembler::probe):

  • assembler/MacroAssemblerARMv7.h:

(JSC::MacroAssemblerARMv7::armV7Condition):
(JSC::MacroAssemblerARMv7::trustedImm32FromPtr): Deleted.

  • assembler/MacroAssemblerPrinter.cpp:

(JSC::Printer::printCallback):

  • assembler/MacroAssemblerPrinter.h:
  • assembler/MacroAssemblerX86Common.cpp:

(JSC::ctiMasmProbeTrampoline):
(JSC::MacroAssembler::probe):

  • assembler/Printer.h:

(JSC::Printer::Context::Context):

  • assembler/ProbeContext.cpp: Added.

(JSC::Probe::executeProbe):
(JSC::Probe::handleProbeStackInitialization):
(JSC::Probe::probeStateForContext):

  • assembler/ProbeContext.h: Added.

(JSC::Probe::CPUState::gprName):
(JSC::Probe::CPUState::sprName):
(JSC::Probe::CPUState::fprName):
(JSC::Probe::CPUState::gpr):
(JSC::Probe::CPUState::spr):
(JSC::Probe::CPUState::fpr):
(JSC::Probe:: const):
(JSC::Probe::CPUState::fpr const):
(JSC::Probe::CPUState::pc):
(JSC::Probe::CPUState::fp):
(JSC::Probe::CPUState::sp):
(JSC::Probe::CPUState::pc const):
(JSC::Probe::CPUState::fp const):
(JSC::Probe::CPUState::sp const):
(JSC::Probe::Context::Context):
(JSC::Probe::Context::gpr):
(JSC::Probe::Context::spr):
(JSC::Probe::Context::fpr):
(JSC::Probe::Context::gprName):
(JSC::Probe::Context::sprName):
(JSC::Probe::Context::fprName):
(JSC::Probe::Context::pc):
(JSC::Probe::Context::fp):
(JSC::Probe::Context::sp):
(JSC::Probe::Context::stack):
(JSC::Probe::Context::hasWritesToFlush):
(JSC::Probe::Context::releaseStack):

  • assembler/ProbeStack.cpp: Added.

(JSC::Probe::Page::Page):
(JSC::Probe::Page::flushWrites):
(JSC::Probe::Stack::Stack):
(JSC::Probe::Stack::hasWritesToFlush):
(JSC::Probe::Stack::flushWrites):
(JSC::Probe::Stack::ensurePageFor):

  • assembler/ProbeStack.h: Added.

(JSC::Probe::Page::baseAddressFor):
(JSC::Probe::Page::chunkAddressFor):
(JSC::Probe::Page::baseAddress):
(JSC::Probe::Page::get):
(JSC::Probe::Page::set):
(JSC::Probe::Page::hasWritesToFlush const):
(JSC::Probe::Page::flushWritesIfNeeded):
(JSC::Probe::Page::dirtyBitFor):
(JSC::Probe::Page::physicalAddressFor):
(JSC::Probe::Stack::Stack):
(JSC::Probe::Stack::lowWatermark):
(JSC::Probe::Stack::get):
(JSC::Probe::Stack::set):
(JSC::Probe::Stack::newStackPointer const):
(JSC::Probe::Stack::setNewStackPointer):
(JSC::Probe::Stack::isValid):
(JSC::Probe::Stack::pageFor):

  • assembler/testmasm.cpp:

(JSC::testProbeReadsArgumentRegisters):
(JSC::testProbeWritesArgumentRegisters):
(JSC::testProbePreservesGPRS):
(JSC::testProbeModifiesStackPointer):
(JSC::testProbeModifiesStackPointerToInsideProbeStateOnStack):
(JSC::testProbeModifiesStackPointerToNBytesBelowSP):
(JSC::testProbeModifiesProgramCounter):
(JSC::testProbeModifiesStackValues):
(JSC::run):
(): Deleted.
(JSC::fillStack): Deleted.
(JSC::testProbeModifiesStackWithCallback): Deleted.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/assembler/MacroAssemblerARM64.cpp

    r220921 r220958  
    2929#include "MacroAssembler.h"
    3030
     31#include "ProbeContext.h"
    3132#include <wtf/InlineASM.h>
    3233
    … …  
    127128
    128129#define SAVED_PROBE_RETURN_PC_OFFSET        (PROBE_SIZE + (0 * PTR_SIZE))
    129 #define SAVED_PROBE_LR_OFFSET               (PROBE_SIZE + (1 * PTR_SIZE))
    130 #define SAVED_PROBE_ERROR_FUNCTION_OFFSET   (PROBE_SIZE + (2 * PTR_SIZE))
    131130#define PROBE_SIZE_PLUS_EXTRAS              (PROBE_SIZE + (3 * PTR_SIZE))
    132131
    … …  
    226225
    227226struct IncomingProbeRecord {
    228     uintptr_t probeHandlerFunction;
    229     uintptr_t probeArg;
     227    uintptr_t x24;
     228    uintptr_t x25;
    230229    uintptr_t x26;
    231230    uintptr_t x27;
    232     uintptr_t lr;
    233     uintptr_t sp;
    234     uintptr_t probeErrorFunction;
    235     uintptr_t unused; // Padding for alignment.
     231    uintptr_t x28;
     232    uintptr_t x30; // lr
    236233};
    237234
    238 #define IN_HANDLER_FUNCTION_OFFSET (0 * PTR_SIZE)
    239 #define IN_ARG_OFFSET              (1 * PTR_SIZE)
    240 #define IN_X26_OFFSET              (2 * PTR_SIZE)
    241 #define IN_X27_OFFSET              (3 * PTR_SIZE)
    242 #define IN_LR_OFFSET              (4 * PTR_SIZE)
    243 #define IN_SP_OFFSET              (5 * PTR_SIZE)
    244 #define IN_ERROR_FUNCTION_OFFSET   (6 * PTR_SIZE)
    245 
    246 static_assert(IN_HANDLER_FUNCTION_OFFSET == offsetof(IncomingProbeRecord, probeHandlerFunction), "IN_HANDLER_FUNCTION_OFFSET is incorrect");
    247 static_assert(IN_ARG_OFFSET == offsetof(IncomingProbeRecord, probeArg), "IN_ARG_OFFSET is incorrect");
     235#define IN_X24_OFFSET (0 * PTR_SIZE)
     236#define IN_X25_OFFSET (1 * PTR_SIZE)
     237#define IN_X26_OFFSET (2 * PTR_SIZE)
     238#define IN_X27_OFFSET (3 * PTR_SIZE)
     239#define IN_X28_OFFSET (4 * PTR_SIZE)
     240#define IN_X30_OFFSET (5 * PTR_SIZE)
     241#define IN_SIZE       (6 * PTR_SIZE)
     242
     243static_assert(IN_X24_OFFSET == offsetof(IncomingProbeRecord, x24), "IN_X24_OFFSET is incorrect");
     244static_assert(IN_X25_OFFSET == offsetof(IncomingProbeRecord, x25), "IN_X25_OFFSET is incorrect");
    248245static_assert(IN_X26_OFFSET == offsetof(IncomingProbeRecord, x26), "IN_X26_OFFSET is incorrect");
    249246static_assert(IN_X27_OFFSET == offsetof(IncomingProbeRecord, x27), "IN_X27_OFFSET is incorrect");
    250 static_assert(IN_LR_OFFSET == offsetof(IncomingProbeRecord, lr), "IN_LR_OFFSET is incorrect");
    251 static_assert(IN_SP_OFFSET == offsetof(IncomingProbeRecord, sp), "IN_SP_OFFSET is incorrect");
    252 static_assert(IN_ERROR_FUNCTION_OFFSET == offsetof(IncomingProbeRecord, probeErrorFunction), "IN_ERROR_FUNCTION_OFFSET is incorrect");
     247static_assert(IN_X28_OFFSET == offsetof(IncomingProbeRecord, x28), "IN_X22_OFFSET is incorrect");
     248static_assert(IN_X30_OFFSET == offsetof(IncomingProbeRecord, x30), "IN_X23_OFFSET is incorrect");
     249static_assert(IN_SIZE == sizeof(IncomingProbeRecord), "IN_SIZE is incorrect");
    253250static_assert(!(sizeof(IncomingProbeRecord) & 0xf), "IncomingProbeStack must be 16-byte aligned");
    254251
    … …  
    279276static_assert(!(sizeof(OutgoingProbeRecord) & 0xf), "OutgoingProbeStack must be 16-byte aligned");
    280277
    281 #define STATE_PC_NOT_CHANGED 0
    282 #define STATE_PC_CHANGED 1
    283 static_assert(STATE_PC_NOT_CHANGED != STATE_PC_CHANGED, "STATE_PC_NOT_CHANGED and STATE_PC_CHANGED should not be equal");
     278struct LRRestorationRecord {
     279    uintptr_t lr;
     280    uintptr_t unusedDummyToEnsureSizeIs16ByteAligned;
     281};
     282
     283#define LR_RESTORATION_LR_OFFSET (0 * PTR_SIZE)
     284#define LR_RESTORATION_SIZE      (2 * PTR_SIZE)
     285
     286static_assert(LR_RESTORATION_LR_OFFSET == offsetof(LRRestorationRecord, lr), "LR_RESTORATION_LR_OFFSET is incorrect");
     287static_assert(LR_RESTORATION_SIZE == sizeof(LRRestorationRecord), "LR_RESTORATION_SIZE is incorrect");
     288static_assert(!(sizeof(LRRestorationRecord) & 0xf), "LRRestorationRecord must be 16-byte aligned");
    284289
    285290// We use x29 and x30 instead of fp and lr because GCC's inline assembler does not recognize fp and lr.
    … …  
    294299    // MacroAssemblerARM64::probe() has already generated code to store some values in an
    295300    // IncomingProbeRecord. sp points to the IncomingProbeRecord.
     301    //
     302    // Incoming register values:
     303    //     x24: probe function
     304    //     x25: probe arg
     305    //     x26: scratch, was ctiMasmProbeTrampoline
     306    //     x27: scratch
     307    //     x28: Probe::executeProbe
     308    //     x30: return address
    296309
    297310    "mov       x26, sp" "\n"
    … …  
    301314    "bic       x27, x27, #0xf" "\n" // The ARM EABI specifies that the stack needs to be 16 byte aligned.
    302315    "mov       sp, x27" "\n" // Set the sp to protect the Probe::State from interrupts before we initialize it.
     316
     317    "stp       x24, x25, [sp, #" STRINGIZE_VALUE_OF(PROBE_PROBE_FUNCTION_OFFSET) "]" "\n" // Store the probe handler function and arg (preloaded into x24 and x25
    303318
    304319    "stp       x0, x1, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X0_OFFSET) "]" "\n"
    … …  
    310325    "stp       x8, x9, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X8_OFFSET) "]" "\n"
    311326
    312     "ldp       x2, x3, [x26, #" STRINGIZE_VALUE_OF(IN_HANDLER_FUNCTION_OFFSET) "]" "\n" // Preload probe handler function and probe arg.
    313     "ldp       x4, x5, [x26, #" STRINGIZE_VALUE_OF(IN_X26_OFFSET) "]" "\n" // Preload saved r26 and r27.
    314     "ldp       x6, x7, [x26, #" STRINGIZE_VALUE_OF(IN_LR_OFFSET) "]" "\n" // Preload saved lr and sp.
    315     "ldr       x8, [x26, #" STRINGIZE_VALUE_OF(IN_ERROR_FUNCTION_OFFSET) "]" "\n" // Preload probe error function.
     327    "ldp       x2, x3, [x26, #" STRINGIZE_VALUE_OF(IN_X24_OFFSET) "]" "\n" // Preload saved x24 and x25.
     328    "ldp       x4, x5, [x26, #" STRINGIZE_VALUE_OF(IN_X26_OFFSET) "]" "\n" // Preload saved x26 and x27.
     329    "ldp       x6, x7, [x26, #" STRINGIZE_VALUE_OF(IN_X28_OFFSET) "]" "\n" // Preload saved x28 and lr.
     330    "add       x26, x26, #" STRINGIZE_VALUE_OF(IN_SIZE) "\n" // Compute the sp before the probe.
    316331
    317332    "stp       x10, x11, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X10_OFFSET) "]" "\n"
    … …  
    322337    "stp       x20, x21, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X20_OFFSET) "]" "\n"
    323338    "stp       x22, x23, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X22_OFFSET) "]" "\n"
    324     "stp       x24, x25, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X24_OFFSET) "]" "\n"
     339    "stp       x2, x3, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X24_OFFSET) "]" "\n" // Store saved r24 and r25 (preloaded into x2 and x3 above).
    325340    "stp       x4, x5, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X26_OFFSET) "]" "\n" // Store saved r26 and r27 (preloaded into x4 and x5 above).
    326     "stp       x28, x29, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X28_OFFSET) "]" "\n"
    327     "stp       x6, x7, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_LR_OFFSET) "]" "\n" // Save values lr and sp (preloaded into x6 and x7 above).
    328 
    329     "str       x6, [sp, #" STRINGIZE_VALUE_OF(SAVED_PROBE_LR_OFFSET) "]" "\n" // Save a duplicate copy of lr (in x6).
     341    "stp       x6, x29, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_X28_OFFSET) "]" "\n"
     342    "stp       x7, x26, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_LR_OFFSET) "]" "\n" // Save values lr and sp (original sp value computed into x26 above).
     343
    330344    "str       x30, [sp, #" STRINGIZE_VALUE_OF(SAVED_PROBE_RETURN_PC_OFFSET) "]" "\n" // Save a duplicate copy of return pc (in lr).
    331345
    … …  
    334348
    335349    "stp       x0, x1, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_NZCV_OFFSET) "]" "\n" // Store nzcv and fpsr (preloaded into x0 and x1 above).
    336 
    337     "stp       x2, x3, [sp, #" STRINGIZE_VALUE_OF(PROBE_PROBE_FUNCTION_OFFSET) "]" "\n" // Store the probe handler function and arg (preloaded into x2 and x3 above).
    338     "str       x8, [sp, #" STRINGIZE_VALUE_OF(SAVED_PROBE_ERROR_FUNCTION_OFFSET) "]" "\n" // Store the probe handler function and arg (preloaded into x8 above).
    339350
    340351    "add       x9, sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_Q0_OFFSET) "\n"
    … …  
    358369    "mov       x27, sp" "\n" // Save the Probe::State* in a callee saved register.
    359370
    360     // Initialize Probe::State::initializeStackFunction to zero.
    361     "str       xzr, [x27, #" STRINGIZE_VALUE_OF(PROBE_INIT_STACK_FUNCTION_OFFSET) "]" "\n"
    362 
    363371    // Note: we haven't changed the value of fp. Hence, it is still pointing to the frame of
    364372    // the caller of the probe (which is what we want in order to play nice with debuggers e.g. lldb).
    365373    "mov       x0, sp" "\n" // Set the Probe::State* arg.
    366     "blr       x2" "\n" // Call the probe handler function (loaded into x2 above).
     374    "blr       x28" "\n" // Call the probe handler.
    367375
    368376    // Make sure the Probe::State is entirely below the result stack pointer so
    … …  
    447455    // Remaining registers to restore are: fpsr, nzcv, x27, x28, fp, lr, sp, and pc.
    448456
    449     "mov       x30, #" STRINGIZE_VALUE_OF(STATE_PC_NOT_CHANGED) "\n"
    450 
    451457    // The only way to set the pc on ARM64 (from user space) is via an indirect branch
    452458    // or a ret, which means we'll need a free register to do so. For our purposes, lr
    … …  
    455461    // returns. So, the ARM64 probe implementation will allow the probe handler to
    456462    // either modify lr or pc, but not both in the same probe invocation. The probe
    457     // mechanism ensures that we never try to modify both lr and pc, else it will
    458     // fail with a RELEASE_ASSERT_NOT_REACHED in arm64ProbeError().
     463    // mechanism ensures that we never try to modify both lr and pc with a RELEASE_ASSERT
     464    // in Probe::executeProbe().
    459465
    460466    // Determine if the probe handler changed the pc.
     467    "ldr       x30, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_SP_OFFSET) "]" "\n" // preload the target sp.
    461468    "ldr       x27, [sp, #" STRINGIZE_VALUE_OF(SAVED_PROBE_RETURN_PC_OFFSET) "]" "\n"
    462469    "ldr       x28, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_PC_OFFSET) "]" "\n"
    463470    "add       x27, x27, #" STRINGIZE_VALUE_OF(2 * PTR_SIZE) "\n"
    464471    "cmp       x27, x28" "\n"
    465     "beq     " LOCAL_LABEL_STRING(ctiMasmProbeTrampolinePrepareOutgoingRecords) "\n"
    466 
    467     // pc was changed. Determine if the probe handler also changed lr.
    468     "ldr       x27, [sp, #" STRINGIZE_VALUE_OF(SAVED_PROBE_LR_OFFSET) "]" "\n"
    469     "ldr       x28, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_LR_OFFSET) "]" "\n"
    470     "cmp       x27, x28" "\n"
    471     "bne     " LOCAL_LABEL_STRING(ctiMasmProbeTrampolineError) "\n"
    472 
    473     "mov       x30, #" STRINGIZE_VALUE_OF(STATE_PC_CHANGED) "\n"
    474 
    475     LOCAL_LABEL_STRING(ctiMasmProbeTrampolinePrepareOutgoingRecords) ":" "\n"
    476 
    477     "ldr       x29, [sp, #" STRINGIZE_VALUE_OF(SAVED_PROBE_RETURN_PC_OFFSET) "]" "\n" // Preload the probe return site pc.
    478 
    479     LOCAL_LABEL_STRING(ctiMasmProbeTrampolineFillOutgoingProbeRecords) ":" "\n"
    480 
    481     "cbnz       x30, " LOCAL_LABEL_STRING(ctiMasmProbeTrampolineEnd) "\n" // Skip lr restoration setup if state (in lr) == STATE_PC_CHANGED.
    482 
    483     // In order to restore lr, we need to do the restoration at the probe return site.
    484     // The probe return site expects sp to be pointing at an OutgoingProbeRecord such that
    485     // popping the OutgoingProbeRecord will yield the desired sp. The probe return site
    486     // also expects the lr value to be restored is stashed in the OutgoingProbeRecord.
    487     // We can make this happen by pushing 2 OutgoingProbeRecords instead of 1:
    488     // 1 for the probe return site, and 1 at ctiMasmProbeTrampolineEnd for returning from
    489     // this probe.
    490 
    491     // Fill in the OutgoingProbeStack for the probe return site.
    492     "ldr       x30, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_SP_OFFSET) "]" "\n"
    493     "sub       x30, x30, #" STRINGIZE_VALUE_OF(OUT_SIZE) "\n"
    494 
     472    "bne     " LOCAL_LABEL_STRING(ctiMasmProbeTrampolineEnd) "\n"
     473
     474     // We didn't change the PC. So, let's prepare for setting a potentially new lr value.
     475
     476     // 1. Make room for the LRRestorationRecord. The probe site will pop this off later.
     477    "sub       x30, x30, #" STRINGIZE_VALUE_OF(LR_RESTORATION_SIZE) "\n"
     478     // 2. Store the lp value to restore at the probe return site.
    495479    "ldr       x27, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_LR_OFFSET) "]" "\n"
    496     "str       x27, [x30, #" STRINGIZE_VALUE_OF(OUT_LR_OFFSET) "]" "\n"
    497 
    498     // Set up the sp and pc values so that ctiMasmProbeTrampolineEnd will return to the probe return site.
    499     "str       x30, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_SP_OFFSET) "]" "\n"
    500     "str       x29, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_PC_OFFSET) "]" "\n" // Store the probe return site pc (preloaded into fp above).
     480    "str       x27, [x30, #" STRINGIZE_VALUE_OF(LR_RESTORATION_LR_OFFSET) "]" "\n"
     481     // 3. Force the return ramp to return to the probe return site.
     482    "ldr       x27, [sp, #" STRINGIZE_VALUE_OF(SAVED_PROBE_RETURN_PC_OFFSET) "]" "\n"
     483    "str       x27, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_PC_OFFSET) "]" "\n"
    501484
    502485    LOCAL_LABEL_STRING(ctiMasmProbeTrampolineEnd) ":" "\n"
    503486
    504     // Fill in the OutgoingProbeStack.
    505     "ldr       x30, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_SP_OFFSET) "]" "\n"
     487    // Fill in the OutgoingProbeRecord.
    506488    "sub       x30, x30, #" STRINGIZE_VALUE_OF(OUT_SIZE) "\n"
    507489
    … …  
    511493    "stp       x27, x28, [x30, #" STRINGIZE_VALUE_OF(OUT_X27_OFFSET) "]" "\n"
    512494    "ldr       x27, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_FP_OFFSET) "]" "\n"
    513     "ldr       x28, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_PC_OFFSET) "]" "\n"
     495    "ldr       x28, [sp, #" STRINGIZE_VALUE_OF(PROBE_CPU_PC_OFFSET) "]" "\n" // Set up the outgoing record so that we'll jump to the new PC.
    514496    "stp       x27, x28, [x30, #" STRINGIZE_VALUE_OF(OUT_FP_OFFSET) "]" "\n"
    515497    "mov       sp, x30" "\n"
    516498
    517     // Restore the remaining registers and pop the OutgoingProbeStack.
     499    // Restore the remaining registers and pop the OutgoingProbeRecord.
    518500    "ldp       x27, x28, [sp], #" STRINGIZE_VALUE_OF(2 * PTR_SIZE) "\n"
    519501    "msr       nzcv, x27" "\n"
    … …  
    522504    "ldp       x29, x30, [sp], #" STRINGIZE_VALUE_OF(2 * PTR_SIZE) "\n"
    523505    "ret" "\n"
    524 
    525     LOCAL_LABEL_STRING(ctiMasmProbeTrampolineError) ":" "\n"
    526     // The probe handler changed both lr and pc. This is not supported for ARM64.
    527     "ldr       x1, [sp, #" STRINGIZE_VALUE_OF(SAVED_PROBE_ERROR_FUNCTION_OFFSET) "]" "\n"
    528     "mov       x0, sp" "\n" // Set the Probe::State* arg.
    529     "blr       x1" "\n"
    530     "brk       #0x1000" // Should never return here.
    531506);
    532507#endif // COMPILER(GCC_OR_CLANG)
    533 
    534 static NO_RETURN_DUE_TO_CRASH void arm64ProbeError(Probe::State*)
    535 {
    536     dataLog("MacroAssembler probe ERROR: ARM64 does not support the probe changing both LR and PC.\n");
    537     RELEASE_ASSERT_NOT_REACHED();
    538 }
    539508
    540509void MacroAssembler::probe(Probe::Function function, void* arg)
    … …  
    542511    sub64(TrustedImm32(sizeof(IncomingProbeRecord)), sp);
    543512
     513    storePair64(x24, x25, sp, TrustedImm32(offsetof(IncomingProbeRecord, x24)));
    544514    storePair64(x26, x27, sp, TrustedImm32(offsetof(IncomingProbeRecord, x26)));
    545     add64(TrustedImm32(sizeof(IncomingProbeRecord)), sp, x26);
    546     storePair64(lr, x26, sp, TrustedImm32(offsetof(IncomingProbeRecord, lr))); // Save lr and original sp value.
    547     move(TrustedImmPtr(reinterpret_cast<void*>(function)), x26);
    548     move(TrustedImmPtr(arg), x27);
    549     storePair64(x26, x27, sp, TrustedImm32(offsetof(IncomingProbeRecord, probeHandlerFunction)));
    550     move(TrustedImmPtr(reinterpret_cast<void*>(arm64ProbeError)), x27);
    551     store64(x27, Address(sp, offsetof(IncomingProbeRecord, probeErrorFunction)));
    552 
     515    storePair64(x28, x30, sp, TrustedImm32(offsetof(IncomingProbeRecord, x28))); // Note: x30 is lr.
    553516    move(TrustedImmPtr(reinterpret_cast<void*>(ctiMasmProbeTrampoline)), x26);
     517    move(TrustedImmPtr(reinterpret_cast<void*>(Probe::executeProbe)), x28);
     518    move(TrustedImmPtr(reinterpret_cast<void*>(function)), x24);
     519    move(TrustedImmPtr(arg), x25);
    554520    m_assembler.blr(x26);
    555521
    556522    // ctiMasmProbeTrampoline should have restored every register except for lr and the sp.
    557     load64(Address(sp, offsetof(OutgoingProbeRecord, lr)), lr);
    558     add64(TrustedImm32(sizeof(OutgoingProbeRecord)), sp);
     523    load64(Address(sp, offsetof(LRRestorationRecord, lr)), lr);
     524    add64(TrustedImm32(sizeof(LRRestorationRecord)), sp);
    559525}
    560526
Note: See TracChangeset for help on using the changeset viewer.