Changeset 220118 in webkit for trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp

Timestamp:

Aug 1, 2017, 6:50:16 PM (8 years ago)

Author:

[email protected]

Message:

Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
​https://bugs.webkit.org/show_bug.cgi?id=174727

Reviewed by Mark Lam.
Source/bmalloc:

This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:

• You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.

• Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind.

• Each isoheap gets its own Heap.

• Each Heap gets a scavenger thread.

• Some things, like Zone/VMHeap/Scavenger, are per-process.

Most of the per-HeapKind functionality is handled by PerHeapKind<>.

This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.

• CMakeLists.txt:
• bmalloc.xcodeproj/project.pbxproj:
• bmalloc/AllocationKind.h: Added.
• bmalloc/Allocator.cpp:

(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):

• bmalloc/Allocator.h:
• bmalloc/BExport.h: Added.
• bmalloc/Cache.cpp:

(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.

• bmalloc/Cache.h:

(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):

• bmalloc/Deallocator.cpp:

(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):

• bmalloc/Deallocator.h:
• bmalloc/Gigacage.cpp: Added.

(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):

• bmalloc/Gigacage.h: Added.

(Gigacage::caged):
(Gigacage::isCaged):

• bmalloc/Heap.cpp:

(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):

• bmalloc/Heap.h:

(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.

• bmalloc/HeapKind.h: Added.
• bmalloc/ObjectType.cpp:

(bmalloc::objectType):

• bmalloc/ObjectType.h:
• bmalloc/PerHeapKind.h: Added.

(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):

• bmalloc/PerThread.h:

(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.

• bmalloc/Scavenger.cpp: Added.

(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):

• bmalloc/Scavenger.h: Added.

(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):

• bmalloc/VMHeap.cpp:

(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):

• bmalloc/VMHeap.h:
• bmalloc/Zone.cpp:

(bmalloc::Zone::Zone):

• bmalloc/Zone.h:
• bmalloc/bmalloc.h:

(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):

• bmalloc/mbmalloc.cpp:

Source/JavaScriptCore:

This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.

This is neutral on JetStream.

• CMakeLists.txt:
• JavaScriptCore.xcodeproj/project.pbxproj:
• b3/B3InsertionSet.cpp:

(JSC::B3::InsertionSet::execute):

• dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

• dfg/DFGArgumentsEliminationPhase.cpp:
• dfg/DFGClobberize.cpp:

(JSC::DFG::readsOverlap):

• dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

• dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

• dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.

(JSC::DFG::performFixedButterflyAccessUncaging):

• dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

• dfg/DFGHeapLocation.cpp:

(WTF::printInternal):

• dfg/DFGHeapLocation.h:
• dfg/DFGNodeType.h:
• dfg/DFGPlan.cpp:

(JSC::DFG::Plan::compileInThreadImpl):

• dfg/DFGPredictionPropagationPhase.cpp:
• dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

• dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::compileGetButterfly):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGTypeCheckHoistingPhase.cpp:

(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):

• ftl/FTLCapabilities.cpp:

(JSC::FTL::canCompile):

• ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):

• heap/GigacageSubspace.cpp: Added.

(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):

• heap/GigacageSubspace.h: Added.
• heap/Heap.cpp:

(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.

• heap/Heap.h:
• heap/LargeAllocation.cpp:

(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):

• heap/MarkedAllocator.cpp:

(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):

• heap/MarkedBlock.cpp:

(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.

• heap/MarkedBlock.h:

(JSC::MarkedBlock::Handle::subspace const):

• heap/MarkedSpace.cpp:

(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.

• heap/MarkedSpace.h:

(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.

• heap/Subspace.cpp:

(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):

• heap/Subspace.h:

(JSC::Subspace::didCreateFirstAllocator):

• heap/SubspaceInlines.h:

(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):

• jit/JITPropertyAccess.cpp:

(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):

• jsc.cpp:

(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):

• llint/LowLevelInterpreter64.asm:
• runtime/ArrayBuffer.cpp:

(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):

• runtime/IndexingHeader.h:
• runtime/InitializeThreading.cpp:

(JSC::initializeThreading):

• runtime/JSArrayBuffer.cpp:
• runtime/JSArrayBufferView.cpp:

(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):

• runtime/JSLock.cpp:

(JSC::JSLock::didAcquireLock):

• runtime/JSObject.h:
• runtime/Options.cpp:

(JSC::recomputeDependentOptions):

• runtime/Options.h:
• runtime/ScopedArgumentsTable.h:
• runtime/VM.cpp:

(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):

• runtime/VM.h:

(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):

• wasm/WasmB3IRGenerator.cpp:

(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):

• wasm/WasmCodeBlock.cpp:

(JSC::Wasm::CodeBlock::isSafeToRun):

• wasm/WasmMemory.cpp:

(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.

• wasm/WasmMemory.h:
• wasm/js/JSWebAssemblyInstance.cpp:

(JSC::JSWebAssemblyInstance::create):

• wasm/js/JSWebAssemblyMemory.cpp:

(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):

• wasm/js/JSWebAssemblyMemory.h:

(JSC::JSWebAssemblyMemory::subspaceFor):

Source/WebCore:

No new tests because no change in behavior.

Needed to teach Metal how to allocate in the Gigacage.

• platform/graphics/cocoa/GPUBufferMetal.mm:

(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):

Source/WebKit:

The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.

• WebProcess/WebProcess.cpp:

(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):

Source/WTF:

For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.

Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.

• WTF.xcodeproj/project.pbxproj:
• wtf/CMakeLists.txt:
• wtf/FastMalloc.cpp:
• wtf/Gigacage.cpp: Added.

(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):

• wtf/Gigacage.h: Added.

(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):

File:

• trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp (modified) (14 diffs)

trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp

@@ -88 +88 @@
 #include <unordered_set>
 #include <wtf/Box.h>
+#include <wtf/Gigacage.h>
 
 namespace JSC { namespace FTL {
@@ -665 +666 @@
             break;
         case GetButterfly:
+        case GetButterflyWithoutCaging:
             compileGetButterfly();
             break;
@@ -3232 +3234 @@
     void compileGetButterfly()
     {
-        setStorage(m_out.loadPtr(lowCell(m_node->child1()), m_heaps.JSObject_butterfly));
+        LValue butterfly = m_out.loadPtr(lowCell(m_node->child1()), m_heaps.JSObject_butterfly);
+        if (m_node->op() != GetButterflyWithoutCaging)
+            butterfly = caged(butterfly);
+        setStorage(butterfly);
     }
 
@@ -3268 +3273 @@
 
         DFG_ASSERT(m_graph, m_node, isTypedView(m_node->arrayMode().typedArrayType()));
-        setStorage(m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector));
+        setStorage(caged(m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector)));
     }
     
@@ -3510 +3515 @@
                     m_out.load32NonNegative(base, m_heaps.DirectArguments_length)));
 
+            // FIXME: I guess we need to cage DirectArguments?
+            // https://bugs.webkit.org/show_bug.cgi?id=174920
             TypedPointer address = m_out.baseIndex(
                 m_heaps.DirectArguments_storage, base, m_out.zeroExtPtr(index));
@@ -3541 +3548 @@
             LValue arguments = m_out.loadPtr(table, m_heaps.ScopedArgumentsTable_arguments);
             
+            // FIXME: I guess we need to cage ScopedArguments?
+            // https://bugs.webkit.org/show_bug.cgi?id=174921
             TypedPointer address = m_out.baseIndex(
                 m_heaps.scopedArgumentsTableArguments, arguments, m_out.zeroExtPtr(index));
@@ -3549 +3558 @@
                 m_out.equal(scopeOffset, m_out.constInt32(ScopeOffset::invalidOffset)));
             
+            // FIXME: I guess we need to cage JSEnvironmentRecord?
+            // https://bugs.webkit.org/show_bug.cgi?id=174922
             address = m_out.baseIndex(
                 m_heaps.JSEnvironmentRecord_variables, scope, m_out.zeroExtPtr(scopeOffset));
@@ -3556 +3567 @@
             m_out.appendTo(overflowCase, continuation);
             
+            // FIXME: I guess we need to cage overflow storage?
+            // https://bugs.webkit.org/show_bug.cgi?id=174923
             address = m_out.baseIndex(
                 m_heaps.ScopedArguments_overflowStorage, base,
@@ -5379 +5392 @@
         m_out.appendTo(is8Bit, is16Bit);
             
+        // FIXME: Need to cage strings!
+        // https://bugs.webkit.org/show_bug.cgi?id=174924
         ValueFromBlock char8Bit = m_out.anchor(
             m_out.load8ZeroExt32(m_out.baseIndex(
@@ -5480 +5495 @@
         LBasicBlock lastNext = m_out.appendTo(is8Bit, is16Bit);
             
+        // FIXME: need to cage strings!
+        // https://bugs.webkit.org/show_bug.cgi?id=174924
         ValueFromBlock char8Bit = m_out.anchor(
             m_out.load8ZeroExt32(m_out.baseIndex(
@@ -8076 +8093 @@
         LValue unmaskedIndex = m_out.phi(Int32, indexStart);
         LValue index = m_out.bitAnd(mask, unmaskedIndex);
+        // FIXME: I think these buffers are caged?
+        // https://bugs.webkit.org/show_bug.cgi?id=174925
         LValue hashMapBucket = m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), buffer, m_out.zeroExt(index, Int64), ScaleEight));
         ValueFromBlock bucketResult = m_out.anchor(hashMapBucket);
@@ -8851 +8870 @@
         int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
         ValueFromBlock outOfLineResult = m_out.anchor(
-            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), storage, realIndex, ScaleEight, offsetOfFirstProperty)));
+            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged(storage), realIndex, ScaleEight, offsetOfFirstProperty)));
         m_out.jump(continuation);
 
@@ -10269 +10288 @@
         m_out.appendTo(loopBody, slowPath);
 
+        // FIXME: Strings needs to be caged.
+        // https://bugs.webkit.org/show_bug.cgi?id=174924
         LValue byte = m_out.load8ZeroExt32(m_out.baseIndex(m_heaps.characters8, buffer, m_out.zeroExtPtr(index)));
         LValue isInvalidAsciiRange = m_out.bitAnd(byte, m_out.constInt32(~0x7F));
@@ -11592 +11613 @@
             m_out.appendTo(performStore, lastNext);
         }
+    }
+    
+    LValue caged(LValue ptr)
+    {
+        if (vm().gigacageEnabled().isStillValid()) {
+            m_graph.watchpoints().addLazily(vm().gigacageEnabled());
+            
+            LValue basePtr = m_out.constIntPtr(g_gigacageBasePtr);
+            LValue mask = m_out.constIntPtr(GIGACAGE_MASK);
+            
+            // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent
+            // codegen for 2-operand andq on x86-64.
+            LValue masked = m_out.bitAnd(ptr, mask);
+            
+            // But B3 will currently mess up the code generation of this add. Basically, any offset from what we
+            // compute here will get reassociated and folded with g_gigacageBasePtr. There's a world in which
+            // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But
+            // it's much easier to just block B3's badness here. That's what we do for now.
+            PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
+            patchpoint->appendSomeRegister(basePtr);
+            patchpoint->appendSomeRegister(masked);
+            patchpoint->setGenerator(
+                [] (CCallHelpers& jit, const StackmapGenerationParams& params) {
+                    jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr());
+                });
+            patchpoint->effects = Effects::none();
+            return patchpoint;
+        }
+        
+        return ptr;
     }