Ignore:
Timestamp:
Aug 7, 2017, 2:31:49 PM (8 years ago)
Author:
[email protected]
Message:

Primitive auxiliaries and JSValue auxiliaries should have separate gigacages
​https://bugs.webkit.org/show_bug.cgi?id=174919

Reviewed by Keith Miller.
Source/bmalloc:


This introduces two kinds of Gigacage, Primitive and JSValue. This translates to two kinds of
HeapKind, PrimitiveGigacage and JSValueGigacage.

The new support functionality required turning Inline.h into BInline.h, and INLINE into BINLINE, and
NO_INLINE into BNO_INLINE.

  • bmalloc.xcodeproj/project.pbxproj:
  • bmalloc/Allocator.cpp:

(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::refillAllocator):
(bmalloc::Allocator::allocateLarge):
(bmalloc::Allocator::allocateLogSizeClass):

  • bmalloc/AsyncTask.h:
  • bmalloc/BInline.h: Copied from Source/bmalloc/bmalloc/Inline.h.
  • bmalloc/Cache.cpp:

(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):

  • bmalloc/Deallocator.cpp:
  • bmalloc/Gigacage.cpp:

(Gigacage::PrimitiveDisableCallbacks::PrimitiveDisableCallbacks):
(Gigacage::ensureGigacage):
(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::Callbacks::Callbacks): Deleted.
(Gigacage::disableGigacage): Deleted.
(Gigacage::addDisableCallback): Deleted.
(Gigacage::removeDisableCallback): Deleted.

  • bmalloc/Gigacage.h:

(Gigacage::name):
(Gigacage::basePtr):
(Gigacage::forEachKind):
(Gigacage::caged):
(Gigacage::isCaged):

  • bmalloc/Heap.cpp:

(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::gigacageBasePtr):

  • bmalloc/Heap.h:
  • bmalloc/HeapKind.h:

(bmalloc::isGigacage):
(bmalloc::gigacageKind):
(bmalloc::heapKind):

  • bmalloc/Inline.h: Removed.
  • bmalloc/Map.h:
  • bmalloc/PerProcess.h:

(bmalloc::PerProcess<T>::getFastCase):
(bmalloc::PerProcess<T>::get):
(bmalloc::PerProcess<T>::getSlowCase):

  • bmalloc/PerThread.h:

(bmalloc::PerThread<T>::getFastCase):

  • bmalloc/Vector.h:

(bmalloc::Vector<T>::push):
(bmalloc::Vector<T>::shrinkCapacity):
(bmalloc::Vector<T>::growCapacity):

Source/JavaScriptCore:


This adapts JSC to there being two gigacages.

To make matters simpler, this turns AlignedMemoryAllocators into per-VM instances rather than
singletons. I don't think we were gaining anything by making them be singletons.

This makes it easy to teach GigacageAlignedMemoryAllocator that there are multiple kinds of
gigacages. We'll have one of those allocators per cage.

From there, this change teaches everyone who previously knew about cages that there are two cages.
This means having to specify either Gigacage::Primitive or Gigacage::JSValue. In most places, this is
easy: typed arrays are Primitive and butterflies are JSValue. But there are a few places where it's
not so obvious, so this change introduces some helpers to make it easy to define what cage you want
to use in one place and refer to it abstractly. We do this in DirectArguments and GenericArguments.h

A lot of the magic of this change is due to CagedBarrierPtr, which combines AuxiliaryBarrier and
CagedPtr. This removes one layer of "get()" calls from a bunch of places.

(JSC::AccessCase::generateImpl):

  • dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::emitAllocateRawObject):
(JSC::DFG::SpeculativeJIT::compileAllocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileReallocatePropertyStorage):
(JSC::DFG::SpeculativeJIT::compileNewTypedArray):
(JSC::DFG::SpeculativeJIT::emitAllocateButterfly):

  • ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileNewTypedArray):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileMaterializeNewObject):
(JSC::FTL::DFG::LowerDFGToB3::allocatePropertyStorageWithSizeImpl):
(JSC::FTL::DFG::LowerDFGToB3::allocateJSArray):
(JSC::FTL::DFG::LowerDFGToB3::caged):

  • heap/FastMallocAlignedMemoryAllocator.cpp:

(JSC::FastMallocAlignedMemoryAllocator::instance): Deleted.

  • heap/FastMallocAlignedMemoryAllocator.h:
  • heap/GigacageAlignedMemoryAllocator.cpp:

(JSC::GigacageAlignedMemoryAllocator::GigacageAlignedMemoryAllocator):
(JSC::GigacageAlignedMemoryAllocator::tryAllocateAlignedMemory):
(JSC::GigacageAlignedMemoryAllocator::freeAlignedMemory):
(JSC::GigacageAlignedMemoryAllocator::dump const):
(JSC::GigacageAlignedMemoryAllocator::instance): Deleted.

  • heap/GigacageAlignedMemoryAllocator.h:
  • jsc.cpp:

(primitiveGigacageDisabled):
(jscmain):
(gigacageDisabled): Deleted.

  • llint/LowLevelInterpreter64.asm:
  • runtime/ArrayBuffer.cpp:

(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):

  • runtime/AuxiliaryBarrier.h:
  • runtime/ButterflyInlines.h:

(JSC::Butterfly::createUninitialized):
(JSC::Butterfly::tryCreate):
(JSC::Butterfly::growArrayRight):

  • runtime/CagedBarrierPtr.h: Added.

(JSC::CagedBarrierPtr::CagedBarrierPtr):
(JSC::CagedBarrierPtr::clear):
(JSC::CagedBarrierPtr::set):
(JSC::CagedBarrierPtr::get const):
(JSC::CagedBarrierPtr::getMayBeNull const):
(JSC::CagedBarrierPtr::operator== const):
(JSC::CagedBarrierPtr::operator!= const):
(JSC::CagedBarrierPtr::operator bool const):
(JSC::CagedBarrierPtr::setWithoutBarrier):
(JSC::CagedBarrierPtr::operator* const):
(JSC::CagedBarrierPtr::operator-> const):
(JSC::CagedBarrierPtr::operator[] const):

  • runtime/DirectArguments.cpp:

(JSC::DirectArguments::overrideThings):
(JSC::DirectArguments::unmapArgument):

  • runtime/DirectArguments.h:

(JSC::DirectArguments::isMappedArgument const):

  • runtime/GenericArguments.h:
  • runtime/GenericArgumentsInlines.h:

(JSC::GenericArguments<Type>::initModifiedArgumentsDescriptor):
(JSC::GenericArguments<Type>::setModifiedArgumentDescriptor):
(JSC::GenericArguments<Type>::isModifiedArgumentDescriptor):

  • runtime/HashMapImpl.cpp:

(JSC::HashMapImpl<HashMapBucket>::visitChildren):

  • runtime/HashMapImpl.h:

(JSC::HashMapBuffer::create):
(JSC::HashMapImpl::buffer const):
(JSC::HashMapImpl::rehash):

  • runtime/JSArray.cpp:

(JSC::JSArray::tryCreateUninitializedRestricted):
(JSC::JSArray::unshiftCountSlowCase):
(JSC::JSArray::setLength):
(JSC::JSArray::pop):
(JSC::JSArray::push):
(JSC::JSArray::fastSlice):
(JSC::JSArray::shiftCountWithArrayStorage):
(JSC::JSArray::shiftCountWithAnyIndexingType):
(JSC::JSArray::unshiftCountWithAnyIndexingType):
(JSC::JSArray::fillArgList):
(JSC::JSArray::copyToArguments):

  • runtime/JSArray.h:

(JSC::JSArray::tryCreate):

  • runtime/JSArrayBufferView.cpp:

(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):

  • runtime/JSLock.cpp:

(JSC::JSLock::didAcquireLock):

  • runtime/JSObject.cpp:

(JSC::JSObject::heapSnapshot):
(JSC::JSObject::getOwnPropertySlotByIndex):
(JSC::JSObject::putByIndex):
(JSC::JSObject::enterDictionaryIndexingMode):
(JSC::JSObject::createInitialIndexedStorage):
(JSC::JSObject::createArrayStorage):
(JSC::JSObject::convertUndecidedToInt32):
(JSC::JSObject::convertUndecidedToDouble):
(JSC::JSObject::convertUndecidedToContiguous):
(JSC::JSObject::constructConvertedArrayStorageWithoutCopyingElements):
(JSC::JSObject::convertUndecidedToArrayStorage):
(JSC::JSObject::convertInt32ToDouble):
(JSC::JSObject::convertInt32ToContiguous):
(JSC::JSObject::convertInt32ToArrayStorage):
(JSC::JSObject::convertDoubleToContiguous):
(JSC::JSObject::convertDoubleToArrayStorage):
(JSC::JSObject::convertContiguousToArrayStorage):
(JSC::JSObject::setIndexQuicklyToUndecided):
(JSC::JSObject::ensureArrayStorageExistsAndEnterDictionaryIndexingMode):
(JSC::JSObject::deletePropertyByIndex):
(JSC::JSObject::getOwnPropertyNames):
(JSC::JSObject::putIndexedDescriptor):
(JSC::JSObject::defineOwnIndexedProperty):
(JSC::JSObject::putByIndexBeyondVectorLengthWithoutAttributes):
(JSC::JSObject::putDirectIndexSlowOrBeyondVectorLength):
(JSC::JSObject::getNewVectorLength):
(JSC::JSObject::ensureLengthSlow):
(JSC::JSObject::reallocateAndShrinkButterfly):
(JSC::JSObject::allocateMoreOutOfLineStorage):
(JSC::JSObject::getEnumerableLength):

  • runtime/JSObject.h:

(JSC::JSObject::getArrayLength const):
(JSC::JSObject::getVectorLength):
(JSC::JSObject::putDirectIndex):
(JSC::JSObject::canGetIndexQuickly):
(JSC::JSObject::getIndexQuickly):
(JSC::JSObject::tryGetIndexQuickly const):
(JSC::JSObject::canSetIndexQuickly):
(JSC::JSObject::setIndexQuickly):
(JSC::JSObject::initializeIndex):
(JSC::JSObject::initializeIndexWithoutBarrier):
(JSC::JSObject::hasSparseMap):
(JSC::JSObject::inSparseIndexingMode):
(JSC::JSObject::butterfly const):
(JSC::JSObject::butterfly):
(JSC::JSObject::outOfLineStorage const):
(JSC::JSObject::outOfLineStorage):
(JSC::JSObject::ensureInt32):
(JSC::JSObject::ensureDouble):
(JSC::JSObject::ensureContiguous):
(JSC::JSObject::ensureArrayStorage):
(JSC::JSObject::arrayStorage):
(JSC::JSObject::arrayStorageOrNull):
(JSC::JSObject::ensureLength):

  • runtime/RegExpMatchesArray.h:

(JSC::tryCreateUninitializedRegExpMatchesArray):

  • runtime/VM.cpp:

(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::primitiveGigacageDisabledCallback):
(JSC::VM::primitiveGigacageDisabled):
(JSC::VM::gigacageDisabledCallback): Deleted.
(JSC::VM::gigacageDisabled): Deleted.

  • runtime/VM.h:

(JSC::VM::gigacageAuxiliarySpace):
(JSC::VM::firePrimitiveGigacageEnabledIfNecessary):
(JSC::VM::primitiveGigacageEnabled):
(JSC::VM::fireGigacageEnabledIfNecessary): Deleted.
(JSC::VM::gigacageEnabled): Deleted.

  • wasm/WasmMemory.cpp:

(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::grow):

Source/WebCore:

No new tests because no change in behavior.

Adapting to API changes - we now specify the AlignedMemoryAllocator differently and we need to be
specific about which Gigacage we're using.

  • bindings/js/WebCoreJSClientData.cpp:

(WebCore::JSVMClientData::JSVMClientData):

  • platform/graphics/cocoa/GPUBufferMetal.mm:

(WebCore::GPUBuffer::GPUBuffer):

Source/WebKit:


The disable callback is all about the primitive gigacage.

  • WebProcess/WebProcess.cpp:

(WebKit::primitiveGigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):
(WebKit::gigacageDisabled): Deleted.

Source/WTF:


This mirrors the changes from bmalloc/Gigacage.h.

Also it teaches CagedPtr how to reason about multiple gigacages.

  • wtf/CagedPtr.h:

(WTF::CagedPtr::get const):
(WTF::CagedPtr::operator[] const):

  • wtf/Gigacage.cpp:

(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):

  • wtf/Gigacage.h:

(Gigacage::disablePrimitiveGigacage):
(Gigacage::addPrimitiveDisableCallback):
(Gigacage::removePrimitiveDisableCallback):
(Gigacage::name):
(Gigacage::basePtr):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):
(Gigacage::disableGigacage): Deleted.
(Gigacage::addDisableCallback): Deleted.
(Gigacage::removeDisableCallback): Deleted.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToB3.cpp

    r220219 r220352  
    32363236        LValue butterfly = m_out.loadPtr(lowCell(m_node->child1()), m_heaps.JSObject_butterfly);
    32373237        if (m_node->op() != GetButterflyWithoutCaging)
    3238             butterfly = caged(butterfly);
     3238            butterfly = caged(Gigacage::JSValue, butterfly);
    32393239        setStorage(butterfly);
    32403240    }
    … …  
    32733273
    32743274        DFG_ASSERT(m_graph, m_node, isTypedView(m_node->arrayMode().typedArrayType()));
    3275         setStorage(caged(m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector)));
     3275        setStorage(caged(Gigacage::Primitive, m_out.loadPtr(cell, m_heaps.JSArrayBufferView_vector)));
    32763276    }
    32773277   
    … …  
    50415041            }
    50425042       
    5043             LValue allocator = allocatorForSize(vm().auxiliarySpace, byteSize, slowCase);
     5043            LValue allocator = allocatorForSize(vm().primitiveGigacageAuxiliarySpace, byteSize, slowCase);
    50445044            LValue storage = allocateHeapCell(allocator, slowCase);
    50455045           
    … …  
    88708870        int32_t offsetOfFirstProperty = static_cast<int32_t>(offsetInButterfly(firstOutOfLineOffset)) * sizeof(EncodedJSValue);
    88718871        ValueFromBlock outOfLineResult = m_out.anchor(
    8872             m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged(storage), realIndex, ScaleEight, offsetOfFirstProperty)));
     8872            m_out.load64(m_out.baseIndex(m_heaps.properties.atAnyNumber(), caged(Gigacage::JSValue, storage), realIndex, ScaleEight, offsetOfFirstProperty)));
    88738873        m_out.jump(continuation);
    88748874
    … …  
    90509050               
    90519051                LValue startOfStorage = allocateHeapCell(
    9052                     allocatorForSize(vm().auxiliarySpace, butterflySize, slowPath),
     9052                    allocatorForSize(vm().jsValueGigacageAuxiliarySpace, butterflySize, slowPath),
    90539053                    slowPath);
    90549054
    … …  
    1000610006
    1000710007        size_t sizeInBytes = sizeInValues * sizeof(JSValue);
    10008         MarkedAllocator* allocator = vm().auxiliarySpace.allocatorFor(sizeInBytes);
     10008        MarkedAllocator* allocator = vm().jsValueGigacageAuxiliarySpace.allocatorFor(sizeInBytes);
    1000910009        LValue startOfStorage = allocateHeapCell(m_out.constIntPtr(allocator), slowPath);
    1001010010        ValueFromBlock fastButterfly = m_out.anchor(
    … …  
    1123411234            payloadSize, m_out.constIntPtr(sizeof(IndexingHeader)));
    1123511235           
    11236         LValue allocator = allocatorForSize(vm().auxiliarySpace, butterflySize, failCase);
     11236        LValue allocator = allocatorForSize(vm().jsValueGigacageAuxiliarySpace, butterflySize, failCase);
    1123711237        LValue startOfStorage = allocateHeapCell(allocator, failCase);
    1123811238           
    … …  
    1161511615    }
    1161611616   
    11617     LValue caged(LValue ptr)
    11618     {
    11619         if (vm().gigacageEnabled().isStillValid()) {
    11620             m_graph.watchpoints().addLazily(vm().gigacageEnabled());
    11621            
    11622             LValue basePtr = m_out.constIntPtr(g_gigacageBasePtr);
    11623             LValue mask = m_out.constIntPtr(GIGACAGE_MASK);
    11624            
    11625             // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent
    11626             // codegen for 2-operand andq on x86-64.
    11627             LValue masked = m_out.bitAnd(ptr, mask);
    11628            
    11629             // But B3 will currently mess up the code generation of this add. Basically, any offset from what we
    11630             // compute here will get reassociated and folded with g_gigacageBasePtr. There's a world in which
    11631             // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But
    11632             // it's much easier to just block B3's badness here. That's what we do for now.
    11633             PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
    11634             patchpoint->appendSomeRegister(basePtr);
    11635             patchpoint->appendSomeRegister(masked);
    11636             patchpoint->setGenerator(
    11637                 [] (CCallHelpers& jit, const StackmapGenerationParams& params) {
    11638                     jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr());
    11639                 });
    11640             patchpoint->effects = Effects::none();
    11641             return patchpoint;
    11642         }
    11643        
    11644         return ptr;
     11617    LValue caged(Gigacage::Kind kind, LValue ptr)
     11618    {
     11619        if (kind == Gigacage::Primitive) {
     11620            if (vm().primitiveGigacageEnabled().isStillValid())
     11621                m_graph.watchpoints().addLazily(vm().primitiveGigacageEnabled());
     11622            else
     11623                return ptr;
     11624        }
     11625       
     11626        LValue basePtr = m_out.constIntPtr(Gigacage::basePtr(kind));
     11627        LValue mask = m_out.constIntPtr(GIGACAGE_MASK);
     11628       
     11629        // We don't have to worry about B3 messing up the bitAnd. Also, we want to get B3's excellent
     11630        // codegen for 2-operand andq on x86-64.
     11631        LValue masked = m_out.bitAnd(ptr, mask);
     11632       
     11633        // But B3 will currently mess up the code generation of this add. Basically, any offset from what we
     11634        // compute here will get reassociated and folded with Gigacage::basePtr. There's a world in which
     11635        // moveConstants() observes that it needs to reassociate in order to hoist the big constants. But
     11636        // it's much easier to just block B3's badness here. That's what we do for now.
     11637        PatchpointValue* patchpoint = m_out.patchpoint(pointerType());
     11638        patchpoint->appendSomeRegister(basePtr);
     11639        patchpoint->appendSomeRegister(masked);
     11640        patchpoint->setGenerator(
     11641            [] (CCallHelpers& jit, const StackmapGenerationParams& params) {
     11642                jit.addPtr(params[1].gpr(), params[2].gpr(), params[0].gpr());
     11643            });
     11644        patchpoint->effects = Effects::none();
     11645        return patchpoint;
    1164511646    }
    1164611647   
Note: See TracChangeset for help on using the changeset viewer.