Changeset 220118 in webkit for trunk/Source/JavaScriptCore/jsc.cpp

Timestamp:

Aug 1, 2017, 6:50:16 PM (8 years ago)

Author:

[email protected]

Message:

Bmalloc and GC should put auxiliaries (butterflies, typed array backing stores) in a gigacage (separate multi-GB VM region)
​https://bugs.webkit.org/show_bug.cgi?id=174727

Reviewed by Mark Lam.
Source/bmalloc:

This adds a mechanism for managing multiple isolated heaps in bmalloc. For now, these isoheaps
(isolated heaps) have a very simple relationship with each other and with the rest of bmalloc:

• You have to choose how many isoheaps you will have statically. See numHeaps in HeapKind.h.

• Because numHeaps is static, each isoheap gets fast thread-local allocation. Basically, we have a Cache for each heap kind.

• Each isoheap gets its own Heap.

• Each Heap gets a scavenger thread.

• Some things, like Zone/VMHeap/Scavenger, are per-process.

Most of the per-HeapKind functionality is handled by PerHeapKind<>.

This approach is ideal for supporting special per-HeapKind behaviors. For now we have two heaps:
the Primary heap for normal malloc and the Gigacage. The gigacage is a 64GB-aligned 64GB virtual
region that we now use for variable-length random-access allocations. No Primary allocations will
go into the Gigacage.

• CMakeLists.txt:
• bmalloc.xcodeproj/project.pbxproj:
• bmalloc/AllocationKind.h: Added.
• bmalloc/Allocator.cpp:

(bmalloc::Allocator::Allocator):
(bmalloc::Allocator::tryAllocate):
(bmalloc::Allocator::allocateImpl):
(bmalloc::Allocator::reallocate):
(bmalloc::Allocator::refillAllocatorSlowCase):
(bmalloc::Allocator::allocateLarge):

• bmalloc/Allocator.h:
• bmalloc/BExport.h: Added.
• bmalloc/Cache.cpp:

(bmalloc::Cache::scavenge):
(bmalloc::Cache::Cache):
(bmalloc::Cache::tryAllocateSlowCaseNullCache):
(bmalloc::Cache::allocateSlowCaseNullCache):
(bmalloc::Cache::deallocateSlowCaseNullCache):
(bmalloc::Cache::reallocateSlowCaseNullCache):
(bmalloc::Cache::operator new): Deleted.
(bmalloc::Cache::operator delete): Deleted.

• bmalloc/Cache.h:

(bmalloc::Cache::tryAllocate):
(bmalloc::Cache::allocate):
(bmalloc::Cache::deallocate):
(bmalloc::Cache::reallocate):

• bmalloc/Deallocator.cpp:

(bmalloc::Deallocator::Deallocator):
(bmalloc::Deallocator::scavenge):
(bmalloc::Deallocator::processObjectLog):
(bmalloc::Deallocator::deallocateSlowCase):

• bmalloc/Deallocator.h:
• bmalloc/Gigacage.cpp: Added.

(Gigacage::Callback::Callback):
(Gigacage::Callback::function):
(Gigacage::Callbacks::Callbacks):
(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):

• bmalloc/Gigacage.h: Added.

(Gigacage::caged):
(Gigacage::isCaged):

• bmalloc/Heap.cpp:

(bmalloc::Heap::Heap):
(bmalloc::Heap::usingGigacage):
(bmalloc::Heap::concurrentScavenge):
(bmalloc::Heap::splitAndAllocate):
(bmalloc::Heap::tryAllocateLarge):
(bmalloc::Heap::allocateLarge):
(bmalloc::Heap::shrinkLarge):
(bmalloc::Heap::deallocateLarge):

• bmalloc/Heap.h:

(bmalloc::Heap::mutex):
(bmalloc::Heap::kind const):
(bmalloc::Heap::setScavengerThreadQOSClass): Deleted.

• bmalloc/HeapKind.h: Added.
• bmalloc/ObjectType.cpp:

(bmalloc::objectType):

• bmalloc/ObjectType.h:
• bmalloc/PerHeapKind.h: Added.

(bmalloc::PerHeapKindBase::PerHeapKindBase):
(bmalloc::PerHeapKindBase::size):
(bmalloc::PerHeapKindBase::at):
(bmalloc::PerHeapKindBase::at const):
(bmalloc::PerHeapKindBase::operator[]):
(bmalloc::PerHeapKindBase::operator[] const):
(bmalloc::StaticPerHeapKind::StaticPerHeapKind):
(bmalloc::PerHeapKind::PerHeapKind):
(bmalloc::PerHeapKind::~PerHeapKind):

• bmalloc/PerThread.h:

(bmalloc::PerThread<T>::destructor):
(bmalloc::PerThread<T>::getSlowCase):
(bmalloc::PerThreadStorage<Cache>::get): Deleted.
(bmalloc::PerThreadStorage<Cache>::init): Deleted.

• bmalloc/Scavenger.cpp: Added.

(bmalloc::Scavenger::Scavenger):
(bmalloc::Scavenger::scavenge):

• bmalloc/Scavenger.h: Added.

(bmalloc::Scavenger::setScavengerThreadQOSClass):
(bmalloc::Scavenger::requestedScavengerThreadQOSClass const):

• bmalloc/VMHeap.cpp:

(bmalloc::VMHeap::VMHeap):
(bmalloc::VMHeap::tryAllocateLargeChunk):

• bmalloc/VMHeap.h:
• bmalloc/Zone.cpp:

(bmalloc::Zone::Zone):

• bmalloc/Zone.h:
• bmalloc/bmalloc.h:

(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::realloc):
(bmalloc::api::tryLargeMemalignVirtual):
(bmalloc::api::free):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::isEnabled):
(bmalloc::api::setScavengerThreadQOSClass):

• bmalloc/mbmalloc.cpp:

Source/JavaScriptCore:

This adopts the Gigacage for the GigacageSubspace, which we use for Auxiliary allocations. Also, in
one place in the code - the FTL codegen for butterfly and typed array access - we "cage" the accesses
themselves. Basically, we do masking to ensure that the pointer points into the gigacage.

This is neutral on JetStream.

• CMakeLists.txt:
• JavaScriptCore.xcodeproj/project.pbxproj:
• b3/B3InsertionSet.cpp:

(JSC::B3::InsertionSet::execute):

• dfg/DFGAbstractInterpreterInlines.h:

(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):

• dfg/DFGArgumentsEliminationPhase.cpp:
• dfg/DFGClobberize.cpp:

(JSC::DFG::readsOverlap):

• dfg/DFGClobberize.h:

(JSC::DFG::clobberize):

• dfg/DFGDoesGC.cpp:

(JSC::DFG::doesGC):

• dfg/DFGFixedButterflyAccessUncagingPhase.cpp: Added.

(JSC::DFG::performFixedButterflyAccessUncaging):

• dfg/DFGFixedButterflyAccessUncagingPhase.h: Added.
• dfg/DFGFixupPhase.cpp:

(JSC::DFG::FixupPhase::fixupNode):

• dfg/DFGHeapLocation.cpp:

(WTF::printInternal):

• dfg/DFGHeapLocation.h:
• dfg/DFGNodeType.h:
• dfg/DFGPlan.cpp:

(JSC::DFG::Plan::compileInThreadImpl):

• dfg/DFGPredictionPropagationPhase.cpp:
• dfg/DFGSafeToExecute.h:

(JSC::DFG::safeToExecute):

• dfg/DFGSpeculativeJIT.cpp:

(JSC::DFG::SpeculativeJIT::compileGetButterfly):

• dfg/DFGSpeculativeJIT32_64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

• dfg/DFGTypeCheckHoistingPhase.cpp:

(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantStructureChecks):
(JSC::DFG::TypeCheckHoistingPhase::identifyRedundantArrayChecks):

• ftl/FTLCapabilities.cpp:

(JSC::FTL::canCompile):

• ftl/FTLLowerDFGToB3.cpp:

(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileGetButterfly):
(JSC::FTL::DFG::LowerDFGToB3::compileGetIndexedPropertyStorage):
(JSC::FTL::DFG::LowerDFGToB3::compileGetByVal):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharAt):
(JSC::FTL::DFG::LowerDFGToB3::compileStringCharCodeAt):
(JSC::FTL::DFG::LowerDFGToB3::compileGetMapBucket):
(JSC::FTL::DFG::LowerDFGToB3::compileGetDirectPname):
(JSC::FTL::DFG::LowerDFGToB3::compileToLowerCase):
(JSC::FTL::DFG::LowerDFGToB3::caged):

• heap/GigacageSubspace.cpp: Added.

(JSC::GigacageSubspace::GigacageSubspace):
(JSC::GigacageSubspace::~GigacageSubspace):
(JSC::GigacageSubspace::tryAllocateAlignedMemory):
(JSC::GigacageSubspace::freeAlignedMemory):
(JSC::GigacageSubspace::canTradeBlocksWith):

• heap/GigacageSubspace.h: Added.
• heap/Heap.cpp:

(JSC::Heap::Heap):
(JSC::Heap::lastChanceToFinalize):
(JSC::Heap::finalize):
(JSC::Heap::sweepInFinalize):
(JSC::Heap::updateAllocationLimits):
(JSC::Heap::shouldDoFullCollection):
(JSC::Heap::collectIfNecessaryOrDefer):
(JSC::Heap::reportWebAssemblyFastMemoriesAllocated): Deleted.
(JSC::Heap::webAssemblyFastMemoriesThisCycleAtThreshold const): Deleted.
(JSC::Heap::sweepLargeAllocations): Deleted.
(JSC::Heap::didAllocateWebAssemblyFastMemories): Deleted.

• heap/Heap.h:
• heap/LargeAllocation.cpp:

(JSC::LargeAllocation::tryCreate):
(JSC::LargeAllocation::destroy):

• heap/MarkedAllocator.cpp:

(JSC::MarkedAllocator::tryAllocateWithoutCollecting):
(JSC::MarkedAllocator::tryAllocateBlock):

• heap/MarkedBlock.cpp:

(JSC::MarkedBlock::tryCreate):
(JSC::MarkedBlock::Handle::Handle):
(JSC::MarkedBlock::Handle::~Handle):
(JSC::MarkedBlock::Handle::didAddToAllocator):
(JSC::MarkedBlock::Handle::subspace const): Deleted.

• heap/MarkedBlock.h:

(JSC::MarkedBlock::Handle::subspace const):

• heap/MarkedSpace.cpp:

(JSC::MarkedSpace::~MarkedSpace):
(JSC::MarkedSpace::freeMemory):
(JSC::MarkedSpace::prepareForAllocation):
(JSC::MarkedSpace::addMarkedAllocator):
(JSC::MarkedSpace::findEmptyBlockToSteal): Deleted.

• heap/MarkedSpace.h:

(JSC::MarkedSpace::firstAllocator const):
(JSC::MarkedSpace::allocatorForEmptyAllocation const): Deleted.

• heap/Subspace.cpp:

(JSC::Subspace::Subspace):
(JSC::Subspace::canTradeBlocksWith):
(JSC::Subspace::tryAllocateAlignedMemory):
(JSC::Subspace::freeAlignedMemory):
(JSC::Subspace::prepareForAllocation):
(JSC::Subspace::findEmptyBlockToSteal):

• heap/Subspace.h:

(JSC::Subspace::didCreateFirstAllocator):

• heap/SubspaceInlines.h:

(JSC::Subspace::forEachAllocator):
(JSC::Subspace::forEachMarkedBlock):
(JSC::Subspace::forEachNotEmptyMarkedBlock):

• jit/JITPropertyAccess.cpp:

(JSC::JIT::emitDoubleLoad):
(JSC::JIT::emitContiguousLoad):
(JSC::JIT::emitArrayStorageLoad):
(JSC::JIT::emitGenericContiguousPutByVal):
(JSC::JIT::emitArrayStoragePutByVal):
(JSC::JIT::emit_op_get_from_scope):
(JSC::JIT::emit_op_put_to_scope):
(JSC::JIT::emitIntTypedArrayGetByVal):
(JSC::JIT::emitFloatTypedArrayGetByVal):
(JSC::JIT::emitIntTypedArrayPutByVal):
(JSC::JIT::emitFloatTypedArrayPutByVal):

• jsc.cpp:

(fillBufferWithContentsOfFile):
(functionReadFile):
(gigacageDisabled):
(jscmain):

• llint/LowLevelInterpreter64.asm:
• runtime/ArrayBuffer.cpp:

(JSC::ArrayBufferContents::tryAllocate):
(JSC::ArrayBuffer::createAdopted):
(JSC::ArrayBuffer::createFromBytes):
(JSC::ArrayBuffer::tryCreate):

• runtime/IndexingHeader.h:
• runtime/InitializeThreading.cpp:

(JSC::initializeThreading):

• runtime/JSArrayBuffer.cpp:
• runtime/JSArrayBufferView.cpp:

(JSC::JSArrayBufferView::ConstructionContext::ConstructionContext):
(JSC::JSArrayBufferView::finalize):

• runtime/JSLock.cpp:

(JSC::JSLock::didAcquireLock):

• runtime/JSObject.h:
• runtime/Options.cpp:

(JSC::recomputeDependentOptions):

• runtime/Options.h:
• runtime/ScopedArgumentsTable.h:
• runtime/VM.cpp:

(JSC::VM::VM):
(JSC::VM::~VM):
(JSC::VM::gigacageDisabledCallback):
(JSC::VM::gigacageDisabled):

• runtime/VM.h:

(JSC::VM::fireGigacageEnabledIfNecessary):
(JSC::VM::gigacageEnabled):

• wasm/WasmB3IRGenerator.cpp:

(JSC::Wasm::B3IRGenerator::B3IRGenerator):
(JSC::Wasm::B3IRGenerator::emitCheckAndPreparePointer):

• wasm/WasmCodeBlock.cpp:

(JSC::Wasm::CodeBlock::isSafeToRun):

• wasm/WasmMemory.cpp:

(JSC::Wasm::makeString):
(JSC::Wasm::Memory::create):
(JSC::Wasm::Memory::~Memory):
(JSC::Wasm::Memory::addressIsInActiveFastMemory):
(JSC::Wasm::Memory::grow):
(JSC::Wasm::Memory::initializePreallocations): Deleted.
(JSC::Wasm::Memory::maxFastMemoryCount): Deleted.

• wasm/WasmMemory.h:
• wasm/js/JSWebAssemblyInstance.cpp:

(JSC::JSWebAssemblyInstance::create):

• wasm/js/JSWebAssemblyMemory.cpp:

(JSC::JSWebAssemblyMemory::grow):
(JSC::JSWebAssemblyMemory::finishCreation):

• wasm/js/JSWebAssemblyMemory.h:

(JSC::JSWebAssemblyMemory::subspaceFor):

Source/WebCore:

No new tests because no change in behavior.

Needed to teach Metal how to allocate in the Gigacage.

• platform/graphics/cocoa/GPUBufferMetal.mm:

(WebCore::GPUBuffer::GPUBuffer):
(WebCore::GPUBuffer::contents):

Source/WebKit:

The WebProcess should never disable the Gigacage by allocating typed arrays outside the Gigacage. So,
we add a callback that crashes the process.

• WebProcess/WebProcess.cpp:

(WebKit::gigacageDisabled):
(WebKit::m_webSQLiteDatabaseTracker):

Source/WTF:

For the Gigacage project to have minimal impact, we need to have some abstraction that allows code to
avoid having to guard itself with #if's. This adds a Gigacage abstraction that overlays the Gigacage
namespace from bmalloc, which always lets you call things like Gigacage::caged and Gigacage::tryMalloc.

Because of how many places need to possibly allocate in a gigacage, or possibly perform caged accesses,
it's better to hide the question of whether or not it's enabled inside this API.

• WTF.xcodeproj/project.pbxproj:
• wtf/CMakeLists.txt:
• wtf/FastMalloc.cpp:
• wtf/Gigacage.cpp: Added.

(Gigacage::tryMalloc):
(Gigacage::tryAllocateVirtualPages):
(Gigacage::freeVirtualPages):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):

• wtf/Gigacage.h: Added.

(Gigacage::ensureGigacage):
(Gigacage::disableGigacage):
(Gigacage::addDisableCallback):
(Gigacage::removeDisableCallback):
(Gigacage::caged):
(Gigacage::isCaged):
(Gigacage::tryAlignedMalloc):
(Gigacage::alignedFree):
(Gigacage::free):

File:

• trunk/Source/JavaScriptCore/jsc.cpp (modified) (6 diffs)

trunk/Source/JavaScriptCore/jsc.cpp

@@ -79 +79 @@
 #include "TypeProfiler.h"
 #include "TypeProfilerLog.h"
+#include "TypedArrayInlines.h"
 #include "WasmContext.h"
 #include "WasmFaultSignalHandler.h"
@@ -985 +986 @@
 
 static bool fillBufferWithContentsOfFile(const String& fileName, Vector<char>& buffer);
+static RefPtr<Uint8Array> fillBufferWithContentsOfFile(const String& fileName);
 
 class CommandLine;
@@ -1709 +1711 @@
 }
 
+static RefPtr<Uint8Array> fillBufferWithContentsOfFile(FILE* file)
+{
+    fseek(file, 0, SEEK_END);
+    size_t bufferCapacity = ftell(file);
+    fseek(file, 0, SEEK_SET);
+    RefPtr<Uint8Array> result = Uint8Array::create(bufferCapacity);
+    size_t readSize = fread(result->data(), 1, bufferCapacity, file);
+    if (readSize != bufferCapacity)
+        return nullptr;
+    return result;
+}
+
+static RefPtr<Uint8Array> fillBufferWithContentsOfFile(const String& fileName)
+{
+    FILE* f = fopen(fileName.utf8().data(), "rb");
+    if (!f) {
+        fprintf(stderr, "Could not open file: %s\n", fileName.utf8().data());
+        return nullptr;
+    }
+
+    RefPtr<Uint8Array> result = fillBufferWithContentsOfFile(f);
+    fclose(f);
+
+    return result;
+}
+
 static bool fillBufferWithContentsOfFile(FILE* file, Vector<char>& buffer)
 {
@@ -2277 +2305 @@
     }
 
-    Vector<char> content;
-    if (!fillBufferWithContentsOfFile(fileName, content))
+    RefPtr<Uint8Array> content = fillBufferWithContentsOfFile(fileName);
+    if (!content)
         return throwVMError(exec, scope, "Could not open file.");
 
     if (!isBinary)
-        return JSValue::encode(jsString(exec, stringFromUTF(content)));
+        return JSValue::encode(jsString(exec, String::fromUTF8WithLatin1Fallback(content->data(), content->length())));
 
     Structure* structure = exec->lexicalGlobalObject()->typedArrayStructure(TypeUint8);
-    auto length = content.size();
-    JSObject* result = createUint8TypedArray(exec, structure, ArrayBuffer::createFromBytes(content.releaseBuffer().leakPtr(), length, [] (void* p) { fastFree(p); }), 0, length);
+    JSObject* result = JSUint8Array::create(vm, structure, WTFMove(content));
     RETURN_IF_EXCEPTION(scope, encodedJSValue());
 
@@ -3776 +3803 @@
 }
 
+static void gigacageDisabled(void*)
+{
+    dataLog("Gigacage disabled! Aborting.\n");
+    UNREACHABLE_FOR_PLATFORM();
+}
+
 int jscmain(int argc, char** argv)
 {
@@ -3794 +3827 @@
     JSC::Wasm::enableFastMemory();
 #endif
+    if (GIGACAGE_ENABLED)
+        Gigacage::addDisableCallback(gigacageDisabled, nullptr);
 
     int result;