Changeset 184337 in webkit for trunk/Source/JavaScriptCore/parser/Lexer.cpp

Timestamp:

May 14, 2015, 9:07:54 AM (10 years ago)

Author:

Yusuke Suzuki

Message:

[ES6] Implement tagged templates
​https://bugs.webkit.org/show_bug.cgi?id=143183

Reviewed by Oliver Hunt.

This patch implements ES6 tagged templates.
In tagged templates, the function takes the template object.

The template object contains the raw and cooked template strings,
so when parsing the tagged templates, we need to tokenize the raw and cooked strings.
While tagged templates require the both strings, the template literal only requires
the cooked strings. So when tokenizing under the template literal context,
we only builds the cooked strings.

As per ES6 spec, the template objects for the same raw strings are shared in the same realm.
The template objects is cached. And every time we evaluate the same tagged templates,
the same (cached) template objects are used.
Since the spec freezes this template objects completely,
we cannot attach some properties to it.
So we can say that it behaves as if the template objects are the primitive values (like JSString).
Since we cannot attach properties, the only way to test the identity of the template object is comparing. (===)
As the result, when there is no reference to the template object, we can garbage collect it
because the user has no way to test that the newly created template object does not equal
to the already collected template object.

So, to implement tagged templates, we implement the following components.

1. JSTemplateRegistryKey

It holds the template registry key and it does not exposed to users.
TemplateRegistryKey holds the vector of raw and cooked strings with the pre-computed hash value.
When obtaining the template object for the (statically, a.k.a. at the parsing time) given raw string vectors,
we use this JSTemplateRegistryKey as a key to the map and look up the template object from
TemplateRegistry.
JSTemplateRegistryKey is created at the bytecode compiling time and
stored in the CodeBlock as like as JSString content values.

2. TemplateRegistry

This manages the cached template objects.
It holds the weak map (JSTemplateRegistryKey -> the template object).
The template object is weakly referenced.
So if there is no reference to the template object,
the template object is automatically GC-ed.
When looking up the template object, it searches the cached template object.
If it is found, it is returned to the users.
If there is no cached template objects, it creates the new template object and
stores it with the given template registry key.

• CMakeLists.txt:
• JavaScriptCore.vcxproj/JavaScriptCore.vcxproj:
• JavaScriptCore.vcxproj/JavaScriptCore.vcxproj.filters:
• JavaScriptCore.xcodeproj/project.pbxproj:
• bytecompiler/BytecodeGenerator.cpp:

(JSC::BytecodeGenerator::addTemplateRegistryKeyConstant):
(JSC::BytecodeGenerator::emitGetTemplateObject):

• bytecompiler/BytecodeGenerator.h:
• bytecompiler/NodesCodegen.cpp:

(JSC::TaggedTemplateNode::emitBytecode):
(JSC::TemplateLiteralNode::emitBytecode): Deleted.

• parser/ASTBuilder.h:

(JSC::ASTBuilder::createTaggedTemplate):
(JSC::ASTBuilder::createTemplateLiteral): Deleted.

• parser/Lexer.cpp:

(JSC::Lexer<T>::setCode):
(JSC::Lexer<T>::parseTemplateLiteral):
(JSC::Lexer<T>::lex):
(JSC::Lexer<T>::scanTrailingTemplateString):
(JSC::Lexer<T>::clear):

• parser/Lexer.h:

(JSC::Lexer<T>::makeEmptyIdentifier):

• parser/NodeConstructors.h:

(JSC::TaggedTemplateNode::TaggedTemplateNode):
(JSC::TemplateLiteralNode::TemplateLiteralNode): Deleted.

• parser/Nodes.h:

(JSC::TemplateLiteralNode::templateStrings):
(JSC::TemplateLiteralNode::templateExpressions):
(JSC::TaggedTemplateNode::templateLiteral):

• parser/Parser.cpp:

(JSC::Parser<LexerType>::parseTemplateString):
(JSC::Parser<LexerType>::parseTemplateLiteral):
(JSC::Parser<LexerType>::parsePrimaryExpression):
(JSC::Parser<LexerType>::parseMemberExpression):

• parser/Parser.h:
• parser/ParserArena.h:

(JSC::IdentifierArena::makeEmptyIdentifier):

• parser/SyntaxChecker.h:

(JSC::SyntaxChecker::createTaggedTemplate):
(JSC::SyntaxChecker::createTemplateLiteral): Deleted.

• runtime/CommonIdentifiers.h:
• runtime/JSGlobalObject.cpp:

(JSC::getTemplateObject):
(JSC::JSGlobalObject::JSGlobalObject):
(JSC::JSGlobalObject::init):

• runtime/JSGlobalObject.h:

(JSC::JSGlobalObject::templateRegistry):

• runtime/JSTemplateRegistryKey.cpp: Added.

(JSC::JSTemplateRegistryKey::JSTemplateRegistryKey):
(JSC::JSTemplateRegistryKey::create):
(JSC::JSTemplateRegistryKey::destroy):

• runtime/JSTemplateRegistryKey.h: Added.
• runtime/ObjectConstructor.cpp:

(JSC::objectConstructorFreeze):

• runtime/ObjectConstructor.h:
• runtime/TemplateRegistry.cpp: Added.

(JSC::TemplateRegistry::TemplateRegistry):
(JSC::TemplateRegistry::getTemplateObject):

• runtime/TemplateRegistry.h: Added.
• runtime/TemplateRegistryKey.h: Added.

(JSC::TemplateRegistryKey::isDeletedValue):
(JSC::TemplateRegistryKey::isEmptyValue):
(JSC::TemplateRegistryKey::hash):
(JSC::TemplateRegistryKey::rawStrings):
(JSC::TemplateRegistryKey::cookedStrings):
(JSC::TemplateRegistryKey::operator==):
(JSC::TemplateRegistryKey::operator!=):
(JSC::TemplateRegistryKey::Hasher::hash):
(JSC::TemplateRegistryKey::Hasher::equal):
(JSC::TemplateRegistryKey::TemplateRegistryKey):

• runtime/VM.cpp:

(JSC::VM::VM):

• runtime/VM.h:
• tests/stress/tagged-templates-identity.js: Added.

(shouldBe):

• tests/stress/tagged-templates-raw-strings.js: Added.

(shouldBe):
(tag):
(testEval):

• tests/stress/tagged-templates-syntax.js: Added.

(tag):
(testSyntax):
(testSyntaxError):

• tests/stress/tagged-templates-template-object.js: Added.

(shouldBe):
(tag):

• tests/stress/tagged-templates-this.js: Added.

(shouldBe):
(tag):

• tests/stress/tagged-templates.js: Added.

(shouldBe):
(raw):
(cooked):
(Counter):

File:

• trunk/Source/JavaScriptCore/parser/Lexer.cpp (modified) (9 diffs)

trunk/Source/JavaScriptCore/parser/Lexer.cpp

@@ -570 +570 @@
     m_buffer8.reserveInitialCapacity(initialReadBufferCapacity);
     m_buffer16.reserveInitialCapacity((m_codeEnd - m_code) / 2);
+    m_bufferForRawTemplateString16.reserveInitialCapacity(initialReadBufferCapacity);
     
     if (LIKELY(m_code < m_codeEnd))
@@ -1375 +1376 @@
 
 template <typename T>
-template <bool shouldBuildStrings> typename Lexer<T>::StringParseResult Lexer<T>::parseTemplateLiteral(JSTokenData* tokenData)
+template <bool shouldBuildStrings> typename Lexer<T>::StringParseResult Lexer<T>::parseTemplateLiteral(JSTokenData* tokenData, RawStringsBuildMode rawStringsBuildMode)
 {
     const T* stringStart = currentSourcePtr();
@@ -1436 +1437 @@
                 if (m_current == '\r') {
                     // Normalize <CR>, <CR><LF> to <LF>.
-                    if (stringStart != currentSourcePtr() && shouldBuildStrings)
-                        append16(stringStart, currentSourcePtr() - stringStart);
-                    if (shouldBuildStrings)
+                    if (shouldBuildStrings) {
+                        if (stringStart != currentSourcePtr())
+                            append16(stringStart, currentSourcePtr() - stringStart);
+                        if (rawStringStart != currentSourcePtr() && rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
+                            m_bufferForRawTemplateString16.append(rawStringStart, currentSourcePtr() - rawStringStart);
+
                         record16('\n');
+                        if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
+                            m_bufferForRawTemplateString16.append('\n');
+                    }
                     lineNumberAdder.add(m_current);
                     shift();
@@ -1447 +1454 @@
                     }
                     stringStart = currentSourcePtr();
+                    rawStringStart = currentSourcePtr();
                 } else {
                     lineNumberAdder.add(m_current);
@@ -1462 +1470 @@
     bool isTail = m_current == '`';
 
-    if (currentSourcePtr() != stringStart && shouldBuildStrings)
-        append16(stringStart, currentSourcePtr() - stringStart);
+    if (shouldBuildStrings) {
+        if (currentSourcePtr() != stringStart)
+            append16(stringStart, currentSourcePtr() - stringStart);
+        if (rawStringStart != currentSourcePtr() && rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
+            m_bufferForRawTemplateString16.append(rawStringStart, currentSourcePtr() - rawStringStart);
+    }
 
     if (shouldBuildStrings) {
         tokenData->cooked = makeIdentifier(m_buffer16.data(), m_buffer16.size());
-        // TODO: While line terminator normalization (e.g. <CR> => <LF>) should be applied to both the raw and cooked representations,
-        // this raw implementation just slices the source string. As a result, line terminators appear in the raw representation without normalization.
-        // For example, when parsing `<CR>`, <CR> appears in the raw representation.
-        // While non-tagged template literals don't use the raw representation, tagged templates use the raw representation.
-        // So line terminator normalization should be applied to the raw representation when implementing tagged templates.
-        tokenData->raw = makeIdentifier(rawStringStart, currentSourcePtr() - rawStringStart);
+        // Line terminator normalization (e.g. <CR> => <LF>) should be applied to both the raw and cooked representations.
+        if (rawStringsBuildMode == RawStringsBuildMode::BuildRawStrings)
+            tokenData->raw = makeIdentifier(m_bufferForRawTemplateString16.data(), m_bufferForRawTemplateString16.size());
+        else
+            tokenData->raw = makeEmptyIdentifier();
     } else {
-        tokenData->cooked = nullptr;
-        tokenData->raw = nullptr;
+        tokenData->cooked = makeEmptyIdentifier();
+        tokenData->raw = makeEmptyIdentifier();
     }
     tokenData->isTail = isTail;
 
     m_buffer16.shrink(0);
+    m_bufferForRawTemplateString16.shrink(0);
 
     if (isTail) {
@@ -2126 +2138 @@
         StringParseResult result = StringCannotBeParsed;
         if (lexerFlags & LexerFlagsDontBuildStrings)
-            result = parseTemplateLiteral<false>(tokenData);
+            result = parseTemplateLiteral<false>(tokenData, RawStringsBuildMode::BuildRawStrings);
         else
-            result = parseTemplateLiteral<true>(tokenData);
+            result = parseTemplateLiteral<true>(tokenData, RawStringsBuildMode::BuildRawStrings);
 
         if (UNLIKELY(result != StringParsedSuccessfully)) {
@@ -2332 +2344 @@
 #if ENABLE(ES6_TEMPLATE_LITERAL_SYNTAX)
 template <typename T>
-JSTokenType Lexer<T>::scanTrailingTemplateString(JSToken* tokenRecord)
+JSTokenType Lexer<T>::scanTrailingTemplateString(JSToken* tokenRecord, RawStringsBuildMode rawStringsBuildMode)
 {
     JSTokenData* tokenData = &tokenRecord->m_data;
@@ -2341 +2353 @@
     // Leading closing brace } is already shifted in the previous token scan.
     // So in this re-scan phase, shift() is not needed here.
-    StringParseResult result = parseTemplateLiteral<true>(tokenData);
+    StringParseResult result = parseTemplateLiteral<true>(tokenData, rawStringsBuildMode);
     JSTokenType token = ERRORTOK;
     if (UNLIKELY(result != StringParsedSuccessfully)) {
@@ -2375 +2387 @@
     m_buffer16.swap(newBuffer16);
 
+    Vector<UChar> newBufferForRawTemplateString16;
+    m_bufferForRawTemplateString16.swap(newBufferForRawTemplateString16);
+
     m_isReparsing = false;
 }