MongoDB World 2019: BSON Transpilers: Transpiling from Any Language to Any Language!
0 likes263 views
The document outlines a presentation by Anna Herlihy on how to integrate new programming languages into MongoDB Compass through the use of bson-transpilers. It discusses technical requirements, community involvement, and details the process of adding output languages, including tree-building, symbol tables, and visitor patterns for language translation. The presentation encourages contributions from developers to enable support for multiple languages within Compass.
![Visiting a Symbol
visitSymbol(node) {
const name = node.symbol;
node.type = this.Symbols[name];
if (node.type === undefined) {
throw new ReferenceError(`Symbol ${name} is undefined`);
}
return name;
}
First get the symbol
itself from the node](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-47-320.jpg)
![Visiting a Symbol
visitSymbol(node) {
const name = node.symbol;
node.type = this.Symbols[name];
if (node.type === undefined) {
throw new ReferenceError(`Symbol ${name} is undefined`);
}
return name;
}
Look up the name of
the symbol in the table
and “decorate” the
node with the results](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-48-320.jpg)
![Visiting a Symbol
visitSymbol(node) {
const name = node.symbol;
node.type = this.Symbols[name];
if (node.type === undefined) {
throw new ReferenceError(`Symbol ${name} is undefined`);
}
return name;
}
If it’s not in the table,
throw an exception](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-49-320.jpg)
![Visiting a Symbol
visitSymbol(node) {
const name = node.symbol;
node.type = this.Symbols[name];
if (node.type === undefined) {
throw new ReferenceError(`Symbol ${name} is undefined`);
}
return name;
}
Return the name of
the function to the
parent](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-50-320.jpg)
![What’s in the Symbol Table?
ObjectId:
id: "ObjectId"
callable: *constructor
args:
- [ *StringType, *NumericType, null ]
type: *ObjectIdType
attr: {}
template: *ObjectIdSymbolTemplate
argsTemplate: *ObjectIdSymbolArgsTemplate](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-51-320.jpg)
![What’s in a Symbol?
ObjectId:
id: "ObjectId"
callable: *constructor
args:
- [ *StringType, *NumericType, null ]
If the symbol is callable, this is where the arguments are defined. Each
element in the array is a positional argument and contains the list of
acceptable types. So ObjectId accepts one string or number argument, or no
arguments at all.](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-54-320.jpg)
![What’s in a Symbol?
ObjectId:
id: "ObjectId"
callable: *constructor
args:
- [ *StringType, *NumericType, null ]
type: *ObjectIdType
The return type of the function, or
if the symbol is a variable, the type
of the variable.](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-55-320.jpg)
![What’s in a Symbol?
ObjectId:
id: "ObjectId"
callable: *constructor
args:
- [ *StringType, *NumericType, null ]
type: *ObjectIdType
attr: {...}
Any attributes of the symbol. This is a sub-symbol table, i.e. a
mapping of names to symbols. Ex: ObjectId.fromDate()](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-56-320.jpg)
![What’s in a Symbol?
ObjectId:
id: "ObjectId"
callable: *constructor
args:
- [ *StringType, *NumericType, null ]
type: *ObjectIdType
attr: {}
template: *ObjectIdSymbolTemplate
These are functions that accept strings and return strings, and are responsible for doing the string
transformations from one language syntax to another language's syntax. template is the symbol
itself, and argsTemplate is applied to the args. These are specific to the output language and
defined in a separate file.](https://image.slidesharecdn.com/murrayhillday1100pmannaherlihymdbworld19bsontranspilers-190701161151/85/MongoDB-World-2019-BSON-Transpilers-Transpiling-from-Any-Language-to-Any-Language-57-320.jpg)
MongoDB World 2019: BSON Transpilers: Transpiling from Any Language to Any Language!
- 1. Anna Herlihy Senior Software Engineer Stockholm BSON-Transpilers or How To Add Your Favorite Language to Compass @annaisworking
- 2. 1. Compass Features 2. Technical Requirements 3. How to contribute!
- 3. Compass Tour
- 4. Export To Language (query)
- 5. Export To Language (aggregation)
- 6. But wait... Wouldn’t it be great if you could just write whatever language you want, directly into Compass?
- 8. BSON is a large enough subset to treat the problem as if we’re supporting the entire language syntax
- 9. Accept query or aggregation in any language Export query or aggregation to any language + = Requirements
- 14. ???
- 18. Complexity
- 19. For every input language we support: 1. We want to only have to do the work once 2. We want to define the input language without knowing or caring how many output languages exist
- 20. We do not want to have to rewrite the translation for every possible combination, that would be O(n2) and we want O(n)
- 21. For every input language we support: 1. We want to only have to do the work once 2. We want to define the input language without knowing or caring how many output languages exist
- 23. For every input language we support: 1. We want to only have to do the work once 2. We want to define the input language without knowing or caring how many output languages exist Same principle for target languages
- 25. Encouraging Community Involvement 1. Many communities are small + passionate 2. Open source ethos :) 3. How many people are compiler experts?
- 26. How to add your own output language
- 27. Compiler 101
- 29. Tree Building { today: Date() } START Object Literal {} Symbol today Function Call Args () Symbol Date key funcname value args The tree-building stage includes lexing, parsing, syntax analysis, and more!
- 31. Parser Generator ● Parsers are hard! ● Support for most languages ● We apply ANTLR to our input and we get a tree. The tree building stage is completely handled! ● Bonus: ANTLR generates a visitor class! OUR SAVIOR
- 33. The Visitor Pattern Visitors traverse trees by “visiting” each node For each type of node, the visitor calls the corresponding function. ➢ When the visitor sees a node that is a “string” type, it will call the visitString method and expect the generated code to be returned.
- 34. Extremely Simple Visitor Example ‘testing…testing’ START JavaScript Code
- 35. Extremely Simple Visitor Example ‘testing…testing’ START JavaScript Code “testing...testing” END Java Code
- 36. Extremely Simple Visitor Example ‘testing…testing’ START ANTLR “testing...testing” END string ’‘ testing...testing value
- 37. Extremely Simple Visitor Example ‘testing…testing’ START ANTLR “testing...testing” END string ’‘ testing...testing value
- 38. Extremely Simple Visitor Example ‘testing…testing’ START “testing...testing” END VISITOR string ’‘ testing...testing value
- 39. Extremely Simple Visitor Example class Visitor() extends ANTLRVisitor { visitString(node) { const val = node.value; return ‘“‘ + val + ‘”‘; } } string ’‘ testing...testing ‘testing…testing’ START “testing...testing” END value
- 40. What about functions or variables? ● We need to support native language features as well as BSON-specific types. ● ObjectId, Date, Decimal128, Timestamp, etc… ● How can we tell the difference between variables?
- 41. ObjectId() START Date() START JavaScript Input JavaScript Input
- 42. new java.util.Date() new ObjectId() ENDEND JavaScript Input Java Output ObjectId() START Date() START
- 43. Function Call Args () Symbol Date Function Call Args () Symbol ObjectId new java.util.Date() new ObjectId() ENDEND ObjectId() START Date() START
- 44. Function Call Args () Symbol Date Function Call Args () Symbol ObjectId The same visit methods are going to be called for both... To the visitor, these two trees look the same...
- 45. We need a Symbol Table!
- 46. Symbol Table ● Need a place to keep track of all the symbols, i.e. variable or function names. ● When the visitor reaches a Symbol node, it looks it up in the Symbol Table ● This is also a convenient place to differentiate between output languages...
- 47. Visiting a Symbol visitSymbol(node) { const name = node.symbol; node.type = this.Symbols[name]; if (node.type === undefined) { throw new ReferenceError(`Symbol ${name} is undefined`); } return name; } First get the symbol itself from the node
- 48. Visiting a Symbol visitSymbol(node) { const name = node.symbol; node.type = this.Symbols[name]; if (node.type === undefined) { throw new ReferenceError(`Symbol ${name} is undefined`); } return name; } Look up the name of the symbol in the table and “decorate” the node with the results
- 49. Visiting a Symbol visitSymbol(node) { const name = node.symbol; node.type = this.Symbols[name]; if (node.type === undefined) { throw new ReferenceError(`Symbol ${name} is undefined`); } return name; } If it’s not in the table, throw an exception
- 50. Visiting a Symbol visitSymbol(node) { const name = node.symbol; node.type = this.Symbols[name]; if (node.type === undefined) { throw new ReferenceError(`Symbol ${name} is undefined`); } return name; } Return the name of the function to the parent
- 51. What’s in the Symbol Table? ObjectId: id: "ObjectId" callable: *constructor args: - [ *StringType, *NumericType, null ] type: *ObjectIdType attr: {} template: *ObjectIdSymbolTemplate argsTemplate: *ObjectIdSymbolArgsTemplate
- 52. What’s in the Symbol Table? ObjectId: id: "ObjectId" The name of the attribute. Mostly used for error reporting.
- 53. What’s in the Symbol Table? ObjectId: id: "ObjectId" callable: *constructor There are 3 types of symbol: *func: a function name *constructor: also a function name, but may require a “new“ *var: a variable. Indicates that the symbol cannot be called.
- 54. What’s in a Symbol? ObjectId: id: "ObjectId" callable: *constructor args: - [ *StringType, *NumericType, null ] If the symbol is callable, this is where the arguments are defined. Each element in the array is a positional argument and contains the list of acceptable types. So ObjectId accepts one string or number argument, or no arguments at all.
- 55. What’s in a Symbol? ObjectId: id: "ObjectId" callable: *constructor args: - [ *StringType, *NumericType, null ] type: *ObjectIdType The return type of the function, or if the symbol is a variable, the type of the variable.
- 56. What’s in a Symbol? ObjectId: id: "ObjectId" callable: *constructor args: - [ *StringType, *NumericType, null ] type: *ObjectIdType attr: {...} Any attributes of the symbol. This is a sub-symbol table, i.e. a mapping of names to symbols. Ex: ObjectId.fromDate()
- 57. What’s in a Symbol? ObjectId: id: "ObjectId" callable: *constructor args: - [ *StringType, *NumericType, null ] type: *ObjectIdType attr: {} template: *ObjectIdSymbolTemplate These are functions that accept strings and return strings, and are responsible for doing the string transformations from one language syntax to another language's syntax. template is the symbol itself, and argsTemplate is applied to the args. These are specific to the output language and defined in a separate file.
- 58. Each output language has a file (in YAML) where the templates are defined.
- 59. To Recap ANTLR creates a tree from the user input We visit the ANTLR-generated tree using visitors When the visitor reaches a symbol, it looks up metadata in the Symbol Table. § The metadata includes template functions that specify what code should be generated So how do I add my own output language to Compass?
- 60. Add your own template file!! All you need to do to add an output language is fill out the templates!
- 61. symbols/sample_template.yaml There is a skeleton template file available To add a new output language: § fill out each template with the correct translation to your language. Templates mostly apply to symbols, but there are also templates for literals and other syntax.
- 65. NumberLong(‘1’) START Int64(‘1’) END Visitor symbols/python/templates.yaml LongTemplate(arg) { return `Int64(${arg})`; } symbols/java/templates.yaml LongTemplate(arg) { return `Long.parseLong(${arg})`; }
- 66. NumberLong(‘1’) START Long.parseLong( "1" ) Int64(‘1’) END Visitor symbols/python/templates.yaml LongTemplate(arg) { return `Int64(${arg})`; } symbols/java/templates.yaml LongTemplate(arg) { return `Long.parseLong(${arg})`; }
- 67. NumberLong(‘1’) START Long.parseLong( "1" ) Int64(‘1’) END Visitor symbols/python/templates.yaml LongTemplate(arg) { return `Int64(${arg})`; } symbols/java/templates.yaml LongTemplate(arg) { return `Long.parseLong(${arg})`; } symbols/csharp/templates.yaml LongTemplate(arg) { return `Convert.ToInt64(${arg})``; }
- 68. NumberLong(‘1’) START Long.parseLong( "1" ) Int64(‘1’) Convert.ToInt64( "1" ) END Visitor symbols/python/templates.yaml LongTemplate(arg) { return `Int64(${arg})`; } symbols/java/templates.yaml LongTemplate(arg) { return `Long.parseLong(${arg})`; } symbols/csharp/templates.yaml LongTemplate(arg) { return `Convert.ToInt64(${arg})``; }
- 69. Go forth and write templates!
- 70. Ruby, PHP, Go, R, Rust, C & more! ❖ Want to add an output language? ➢ Just fill out a symbol table file! ❖ Want to write an input language? ➢ Write a visitor
- 71. Future Features! We now have a pluggable transpiler from any language BSON to any language BSON….what can we do with it?
- 72. Import pipelines in whatever language you like
- 73. Expand the syntax to include driver usage
- 74. Generate examples for MongoDB University
- 75. Put it in front of the shell!
- 76. Expand it to support 100% language syntax! Use it for anything at all...
- 77. Thanks to the Compass Team! ★ Alena Khineika ★ Irina Shestak ★ Durran Jordan
- 78. Thank you! Everything I said, in much more detail: github.com/mongodb-js/bson-transpilers > CONTRIBUTING.md Questions? [email protected] or [email protected]