Skip to content

Commit cf0e840

Browse files
authored
Merge pull request #182 from ruby-syntax-tree/search-updates
Search updates
2 parents 78eea51 + 02b3c49 commit cf0e840

File tree

8 files changed

+276
-97
lines changed

8 files changed

+276
-97
lines changed

.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,4 @@
99
/vendor/
1010

1111
test.rb
12+
query.txt

README.md

+24
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ It is built with only standard library dependencies. It additionally ships with
1515
- [CLI](#cli)
1616
- [ast](#ast)
1717
- [check](#check)
18+
- [expr](#expr)
1819
- [format](#format)
1920
- [json](#json)
2021
- [match](#match)
@@ -26,6 +27,7 @@ It is built with only standard library dependencies. It additionally ships with
2627
- [SyntaxTree.read(filepath)](#syntaxtreereadfilepath)
2728
- [SyntaxTree.parse(source)](#syntaxtreeparsesource)
2829
- [SyntaxTree.format(source)](#syntaxtreeformatsource)
30+
- [SyntaxTree.search(source, query, &block)](#syntaxtreesearchsource-query-block)
2931
- [Nodes](#nodes)
3032
- [child_nodes](#child_nodes)
3133
- [Pattern matching](#pattern-matching)
@@ -129,6 +131,24 @@ To change the print width that you are checking against, specify the `--print-wi
129131
stree check --print-width=100 path/to/file.rb
130132
```
131133

134+
### expr
135+
136+
This command will output a Ruby case-match expression that would match correctly against the first expression of the input.
137+
138+
```sh
139+
stree expr path/to/file.rb
140+
```
141+
142+
For a file that contains `1 + 1`, you will receive:
143+
144+
```ruby
145+
SyntaxTree::Binary[
146+
left: SyntaxTree::Int[value: "1"],
147+
operator: :+,
148+
right: SyntaxTree::Int[value: "1"]
149+
]
150+
```
151+
132152
### format
133153

134154
This command will output the formatted version of each of the listed files. Importantly, it will not write that content back to the source files. It is meant to display the formatted version only.
@@ -312,6 +332,10 @@ This function takes an input string containing Ruby code and returns the syntax
312332

313333
This function takes an input string containing Ruby code, parses it into its underlying syntax tree, and formats it back out to a string. You can optionally pass a second argument to this method as well that is the maximum width to print. It defaults to `80`.
314334

335+
### SyntaxTree.search(source, query, &block)
336+
337+
This function takes an input string containing Ruby code, an input string containing a valid Ruby `in` clause expression that can be used to match against nodes in the tree (can be generated using `stree expr`, `stree match`, or `Node#construct_keys`), and a block. Each node that matches the given query will be yielded to the block. The block will receive the node as its only argument.
338+
315339
## Nodes
316340

317341
There are many different node types in the syntax tree. They are meant to be treated as immutable structs containing links to child nodes with minimal logic contained within their implementation. However, for the most part they all respond to a certain set of APIs, listed below.

lib/syntax_tree.rb

+7
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
require_relative "syntax_tree/visitor/with_environment"
2222

2323
require_relative "syntax_tree/parser"
24+
require_relative "syntax_tree/pattern"
2425
require_relative "syntax_tree/search"
2526

2627
# Syntax Tree is a suite of tools built on top of the internal CRuby parser. It
@@ -74,4 +75,10 @@ def self.read(filepath)
7475

7576
File.read(filepath, encoding: encoding)
7677
end
78+
79+
# Searches through the given source using the given pattern and yields each
80+
# node in the tree that matches the pattern to the given block.
81+
def self.search(source, query, &block)
82+
Search.new(Pattern.new(query).compile).scan(parse(source), &block)
83+
end
7784
end

lib/syntax_tree/cli.rb

+29-4
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,20 @@ def run(item)
188188
end
189189
end
190190

191+
# An action of the CLI that outputs a pattern-matching Ruby expression that
192+
# would match the first expression of the input given.
193+
class Expr < Action
194+
def run(item)
195+
case item.handler.parse(item.source)
196+
in Program[statements: Statements[body: [expression]]]
197+
puts expression.construct_keys
198+
else
199+
warn("The input to `stree expr` must be a single expression.")
200+
exit(1)
201+
end
202+
end
203+
end
204+
191205
# An action of the CLI that formats the input source and prints it out.
192206
class Format < Action
193207
def run(item)
@@ -219,10 +233,15 @@ class Search < Action
219233

220234
def initialize(query)
221235
query = File.read(query) if File.readable?(query)
222-
@search = SyntaxTree::Search.new(query)
223-
rescue SyntaxTree::Search::UncompilableError => error
224-
warn(error.message)
225-
exit(1)
236+
pattern =
237+
begin
238+
Pattern.new(query).compile
239+
rescue Pattern::CompilationError => error
240+
warn(error.message)
241+
exit(1)
242+
end
243+
244+
@search = SyntaxTree::Search.new(pattern)
226245
end
227246

228247
def run(item)
@@ -281,6 +300,10 @@ def run(item)
281300
#{Color.bold("stree doc [--plugins=...] [-e SCRIPT] FILE")}
282301
Print out the doc tree that would be used to format the given files
283302
303+
#{Color.bold("stree expr [-e SCRIPT] FILE")}
304+
Print out a pattern-matching Ruby expression that would match the first
305+
expression of the given files
306+
284307
#{Color.bold("stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE")}
285308
Print out the formatted version of the given files
286309
@@ -436,6 +459,8 @@ def run(argv)
436459
Debug.new(options)
437460
when "doc"
438461
Doc.new(options)
462+
when "e", "expr"
463+
Expr.new(options)
439464
when "f", "format"
440465
Format.new(options)
441466
when "help"

lib/syntax_tree/pattern.rb

+172
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# frozen_string_literal: true
2+
3+
module SyntaxTree
4+
# A pattern is an object that wraps a Ruby pattern matching expression. The
5+
# expression would normally be passed to an `in` clause within a `case`
6+
# expression or a rightward assignment expression. For example, in the
7+
# following snippet:
8+
#
9+
# case node
10+
# in Const[value: "SyntaxTree"]
11+
# end
12+
#
13+
# the pattern is the `Const[value: "SyntaxTree"]` expression. Within Syntax
14+
# Tree, every node generates these kinds of expressions using the
15+
# #construct_keys method.
16+
#
17+
# The pattern gets compiled into an object that responds to call by running
18+
# the #compile method. This method itself will run back through Syntax Tree to
19+
# parse the expression into a tree, then walk the tree to generate the
20+
# necessary callable objects. For example, if you wanted to compile the
21+
# expression above into a callable, you would:
22+
#
23+
# callable = SyntaxTree::Pattern.new("Const[value: 'SyntaxTree']").compile
24+
# callable.call(node)
25+
#
26+
# The callable object returned by #compile is guaranteed to respond to #call
27+
# with a single argument, which is the node to match against. It also is
28+
# guaranteed to respond to #===, which means it itself can be used in a `case`
29+
# expression, as in:
30+
#
31+
# case node
32+
# when callable
33+
# end
34+
#
35+
# If the query given to the initializer cannot be compiled into a valid
36+
# matcher (either because of a syntax error or because it is using syntax we
37+
# do not yet support) then a SyntaxTree::Pattern::CompilationError will be
38+
# raised.
39+
class Pattern
40+
# Raised when the query given to a pattern is either invalid Ruby syntax or
41+
# is using syntax that we don't yet support.
42+
class CompilationError < StandardError
43+
def initialize(repr)
44+
super(<<~ERROR)
45+
Syntax Tree was unable to compile the pattern you provided to search
46+
into a usable expression. It failed on to understand the node
47+
represented by:
48+
49+
#{repr}
50+
51+
Note that not all syntax supported by Ruby's pattern matching syntax
52+
is also supported by Syntax Tree's code search. If you're using some
53+
syntax that you believe should be supported, please open an issue on
54+
GitHub at https://github.com/ruby-syntax-tree/syntax_tree/issues/new.
55+
ERROR
56+
end
57+
end
58+
59+
attr_reader :query
60+
61+
def initialize(query)
62+
@query = query
63+
end
64+
65+
def compile
66+
program =
67+
begin
68+
SyntaxTree.parse("case nil\nin #{query}\nend")
69+
rescue Parser::ParseError
70+
raise CompilationError, query
71+
end
72+
73+
compile_node(program.statements.body.first.consequent.pattern)
74+
end
75+
76+
private
77+
78+
def combine_and(left, right)
79+
->(node) { left.call(node) && right.call(node) }
80+
end
81+
82+
def combine_or(left, right)
83+
->(node) { left.call(node) || right.call(node) }
84+
end
85+
86+
def compile_node(root)
87+
case root
88+
in AryPtn[constant:, requireds:, rest: nil, posts: []]
89+
compiled_constant = compile_node(constant) if constant
90+
91+
preprocessed = requireds.map { |required| compile_node(required) }
92+
93+
compiled_requireds = ->(node) do
94+
deconstructed = node.deconstruct
95+
96+
deconstructed.length == preprocessed.length &&
97+
preprocessed
98+
.zip(deconstructed)
99+
.all? { |(matcher, value)| matcher.call(value) }
100+
end
101+
102+
if compiled_constant
103+
combine_and(compiled_constant, compiled_requireds)
104+
else
105+
compiled_requireds
106+
end
107+
in Binary[left:, operator: :|, right:]
108+
combine_or(compile_node(left), compile_node(right))
109+
in Const[value:] if SyntaxTree.const_defined?(value)
110+
clazz = SyntaxTree.const_get(value)
111+
112+
->(node) { node.is_a?(clazz) }
113+
in Const[value:] if Object.const_defined?(value)
114+
clazz = Object.const_get(value)
115+
116+
->(node) { node.is_a?(clazz) }
117+
in ConstPathRef[
118+
parent: VarRef[value: Const[value: "SyntaxTree"]], constant:
119+
]
120+
compile_node(constant)
121+
in DynaSymbol[parts: []]
122+
symbol = :""
123+
124+
->(node) { node == symbol }
125+
in DynaSymbol[parts: [TStringContent[value:]]]
126+
symbol = value.to_sym
127+
128+
->(attribute) { attribute == value }
129+
in HshPtn[constant:, keywords:, keyword_rest: nil]
130+
compiled_constant = compile_node(constant)
131+
132+
preprocessed =
133+
keywords.to_h do |keyword, value|
134+
raise NoMatchingPatternError unless keyword.is_a?(Label)
135+
[keyword.value.chomp(":").to_sym, compile_node(value)]
136+
end
137+
138+
compiled_keywords = ->(node) do
139+
deconstructed = node.deconstruct_keys(preprocessed.keys)
140+
141+
preprocessed.all? do |keyword, matcher|
142+
matcher.call(deconstructed[keyword])
143+
end
144+
end
145+
146+
if compiled_constant
147+
combine_and(compiled_constant, compiled_keywords)
148+
else
149+
compiled_keywords
150+
end
151+
in RegexpLiteral[parts: [TStringContent[value:]]]
152+
regexp = /#{value}/
153+
154+
->(attribute) { regexp.match?(attribute) }
155+
in StringLiteral[parts: []]
156+
->(attribute) { attribute == "" }
157+
in StringLiteral[parts: [TStringContent[value:]]]
158+
->(attribute) { attribute == value }
159+
in SymbolLiteral[value:]
160+
symbol = value.value.to_sym
161+
162+
->(attribute) { attribute == symbol }
163+
in VarRef[value: Const => value]
164+
compile_node(value)
165+
in VarRef[value: Kw[value: "nil"]]
166+
->(attribute) { attribute.nil? }
167+
end
168+
rescue NoMatchingPatternError
169+
raise CompilationError, PP.pp(root, +"").chomp
170+
end
171+
end
172+
end

0 commit comments

Comments
 (0)