Ruby scanner sees Symbol instead of Operator

[nathany]

In a shortened condition, it seems that the operator : is wrongly classified as a Symbol when it is followed by an identifier without a space. See: http://coderay.rubychan.de/rays/show/6917 - @bovi (Daniel Bovensiepen)

---

It's a known problem. It's hard to tell ?: and :sym apart without doing false positives. In this case, it really looks like a Symbol :) - @korny

But I can try. Thank you for the test code:

n=->k{k==0?'':n[k-1]+'{'+n[k-1]+'}'}

From Redmine: http://odd-eyed-code.org/issues/266

[korny]

Funny :) Pygments is having the same issue.

[nathany]

Indeed. I surprised Ruby can parse code like that :-P

[korny]

Easy to parse, hard to scan…

[nathany]

This reminds me that YARD uses its own syntax highlighter based on Ripper. I'll have to see how it fairs.

Sadly, Ripper is MRI 1.9-only for now, it's not yet supported by JRuby.

[nathany]

require 'ripper'
require 'pp'
pp Ripper.lex("n=->k{k==0?'':n[k-1]+'{'+n[k-1]+'}'}"); nil

...
 [[1, 13], :on_op, ":"],
 [[1, 14], :on_ident, "n"],
...

[korny]

Ripper is awesome, but I have no idea how to combine it with CodeRay. They do different things, so how to keep them in sync? Also, performance?

I'd much rather have an even better heuristic here, eg. colon after single quote must be an operator.

[nathany]

I haven't measured Ripper's performance, but it mostly written in C, so I'm guessing it's pretty good. The main issue is that it severely limits the supported platforms (MRI 1.9.x+ only), so it may be better as an external experiment for now.

👍 Better heuristic's in CodeRay would always be appreciated, esp. for Ruby.

[korny]

I haven't measured Ripper's performance, but it mostly written in C, so I'm guessing it's pretty good.

Ripper may be fast, but the syncing is definitely not.

[korny]

I think this would be a cool feature, but it should not delay 1.1, so I'll set it to "later".