CodeRay

h1. Scanner Requests

Scanners are the heart of CodeRay. They split input code into tokens and classify them.

Each language has its own scanner: You can see what languages are currently supported in the "repository":http://code.licenser.net/repositories/browse/coderay/trunk/lib/coderay/scanners.

h2. Why is the CodeRay language support list so short?

CodeRay developing is a slow process, because the total number of active developers is 1 and he insists on high software quality.

Special attention is paid to the scanners: every CodeRay scanner is being tested carefully against lots of example source code, and also randomized and junk code to make it safe. A CodeRay scanner is not officially released unless it highlights very, very well.

h2. I need a new Scanner - What can I do?

Here's what you can do to speed up the development of a new scanner:

# Request it! File a "new ticket":http://code.licenser.net/projects/coderay/issues/new unless it already "exists":http://code.licenser.net/projects/coderay/issues?query_id=3; add a +1 or something to existing tickets to show your interest.
# Upload or link to *example code* in the ticket discussion.
#* Typical code in large quantities is very helpful, also for benchmarking.
#* But we also need the most *weird and strange code* you can find to make the scanner.
# Provide links to useful *information about the language lexic*, such as:
#* a list of reserved words (Did you know that "void" is a JavaScript keyword?)
#* rules for string and number literals (Can a double quoted string contain a newline?)
#* rules for comments and other token types (Does XYZ have a special syntax for multiline comments?)
#* a description of any unusual syntactic features (There's this weird %w() thing in Ruby...)
#* If there are different versions / implementations / dialects of this language: How do they differ?
# Give examples for *good and bad highlighters / syntax definitions* for the language (usually from editors or other libraries)
# Find *more example code*!

Also, read the next paragraph.

h2. I want to write a Scanner myself

Wow, you're brave! Writing CodeRay scanners is not an easy task because:

* You need excellent knowledge about the language you want to scan. Every language has a dark side!
* You need good knowledge of (Ruby) regular expressions.
* There's no documentation to speak of.
** But this is a wiki ^hint hint^ ;o)

But it has been done before, so go and try it!

# You should still request the scanner (as described above) and announce that you are working on a patch yourself.
# Check out the "repository":http://code.licenser.net/wiki/coderay/Repository and try the test suite (@[lang=xyz] rake test:scanners@).
# Copy a scanner of your choice as a base. You would know what language comes closest.
# Create a test case directory in @test/scanners@.
# --- Advertisement --- (No, just kidding.)
# Write your scanner!
# Also, look into @lib/coderay/scanners/_map.rb@ and @lib/coderay/helpers/file_type.rb@.
# Make a patch (scanner, test cases and other changes) and upload it to the ticket.
# Follow the following discussion.
# Prepare to be added to the THX list.

Contact me (murphy) if you have any questions.

h2. How does a Scanner look?

For example, the JSON scanner:

<pre><code class="ruby">
module CodeRay
module Scanners

class JSON < Scanner

include Streamable

register_for :json

CONSTANTS = %w( true false null )
IDENT_KIND = WordList.new(:key).add(CONSTANTS, :reserved)

ESCAPE = / [bfnrt\\"\/] /x
UNICODE_ESCAPE = / u[a-fA-F0-9]{4} /x

def scan_tokens tokens, options

state = :initial
stack = []
string_delimiter = nil
key_expected = false

until eos?

kind = nil
match = nil

case state

when :initial
if match = scan(/ \s+ | \\\n /x)
tokens << [match, :space]
next
elsif match = scan(/ [:,\[{\]}] /x)
kind = :operator
case match
when '{' then stack << :object; key_expected = true
when '[' then stack << :array
when ':' then key_expected = false
when ',' then key_expected = true if stack.last == :object
when '}', ']' then stack.pop # no error recovery, but works for valid JSON
end
elsif match = scan(/ true | false | null /x)
kind = IDENT_KIND[match]
elsif match = scan(/-?(?:0|[1-9]\d*)/)
kind = :integer
if scan(/\.\d+(?:[eE][-+]?\d+)?|[eE][-+]?\d+/)
match << matched
kind = :float
end
elsif match = scan(/"/)
state = key_expected ? :key : :string
tokens << [:open, state]
kind = :delimiter
else
getch
kind = :error
end

when :string, :key
if scan(/[^\\"]+/)
kind = :content
elsif scan(/"/)
tokens << ['"', :delimiter]
tokens << [:close, state]
state = :initial
next
elsif scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
kind = :char
elsif scan(/\\./m)
kind = :content
elsif scan(/ \\ | $ /x)
tokens << [:close, :delimiter]
kind = :error
state = :initial
else
raise_inspect "else case \" reached; %p not handled." % peek(1), tokens
end

else
raise_inspect 'Unknown state', tokens

end

match ||= matched
if $DEBUG and not kind
raise_inspect 'Error token %p in line %d' %
[[match, kind], line], tokens
end
raise_inspect 'Empty token', tokens unless match

tokens << [match, kind]

end

if [:string, :key].include? state
tokens << [:close, state]
end

tokens
end

end

end
end
</code></pre>