CodeRay

h1. Scanner Requests

Scanners are the heart of CodeRay. They split input code into tokens and classify them.

Each language has its own scanner: You can see what languages are currently supported in the "repository":https://github.com/rubychan/coderay.

h2. Why is the CodeRay language support list so short?

CodeRay developing is a slow process, because the total number of active developers is 1 and he insists on high software quality.

Special attention is paid to the scanners: Every "CodeRay scanner":https://github.com/rubychan/coderay/tree/master/lib/coderay/scanners is being tested carefully against lots of example source code, and also randomized and junk code to make it safe. A CodeRay scanner is not officially released unless it highlights very, very well.

h2. I need a new Scanner - What can I do?

Here's what you can do to speed up the development of a new scanner:

# Request it! File a "new ticket":http://odd-eyed-code.org/projects/coderay/issues/new unless it already "exists":http://odd-eyed-code.org/projects/coderay/issues?query_id=3 or add a +1 or something to existing tickets to show your interest.

# Upload or link to *example code* in the ticket discussion.

#* Typical code in large quantities is very helpful, also for benchmarking.

#* But we also need the most *weird and strange code* you can find to make the scanner.

# Provide links to useful *information about the language lexic*, such as:

#* a list of reserved words (Did you know that "void" is a JavaScript keyword?)

#* rules for string and number literals (Can a double quoted string contain a newline?)

#* rules for comments and other token types (Does Language have a special syntax for multiline comments?)

#* a description of any unusual syntactic features (There's this weird %w() thing in Ruby...)

#* If there are different versions / implementations / dialects of this language: How do they differ?

# Give examples for *good and bad highlighters / syntax definitions* for the language (usually from editors or other libraries),

# Find *more example code*!

Also, read the next section.

h2. I want to write a Scanner myself

Wow, you're brave! Writing CodeRay scanners is not an easy task because:

* You need excellent knowledge about the language you want to scan. Every language has a dark side!

* You need good knowledge of (Ruby) regular expressions.

* There's no documentation to speak of.

** But this is a wiki ^hint hint^ ;o)

But it has been done before, so go and try it!

# You should still request the scanner (as described above) and announce that you are working on a patch yourself.

# Check out the [[Repository]] and try the [[Test Suite]].

# Copy a scanner of your choice as a base. You would know what language comes closest.

# Make sure you have run @rake test:scanners@ to get the scanner test suite.

# Create a test case directory in @test/scanners/<lang>@ and add example files for your language.

# Run your tests cases with @rake test:scanner:<lang>@ and write your scanner!

# Also, look into @lib/coderay/scanners/_map.rb@ and @lib/coderay/helpers/file_type.rb@.

# Make a patch (scanner, test cases and other changes) and upload it to the ticket.

# Follow the following discussion.

# Prepare to be added to the THX list.

Contact me (murphy rubychan de) if you have any questions.

h2. How does a Scanner look?

For example, the JSON scanner:

<pre><code class="ruby">

# Namespace; use this form instead of CodeRay::Scanners to avoid messages like

# "uninitialized constant CodeRay" when testing it.

module CodeRay

module Scanners

  # Always inherit from CodeRay::Scanners::Scanner.

  # 

  # Scanner inherits directly from StringScanner, the Ruby class for fast

  # string scanning. Read the documentation to understand what's going on here:

  # 

  #   http://www.ruby-doc.org/stdlib/libdoc/strscan/rdoc/index.html

  class JSON < Scanner

    # Deprecation notice: The Streamable module is gone.

    # Scanners are plugins and must be registered like this:

    register_for :json

    # You can provide a file extension associated with this language.

    file_extension 'json'

    # List all token kinds that are not considered to be running code

    # in this language. For a typical language, this would just be

    # :comment, but for a data or markup language like JSON, no tokens

    # should count as Line of Code.

    KINDS_NOT_LOC = [

      :float, :char, :content, :delimiter,

      :error, :integer, :operator, :value,

    ]  # :nodoc:

    # See the WordList documentation.

    CONSTANTS = %w( true false null )

    IDENT_KIND = WordList.new(:key).add(CONSTANTS, :value)

    ESCAPE = / [bfnrt\\"\/] /x

    UNICODE_ESCAPE = / u[a-fA-F0-9]{4} /x

    # This is the only method you need to define. It scans code.

    # 

    # encoder is an object which encodes tokens. It provides the following API:

    # * encoder.text_token(text, kind) for tokens

    # * encoder.begin_group(kind) and encoder.end_group(kind) for token groups

    # * encoder.begin_line(kind) and encoder.end_line(kind) for line tokens

    # 

    # options is a hash. Standard options are:

    # * keep_state: Try to save the current scanner state and restore it in the

    #   next call of scan_tokens.

    # 

    # scan_tokens must return the encoder variable it was given.

    # 

    # You are completely free to use any style you want, just make sure encoder

    # gets what it needs. But typically, a Scanner follows the following scheme:

    def scan_tokens encoder, options

      # The scanner is always in a certain state, which is :initial by default.

      # We use local variables and symbols to maximize speed.

      state = :initial

      # Sometimes, you need a stack. Ruby arrays are perfect for this.

      stack = []

      # Define more flags and variables as you need them.

      key_expected = false

      # The main loop; eos? is true when the end of the code is reached.

      until eos?

        # Deprecation notice: The use of local variables kind and match no longer

        # recommended.

        # Depending on the state, we want to do different things.

        case state

        # Normally, we use this case.

        when :initial

          # I like the / ... /x style regexps because white space makes them more

          # readable. x means white space is ignored.

          if match = scan(/ \s+ /x)

            # White space and masked line ends are :space.

            # Make sure you never send an empty token! /\s*/ for example would be

            # very bad (actually creating an infinite loop).

            encoder.text_token match, :space

          elsif match = scan(/ [:,\[{\]}] /x)

            # Operators of JSON. stack is used to determine where we are. stack and

            # key_expected are set depending on which operator was found.

            # key_expected is used to decide whether a "quoted" thing should be

            # classified as key or string.

            encoder.text_token match, :operator

            case match

            when '{' then stack << :object; key_expected = true

            when '[' then stack << :array

            when ':' then key_expected = false

            when ',' then key_expected = true if stack.last == :object

            when '}', ']' then stack.pop  # no error recovery, but works for valid JSON

            end

          elsif match = scan(/ true | false | null /x)

            # These are the only idents that are allowed in JSON. Normally, IDENT_KIND

            # would be used to tell keywords and idents apart.

            encoder.text_token match, IDENT_KIND[match]

          elsif match = scan(/ -? (?: 0 | [1-9]\d* ) /x)

            # Pay attention to the details: JSON doesn't allow numbers like 00.

            if scan(/ \.\d+ (?:[eE][-+]?\d+)? | [eE][-+]? \d+ /x)

              match << matched

              encoder.text_token match, :float

            else

              encoder.text_token match, :integer

            end

          elsif match = scan(/"/)

            # A "quoted" token was found, and we know whether it is a key or a string.

            state = key_expected ? :key : :string

            # This opens a token group and encodes the delimiter token.

            encoder.begin_group state

            encoder.text_token match, :delimiter

          else

            # Don't forget to add this case: If we reach invalid code, we try to discard

            # chars one by one and mark them as :error.

            encoder.text_token getch, :error

          end

        # String scanning is a bit more complicated, so we use another state for it.

        # The scanner stays in :string state until the string ends or an error occurs.

        # 

        # JSON uses the same notation for strings and keys. We want keys to be in a

        # different color, but the lexical rules are the same. This is why we use this

        # case also for the :key state.

        when :string, :key

          # Another if-elsif-else-switch, for strings this time.

          if match = scan(/[^\\"]+/)

            # Everything that is not \ or " is just string content.

            encoder.text_token match, :content

          elsif match = scan(/"/)

            # A " is found, which means this string or key is ending here.

            # A special token class, :delimiter, is used for tokens like this one.

            encoder.text_token match, :delimiter

            # Always close your token groups using the right token kind!

            encoder.end_group state

            # We're going back to normal scanning here.

            state = :initial

            # Deprecation notice: Don't use "next" any more.

          elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)

            # A valid special character should be classified as :char.

            encoder.text_token match, :char

          elsif match = scan(/\\./m)

            # Anything else that is escaped (including \n, we use the m modifier) is

            # just content.

            encoder.text_token match, :content

          elsif match = scan(/ \\ | $ /x)

            # A string that suddenly ends in the middle, or reaches the end of the

            # line. This is an error; we go back to :initial now.

            encoder.end_group state

            encoder.text_token match, :error

            state = :initial

          else

            # Nice for debugging. Should never happen.

            raise_inspect "else case \" reached; %p not handled." % [peek(1)], encoder

          end

        else

          # Nice for debugging. Should never happen.

          raise_inspect 'Unknown state: %p' % [state], encoder

        end

        # Deprecation notice: The block using the match local variable block is gone.

      end

      # If we still have a string or key token group open, close it.

      if [:string, :key].include? state

        encoder.end_group state

      end

      # Return the encoder.

      encoder

    end

  end

end

end

</code></pre>

Highlighted with CodeRay, yeah :D