Scanner Requests¶

Scanners are the heart of CodeRay. They split input code into tokens and classify them.

Each language has its own scanner: You can see what languages are currently supported in the repository.

Why is the CodeRay language support list so short?¶

CodeRay developing is a slow process, because the total number of active developers is 1 and he insists on high software quality.

Special attention is paid to the scanners: every CodeRay scanner is being tested carefully against lots of example source code, and also randomized and junk code to make it safe. A CodeRay scanner is not officially released unless it highlights very, very well.

I need a new Scanner - What can I do?¶

Here's what you can do to speed up the development of a new scanner:

1. Request it! File a new ticket unless it already exists add a +1 or something to existing tickets to show your interest.
2. Upload or link to example code in the ticket discussion.
   • Typical code in large quantities is very helpful, also for benchmarking.
   • But we also need the most weird and strange code you can find to make the scanner.
3. Provide links to useful information about the language lexic, such as:
   • a list of reserved words (Did you know that "void" is a JavaScript keyword?)
   • rules for string and number literals (Can a double quoted string contain a newline?)
   • rules for comments and other token types (Does XYZ have a special syntax for multiline comments?)
   • a description of any unusual syntactic features (There's this weird %w() thing in Ruby...)
   • If there are different versions / implementations / dialects of this language: How do they differ?
4. Give examples for good and bad highlighters / syntax definitions for the language (usually from editors or other libraries)
5. Find more example code!

Also, read the next paragraph.

I want to write a Scanner myself¶

Wow, you're brave! Writing CodeRay scanners is not an easy task because:

• You need excellent knowledge about the language you want to scan. Every language has a dark side!
• You need good knowledge of (Ruby) regular expressions.
• There's no documentation to speak of.
  • But this is a wiki ^{hint hint} ;o)

But it has been done before, so go and try it!

1. You should still request the scanner (as described above) and announce that you are working on a patch yourself.
2. Check out the repository and try the test suite ([lang=xyz] rake test:scanners).
3. Copy a scanner of your choice as a base. You would know what language comes closest.
4. Create a test case directory in test/scanners.
5. --- Advertisement --- (No, just kidding.)
6. Write your scanner!
7. Also, look into lib/coderay/scanners/_map.rb and lib/coderay/helpers/file_type.rb.
8. Make a patch (scanner, test cases and other changes) and upload it to the ticket.
9. Follow the following discussion.
10. Prepare to be added to the THX list.

Contact me (murphy) if you have any questions.