CodeRay

Since the public activity is more and more moving to the CodeRay page on GitHub, it's time to start deleting this Redmine project. Today, I shut down the Forum (which has not been used since 2010) and the view to the obsolete SVN Repository.

• If you want to comment on CodeRay, please use GitHub, or mail me (murphy rubychan de).
• If you want to access the old repository, it's still there: http://svn.rubychan.de/coderay/

I plan to move the wiki pages and the tickets over to GitHub, too, when I find the time. Sorry about the confusion.

TL;DR¶

gem install coderay; it's faster and better.

Why 1.0?¶

CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.

While the command line and programmer interfaces are similar to previous versions, the internals have changed completely.

What's new?¶

• Support for two new languages: Clojure and HAML
• Improved command line support; just run coderay to get help.
• The rewritten Diff scanner highlights code and differences in single-line changes.
• Direct Streaming uses less memory and delivers even more speed.
• HTML output uses some CSS 3 features, like transparent background colors.
• The repository was moved to GitHub.
• A .gemspec was added.

There are also numerous bug fixes, tweaks, cleanups, additional RDoc documentation, and new tests.

I was using CodeRay 0.9. Will it break my code?¶

The Scheme and Nitro-XHTML scanners, the old themes, the coderay_stylesheet command, and some token kinds have been removed.

The most important change to be aware of is that Token classes (used as CSS classes) are readable names; this will break your 0.9 stylesheet. You need to change the token names accoring to coderay stylesheet, the new (and only) Alpha theme.

For everything else, the Changelog has a very detailed list of changes.

Future¶

Development will now focus on creating more scanners and encoders, as well as themes.

This will only be possible with help from the community. Join the project by forking rubychan/coderay!

TL;DR¶

gem install coderay if you work with JRuby's 1.9 mode.

A message from GitHub¶

I knew this was going to happen when I cloned the original CodeRay (Subversion) repository to GitHub: Somebody filed a ticket there. banister is using CodeRay in the awesome Pry shell, and found it doesn't work with JRuby's preliminary Ruby 1.9 support.

Further investigation revealed at least two bugs in JRuby's current support for Ruby 1.9 regexps. I've found good workarounds for them (#297), and hope CodeRay users once again help to debug JRuby :)

That said, CodeRay is still buggy on JRuby --1.9; the test suite throws a lot of encoding-related errors and even hangs up in one case. So, I can't recommend using it in production mode on this platform. (JRuby's default --1.8 mode works perfectly fine.)

Icing on the Cake¶

In addition to these fixes,

• the CSS scanner counts lines of code correctly (#296)
• the SQL scanner recognizes a few more keywords (#221)
• ColdFusion XML file types are recognized as such (#298)
• a warning about obsolete has_rdoc was removed

Upgrade from any older 0.9 version is recommended, easy, and painless.

Tested with:

• MRI
  • 1.8.6-p420
  • 1.8.7-p334
  • 1.9.1-p431
  • 1.9.2-p180
  • 1.9.3dev-trunk 31400
• JRuby
  • 1.6.1 (1.8 mode works, 1.9 mode is buggy)
• Rubinius
  • 1.2.4dev
  • latest Rubinius Hydra branch

I don't recommend using older Ruby versions. MacRuby's support for StringScanner is still buggy, but you can try.

For the record: I love rvm.

Distributions¶

• RubyGem: [sudo] gem install coderay
• Gemcutter
• Subversion access: http://svn.rubychan.de/coderay/tags/0.9.8

No world domination progress¶

I decided that I don't want to dominate the world with my syntax highlighting library. But it's still awesome that CodeRay downloads have passed the number 100,000 ;)

Future¶

CodeRay 1.0 RC1 will be released soon. I will migrate the repository to GitHub after releasing 1.0. After that, I will open up development so everybody can join (fork, push etc.) in.

Feedback welcome!

TL;DR¶

gem install coderay ASAP. Older versions are prone to attacks.

An Email from France¶

On Friday morning, I found this mail in my inbox:

Hello,

Sorry but I tried to parse some messy js yesturday on http://coderay.rubychan.de/ and since that, the website is down.
See http://www.redmine.org/issues/7265#note-4 for explanation.
I just found out your mail address, hope you'll receive it quickly !

Regards,
Etienne Massip

Well, it wasn't just a server hiccup this time; Etienne had really found an exploit for CodeRay's JavaScript scanner (without malicious intent, that is). Of course, the server being down (or rather, busy — on 99% CPU load on the Rails process) wasn't the biggest problem. This can happen to every application using CodeRay!

The CodeRay website was up again after a few hours, thanks to Etienne's report. But if you are using CodeRay on your website or project, hear the chimes of danger:

If you send user input through CodeRay 0.9.6 or older, your server may be in danger of DoS attacks!¶

A simple code piece like {"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n, when sent to the JavaScript scanner, can already take several seconds before the scanner call returns. This grows exponentially with the string length, so it's easy to start a scanner call which would take years to complete. Which means Good Bye to your application process.

Read a detailed explanation of the problem and the fix in the bug ticket.

Your turn: Update!¶

Upgrading from 0.9.6 and earlier 0.9 versions should be painless.

Tested with MRI 1.8.6-p399, 1.8.7-p330, 1.9.1-p378, 1.9.2-p136, JRuby 1.5.6, JRuby 1.6.0.dev, and Rubinius 1.2.0. I don't recommend using older Ruby versions. MacRuby's support for StringScanner is still buggy, but you can try.

Distributions¶

• RubyGem: [sudo] gem install coderay
• Gemcutter
• Subversion access: http://svn.rubychan.de/coderay/tags/0.9.7

World domination progress¶

CodeRay is typically ranking somewhere in the 300s of the most downloaded gems on RubyGems.org. Version 0.9.6 was downloaded over 11,000 times in the last 7 weeks. But still no reaction from @thedeadserious about listing it on The Ruby Toolbox.

Future¶

This was an unplanned release. I'm back in Germany, so work on 1.0 will continue in the months ahead.

Any feedback is welcome!

TL;DR¶

gem install coderay. You know you want it!

WoNáDo what?¶

Wolfgang Nádasi-Donner is a German Ruby/regexp enthusiast, and one of the most active members at the German Ruby board RubyForen. His articles and posts have been a great inspiration for me :) He contributed to CodeRay in many ways over the years, with the latest support being a bug report for the Ruby scanner concerning the new Hash syntax (#257). I like to thank him by naming this release of CodeRay after him!

Ruby scanner improved¶

So, this release has improved support for Ruby 1.9 syntax. And Jan Lelis, another member of the RubyForen, found a bug in the scanner framework (#260). It's all been corrected, see wiki for details.

Upgrading from 0.9.5 should be painless.

I tested with Ruby 1.8.6-p399, 1.8.7-p302, 1.9.1-p378, 1.9.2-p0, JRuby 1.5.5, and Rubinius 1.1.1. I don't recommend using older versions. MacRuby's support for StringScanner is still buggy, but you can try.

Distributions¶

• RubyGem: [sudo] gem install coderay
• Gemcutter
• Subversion access: http://svn.rubychan.de/coderay/tags/0.9.6

World domination progress¶

CodeRay has passed the 50,000 downloads mark :D

Future¶

This is the last planned release before 1.0. But since I'm currently on holiday in New Zealand, it will take more time, and might be released around spring 2011.

Any feedback is welcome!

You may have noticed that the CodeRay subversion repository (http://svn.rubychan.de/coderay/trunk/) was unavailable since 2010-10-24. I had some trouble with revisions missing randomly o_O

It is now fixed and back online. Thanks to Licenser for providing additional backups!

TL;DR¶

Run gem install coderay because 0.9.5 > 0.9.4.

Support for another Ruby engine!¶

I'm proud to announce that CodeRay now officially supports Rubinius, the crazy Ruby-in-Ruby-on-LLVM implementation. However, there's a little problem with the current release (1.1.0). The previous version (1.0.1) and the soon-to-be-released version (1.1.1) are supported, but not this one.

The CodeRay homepage now also has a list of supported Ruby engines.

Fixes¶

• diff code no longer produces invalid HTML.
• Ruby multibyte tokens are now highlighted correctly and consistently on all Ruby engines.
• Improved support of Ruby 1.9 encodings; please always give UTF-8 strings to CodeRay. Invalid chars do not crash the scanner, but will produce ? symbols in the output.
• Ruby 1.9 syntax: The new Hash syntax ({ a: b }) and the lambda shortcut (->) are now highlighted correctly.

Upgrading from 0.9.4 should not cause any problems. Read the full list of changes in the Wiki.

This release was tested with Ruby 1.8.6-p399, 1.8.7-p302, 1.9.1-p378, 1.9.2-p0, JRuby 1.5.2, and Rubinius 1.0.1 (all x86_64).

Distributions¶

• RubyGem: [sudo] gem install coderay
• Gemcutter
• Subversion access: http://svn.rubychan.de/coderay/tags/0.9.5
• Tarball release: Why do you need this again?

World domination progress¶

CodeRay 0.9.4 has been downloaded 7,686 times since August 31 (that's about 280 times per day).

The Read Ruby book project (by runpaint) is now using CodeRay instead of Pygments to simplify the compilation process.

Thanks¶

This release is codenamed Germany.rb, because it was an awesome usergroup meeting and I have got lots of ideas and support from there. I thank Florian Gilcher for hints on Rubinius and Ruby 1.9 encodings, and Jan Lelis for reminding me of the Ruby 1.9 lambda shortcut ;)

I've given a presentation on the history of CodeRay development at the Germany.rb (slides).

Future¶

1.0 is also in progress. I've tweaked the Ruby scanner again, and it's now capable of scanning the entire source code of Rails 1.2 (about 2.7 MB) in just under 1 second (using MRI 1.9.2 in my i7 iMac), which means it is 40% faster than the current stable release.

Support for new languages¶

I know, you don't care about speed, or tweaks, or obscure details, or little bugfixes. You want new languages. I really recommend you use Pygments then.

As for me, I'm travelling to New Zealand starting October 24, and won't come back to Berlin until 2011. I don't plan to write a lot of code while wandering on the tracks of the Fellowship, which means CodeRay 1.0 will not be released this year. That also means you will have to wait for new languages (which are planned for 1.1).

If you don't want to wait, go ahead and do it yourself! Write a scanner, put it on GitHub, and release it as a gem. The community will thank you, and if your scanner is awesome, I will be happy to include it in the distribution.

Any feedback is welcome!

TL;DR¶

CodeRay bugfix release, run gem install coderay.

Another release¶

CodeRay 0.9.3 has been downloaded 13,542 times since April 18 (that's about 100 times per day).

This release is codenamed Ramadan: I'm not a muslim, but let's celebrate together :D

Fixes¶

• The coderay executable has been rewritten. Run it without any parameters to get a command overview.
• Small fixes to the Diff, HTML, and RHTML (ERB) scanners.

Upgrading from 0.9.3 should be smooth. Read the full list of changes in the Wiki.

This release was tested with Ruby 1.8.6p399, 1.8.7p302, 1.9.2p0, and JRuby 1.5.2 (all 64 bit). I used the awesome rvm this time; thank you so much for this gem!

Distributions¶

• RubyGem: [sudo] gem install coderay
• Gemcutter
• Subversion access: http://svn.rubychan.de/coderay/tags/0.9.4
• Tarball release: RubyForge

Future¶

1.0 is still in progress.

• See Benchmarks of the next version.
• Inline Diff highlighting, combined with Intelligent Diff highlighting this will be a quite unique feature. It's really complicated and still not finished.
• Lots of other fixes and improvements are on the way.

By the way, CodeRay has no gem dependencies, and I plan to keep it that way :D I don't like it that even trivial gems depend on a wealth of incompatible junk.

Any feedback is welcome!

CodeRay 1.0.0.598.pre was just pushed to RubyGems. Don't use it for production mode - it might throw up :)

I won't go into detail about the new functions here. You can read the logs or the List of Changes.

The new version is neither feature-complete, nor even completely defined yet. But to highlight some of the already implemented changes, here's an overview of what's coming in the 1.0 release:

• New Direct Streaming tokens interface; actually, I rewrote most of CodeRay. It's slimmer and up to 50% faster now :)
• diff highlighting 2.0 with code highlighting and inline changes eye-catchers.
• embracing HTML5, because I love it. Support for Internet Explorer has been officially dropped.
• Dozens of fixes, improvements, and cleanups.

Planned features currently in development include:

• revamped command line interface (the coderay executable)
• a Clojure scanner from Licenser
• HTML scanner with integrated JavaScript and CSS highlighting

1.0 will not be API-compatible with earlier versions. Especially, if you use a custom Scanner or Encoder, it will not work. They must be updated to the new API.

Have fun, and comment if you like. As usual, all feedback can be mailed to me (murphy rubychan de).

code.licenser.net and redmine.rubychan.de are currently struck by a series of downtimes. You will see a "Service temporarily unavailable" error message.

We are sorry for the inconveniences and are investigating the problem. If you experience such an outage, and it is not fixed after some minutes, please send a mail to <murphy rubychan de> with a short notice so that we can react.

Well, Shit Happens.