CodeRay

h1=. CodeRay Version History

p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._

{{toc}}

h2. Changes in 1.0

CodeRay 1.0 is a major rewrite of the library, and incompatible to earlier versions.

The command line and programmer interfaces are similar to 0.9, but the internals have completely changed.

h3. General changes

* *NEW*: The new Diff scanner colorizes code inside of the diff, and highlights inline changes.

* *NEW*: Extended support and usage of HTML5 and CSS 3 features.

* *NEW*: Direct Streaming

* *NEW* scanners: Clojure and HAML

* *CHANGED*: Token classes (used as CSS classes) are readable names; breaks you stylesheet!

* *IMPROVED* documentation

* *IMPROVED* speed: faster startup (using @autoload@), scanning, and encoding

* *IMPROVED* Ruby 1.9 encodings support

* *IMPROVED* Tests: There are more of them now!

h3. Direct Streaming

CodeRay 1.0 introduces _Direct Streaming_ as a faster and simpler alternative to Tokens. It means that all Scanners,

Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with

this version.

The main benefits of this change are:

* more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9)

* the ability to stream output into a pipe on the command line

* a simpler API

* less code

Changes related to the new tokens handling include:

* *CHANGED*: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the

  Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a

  special case; Tokens just encodes the given tokens into an Array for later use.

* *CHANGED*: The token actions (@text_token@, @begin_group@ etc.) are now public methods of @Encoder@ and @Tokens@.

* *REWRITE* of all Scanners, Encoders, Filters, and Tokens.

* *RENAMED* @:open@ and @:close@ actions to @:begin_group@ and @:end_group@.

* *RENAMED* @open_token@ and @close_token@ methods to @begin_group@ and @end_group@.

* *NEW* method @#tokens@ allows to add several tokens to the stream. @Tokens@ and @Encoders::Encoder@ define this

  method.

* *CHANGED* The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible

  with earlier versions.

* *REMOVED* @TokenStream@ and the @Streamable@ API and all related features like @NotStreamableError@ are now obsolete

  and have been removed.

h3. Command Line

The @coderay@ executable was rewritten and has a few new features:

* *NEW* Ability to stream into a pipe; try @coderay file | more -r@

* *NEW* help

* *IMPROVED*: more consistent parameter handling

* *REMOVED* @coderay_stylesheet@ executable; use @coderay stylesheet [name]@.

h3. @Tokens@

* *NEW* methods @count@, @begin_group@, @end_group@, @begin_line@, and @end_line@.

* *REMOVED* methods @#stream?@, @#each_text_token@.

* *REMOVED* @#text@ and @#text_size@ methods. Use the @Text@ encoder instead.

* *REMOVED* special implementation of @#each@ taking a filter parameter. Use @TokenKindFilter@ instead.

h3. *RENAMED*: @TokenKinds@

Renamed from @Tokens::ClassOfKind@ (was also @Tokens::AbbreviationForKind@ for a while).

The term "token class" is no longer used in CodeRay. Instead, tokens have _kinds_.

See "#122":http://redmine.rubychan.de/issues/122.

* *CHANGED* all token CSS classes to readable names.

* *ADDED* token kinds @:filename@, @:namespace@, and @:eyecatcher@.

* *RENAMED* @:pre_constant@ and @:pre_type@ to @:predefined_constant@ and @predefined_type@.

* *RENAMED* @:oct@ and @:bin@ to @:octal@ and @binary@.

* *REMOVED* token kinds @:attribute_name_fat@, @:attribute_value_fat@, @:operator_fat@, @interpreted@,

  @:tag_fat@, @tag_special@, @:xml_text@, @:nesting_delimiter@, @:open@, and @:close@.

* *CHANGED*: Don't raise error for unknown token kinds unless in @$CODERAY_DEBUG@ mode.

* *CHANGED* the value for a token kind that is not highlighted from

  @:NO_HIGHLIGHT@ to @false@.

h3. @Duo@

* *NEW* method @call@ for allowing code like @CodeRay::Duo[:python => :yaml].(code)@ in Ruby 1.9.

h3. @Encoders::CommentFilter@

* *NEW* alias @:remove_comments@

h3. @Encoders::Filter@

* *NEW* option @tokens@.

* *CHANGED*: Now it simply delegates to the output.

* *REMOVED* @include_text_token?@ and @include_block_token?@ methods.

h3. @Encoders::HTML@

The HTML encoder was cleaned up and simplified.

* *NEW*: HTML5 and CSS 3 compatible.

  See "#215":http://redmine.rubychan.de/issues/215.

* *ADDED* support for @:line_number_anchors@.

  See "#208":http://redmine.rubychan.de/issues/208.

* *CHANGED* the default style to @:alpha@.

* *CHANGED*: Use double click to toggle line numbers in table mode (as single

  click jumps to an anchor.)

* *REMOVED* support for @:line_numbers => :list@.

* *FIXED* splitting of lines for @:line_numbers => :inline@, so that the line

  numbers don't get colored, too.

* *RENAMED* @Output#numerize@ to @#number@, which is an actual English word.

h3. @Encoders::LinesOfCode@

* *CHANGED*: @compile@ and @finish@ methods are now protected.

h3. *Renamed*: @Encoders::Terminal@ (was @Encoders::Term@)

* *RENAMED* from @Encoders::Term@, added @:term@ alias.

* *CLEANUP*: Use @#setup@'s @super@, don't use @:procedure@ token class.

* *CHANGED*: @#token@'s second parameter is no longer optional.

* *REMOVED* colors for obsolete token kinds.

* *FIXED* handling of line tokens.

h3. @Encoders::Text@

* *FIXED* default behavior of stripping the trailing newline.

h3. *RENAMED*: @Encoders::TokenKindFilter@ (was @Encoders::TokenClassFilter@)

* *NEW*: Handles token groups.

  See "#223":http://redmine.rubychan.de/issues/223.

* *RENAMED* @include_block_token?@ to @include_group?@.

h3. @Encoders::Statistic@

* *CHANGED*: Tokens actions are counted separately.

h3. @Scanners::Scanner@

* *NEW* methods @#file_extension@ and @#encoding@.

* *NEW*: The @#tokenize@ method also takes an Array of Strings as source. The

  code is highlighted as one and split into parts of the input lengths

  after that using @Tokens#split_into_parts@.

* *NEW* method @#binary_string@

* *REMOVED* helper method @String#to_unix@.

* *REMOVED* method @#streamable?@.

* *REMOVED* @#marshal_load@ and @#marshal_dump@.

* *RENAMED* class method @normify@ to @normalize@; it also deals with encoding now.

* *CHANGED*: @#column@ starts counting with 1 instead of 0

h3. *NEW*: @Scanners::Clojure@

Thanks to Licenser, CodeRay now supports the Clojure language.

h3. @Scanners::CSS@

* *NEW*: Rudimentary support for the @attr@, @counter@, and @counters@ functions.

  See "#224":http://redmine.rubychan.de/issues/224.

* *NEW*: Rudimentary support for CSS 3 colors.

* *CHANGED*: Attribute selectors are highlighted as @:attribute_name@ instead of @:string@.

* *CHANGED*: Comments are scanned as one token instead of three.

h3. @Scanners::Debug@

* *NEW*: Support for line tokens (@begin_line@ and @end_line@ represented by @[@ and @]@.)

* *FIXED*: Don't send @:error@ and @nil@ tokens for buggy input any more.

* *FIXED*: Closes unclosed tokens at the end of @scan_tokens@.

* *IMPROVED*: Highlight unknown tokens as @:error@.

* *CHANGED*: Raises an error when trying to end an invalid token group.

h3. @Scanners::Delphi@

* *FIXED*: Closes open string groups.

h3. @Scanners::Diff@

* *NEW*: Highlighting of code based on file names.

  See ticket "#52":http://redmine.rubychan.de/issues/52.

  Use the @:highlight_code@ option to turn this feature off. It's enabled

  by default.

  This is a very original feature. It enables multi-language highlighting for

  diff files, which is especially helpful for CodeRay development itself. The

  updated version of the scanner test suite generated .debug.diff.html files

  using this.

  Note: This is still experimental. Tokens spanning more than one line

  may get highlighted incorrectly. CodeRay tries to keep scanner states

  between the lines and changes, but the quality of the results depend on

  the scanner.

* *NEW*: Inline change highlighting, as suggested by Eric Thomas.

  See ticket "#227":http://redmine.rubychan.de/issues/227 for details.

  Use the @:inline_diff@ option to turn this feature off. It's enabled by

  default.

  For single-line changes (that is, a single deleted line followed by a single

  inserted line), this feature surrounds the changed parts with an

  @:eyecatcher@ group which appears in a more saturated background color.

  The implementation is quite complex, and highly experimental. The problem

  with multi-layer tokenizing is that the tokens have to be split into parts.

  If the inline change starts, say, in the middle of a string, then additional

  @:end_group@ and @:begin_group@ tokens must be inserted to keep the group

  nesting intact. The extended @Scanner#tokenize@ method and the new

  @Tokens#split_into_parts@ method take care of this.

* *NEW*: Highlight the file name in the change headers as @:filename@.

* *CHANGED*: Highlight unknown lines as @:comment@ instead of @:head@.

* *IMPROVED*: Background colors for Diff output have been optimized.

h3. *RENAMED*: @Scanners::ERB@ (was @Scanners::RHTML@)

h3. *NEW*: @Scanners::HAML@

It uses the new :state options of the HTML and Ruby scanners.

Some rare cases are not considered (like @#{...}@ snippets inside of :javascript blocks),

but it highlights pretty well.

h3. @Scanners::HTML@

* *FIXED*: Closes open string groups.

h3. @Scanners::JavaScript@

* *IMPROVED*: Added @NaN@ and @Infinity@ to list of predefined constants.

* *IMPROVED* recognition of RegExp literals with leading spaces.

h3. @Scanners::Java@

* *NEW*: Package names are highlighted as @:namespace@.

  See "#210":http://redmine.rubychan.de/issues/210.

h3. *REMOVED*: @Scanners::NitroXHTML@

Nitro is "dead":http://www.nitrohq.com/.

h3. *RENAMED*: @Scanners::Text@ (was @Scanners::Plaintext@)

* *IMPROVED*: Just returns the string without scanning (faster).

 This is much faster than scanning until @/\z/@ in Ruby 1.8.

h3. @Scanners::Python@

* *CHANGED*: Docstrings are highlighted as @:comment@.

  See "#190":http://redmine.rubychan.de/issues/190.

h3. *NEW*: @Scanners::Raydebug@

Copied from @Scanners::Debug@, highlights the token dump instead of importing it. It also reacts to the @.raydebug@ file

name suffix now.

h3. @Scanners::Ruby@

* *ADDED* more predefined keywords (see http://murfy.de/ruby-constants).

* *IMPROVED* support for singleton method definitions.

  See "#147":http://redmine.rubychan.de/issues/147.

* *FIXED*: Don't highlight methods with a capital letter as constants

  (eg. @GL.PushMatrix@).

* *NEW*: Highlight buggy floats (like .5) as @:error@.

* *CLEANUP* of documentation, names of constants and variables, state handling.

  Moved @StringState@ class from @patterns.rb@ into a separate file.

* *NEW*: Complicated rule for recognition of @foo=@ style method names.

* *NEW*: Handles @:keep_state@ option (a bit; experimental).

  Actually, Ruby checks if there is @[~>=]@, but not @=>@ following the name.

* *REMOVED* @EncodingError@

h3. *REMOVED* @Scanners::Scheme@

* It is too buggy, and nobody was using it. To be added again when it's fixed.

  See "#59":http://redmine.rubychan.de/issues/59.

h3. @Scanners::SQL@

* *IMPROVED*: Extended list of keywords and functions (thanks to

  Joshua Galvez, Etienne Massip, and others).

  See "#221":http://redmine.rubychan.de/issues/221.

* *FIXED*: Closes open string groups.

* *FIXED*: Words after @.@ are always recognized as @:ident@.

h3. @Scanners::YAML@

* *FIXED*: Allow spaces before colon in mappings.

  See "#231":http://redmine.rubychan.de/issues/231.

h3. *NEW*: @Styles::Alpha@

A style that uses transparent HSLA colors as defined in CSS 3. See "#199":http://redmine.rubychan.de/issues/199.

It also uses the CSS 3 property @user-select: none@ to keep the user from selecting the line numbers. This is especially

nice for @:inline@ line numbers. See "#226":http://redmine.rubychan.de/issues/226.

h3. @WordList@

Stripped down to 19 LOC.

* *RENAMED* @CaseIgnoringWordList@ to @WordList::CaseIgnoring@.

* *REMOVED* caching option because it creates memory leaks.

* *REMOVED* block option.

h3. @FileType@

* *NEW*: Recognizes @.gemspec@, @.rjs@, @.rpdf@ extensions, @Gemfile@, and @Capfile@ as Ruby.

  Thanks to the authors of the TextMate Ruby bundle!

* *REMOVED* @FileType#shebang@ is a protected method now.

h3. @Plugin@

* *IMPROVED*: @register_for@ sets the @plugin_id@; it can now be a @Symbol@.

* *ADDED* @PluginHost#const_missing@ method: Plugins are loaded automatically.

  Using @Scanners::JavaScript@ in your code loads @scanners/java_script.rb@.

* *ADDED* @#all_plugins@ method to simplify getting 

  information about all available plugins (suggested by bnhymn).

* *CHANGED* the default plugin key from @nil@ to @:default@.

h3. @GZip@

* *MOVED* into @CodeRay@ namespace.

* *MOVED* file from @gzip_simple.rb@ to @gzip.rb@.

* *REMOVED* @String@ extensions.

h3. More API changes

* *FIXED* @Encoders::HTML#token@'s second parameter is no longer optional.

* *CHANGED* @Encoders::HTML::Output@'s API.

* *REMOVED* lots of unused methods.

The helper classes were cleaned up; see above for details.

* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.

* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.