CodeRay

h1=. CodeRay Version History

p=. _This files lists all changes in the CodeRay library since the 0.8.4 release._

{{toc}}

h2. Changes in 1.0

h3. Direct Streaming

CodeRay 1.0 introduces _Direct Streaming_ as a faster and simpler alternative to Tokens. It means that all Scanners, Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with this version.

The benefit of this change is more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9), a simpler API, and less code.

Changes related to the new tokens handling include:

* *CHANGED*: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a special case; Tokens just encodes the given tokens into an Array for later use.

* *CHANGED*: The token actions (@text_token@, @begin_group@ etc.) are now public methods of @Encoder@ and @Tokens@.

* *REWRITE* of all Scanners, Encoders, Filters, and Tokens.

* *RENAMED* @:open@ and @:close@ actions to @:begin_group@ and @:end_group@.

* *RENAMED* @open_token@ and @close_token@ methods to @begin_group@ and @end_group@.

* *NEW* method @#tokens@ allows to add several tokens to the stream. @Tokens@ and @Encoders::Encoder@ define this method.

* *CHANGED* The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible with earlier versions.

* *REMOVED* @TokenStream@ and the @Streamable@ API and all related features like @NotStreamableError@ are now obsolete and have been removed.

h3. General changes

* *IMPROVED* documentation in general; additions, corrections and cleanups

* *FIXED* some image links in the documentation

* *IMPROVED* Ruby 1.9 support (_._ not in @$LOAD_PATH@)

h3. @Tokens@

* *REMOVED* methods @#stream?@, @#each_text_token@.

* *REMOVED* @#text@ and @#text_size@ methods. Use the @Text@ encoder instead.

* *REMOVED* special implementation of @#each@ taking a filter parameter. Use @TokenKindFilter@ instead.

* *NEW* methods @encode_with@, @count@, @begin_group@, @end_group@, @begin_line@, and @end_line@.

h3. *RENAMED*: @Tokens::AbbreviationForKind@

Renamed from @ClassOfKind@; the term "token class" is no longer used in CodeRay. Instead, tokens have _kinds_.

See "#122":http://redmine.rubychan.de/issues/122.

* *REMOVED* token kinds @:attribute_name_fat@, @:attribute_value_fat@, @:operator_fat@, @:tag_fat@, @:xml_text@, @:open@, and @:close@.

* *ADDED* token kinds @:filename@, @:namespace@, and @:eyecatcher@.

* *CHANGED*: Don't raise error for unknown token kinds unless in @$CODERAY_DEBUG@ mode.

h3. @Encoders::CommentFilter@

* *NEW* alias @:remove_comments@

h3. @Encoders::Filter@

* *NEW* option @tokens@.

* *CHANGED*: Now it simply delegates to the output.

* *REMOVED* @include_text_token?@ and @include_block_token?@ methods.

h3. @Encoders::HTML@

The HTML encoder was cleaned up and simplified.

* *CHANGED* the default style to @:alpha@.

* *NEW*: HTML 5 and CSS 3 compatible, IE incompatible.

  See "#215":http://redmine.rubychan.de/issues/215.

* *ADDED* support for @:line_number_anchors@.

  See "#208":http://redmine.rubychan.de/issues/208.

* *CHANGED*: Use double click to toggle line numbers in table mode (as single 

  click jumps to an anchor.)

* *REMOVED* support for @:line_numbers => :list@.

* *FIXED* splitting of lines for @:line_numbers => :inline@, so that the line

  numbers don't get colored, too.

* *RENAMED* @Output#numerize@ to @#number@, which is an actual English word.

h3. @Encoders::LinesOfCode@

* *CHANGED*: @compile@ and @finish@ methods are now protected.

h3. @Encoders::Terminal@

* *RENAMED* from @Encoders::Term@, added @:term@ alias.

* *CLEANUP*: Use @#setup@'s @super@, don't use @:procedure@ token class.

* *CHANGED*: @#token@'s second parameter is no longer optional.

* *REMOVED* colors for obsolete token kinds.

* *FIXED* handling of line tokens.

h3. @Encoders::Text@

* *FIXED* default behavior of stripping the trailing newline.

h3. *RENAMED*: @Encoders::TokenKindFilter@

Renamed from @TokenClassFilter@.

* *NEW*: Handles token groups.

  See "#223":http://redmine.rubychan.de/issues/223.

* *RENAMED* @include_block_token?@ to @include_group?@.

h3. @Encoders::Statistic@

* *CHANGED*: Tokens actions are counted separately.

h3. @Encoders::YAML@

* *NEW* method @call@ for allowing code like @CodeRay::Duo[:python => :yaml].(code)@ in Ruby 1.9.

h3. @Scanners::Scanner@

* *REMOVED* helper method @String#to_unix@.

* *REMOVED* method @streamable?@.

* *NEW*: The @#tokenize@ method also takes an Array of Strings as source. The

  code is highlighted as one and split into parts of the input lengths

  after that using @Tokens#split_into_parts@.

h3. @Scanners::CSS@

* *NEW*: Rudimentary support for the @attr@, @counter@, and @counters@ functions.

  See "#224":http://redmine.rubychan.de/issues/224.

* *NEW*: Rudimentary support for CSS 3 colors.

* *CHANGED*: Attribute selectors are highlighted as @:attribute_name@ instead of @:string@.

h3. @Scanners::Debug@

* *NEW*: Support for line tokens (@begin_line@ and @end_line@ represented by @[@ and @]@.)

* *FIXED*: Don't send @:error@ and @nil@ tokens for buggy input any more.

* *FIXED*: Closes unclosed tokens at the end of @scan_tokens@.

* *IMPROVED*: Highlight unknown tokens as @:error@.

* *CHANGED*: Raises an error when trying to end an invalid token group.

h3. @Scanners::Delphi@

* *FIXED*: Closes open string groups.

h3. @Scanners::Diff@

* *NEW*: Highlighting of code based on file names.

  See ticket "#52":http://redmine.rubychan.de/issues/52.

  Use the @:highlight_code@ option to turn this feature off. It's enabled

  by default.

  This is a very original feature. It enables multi-language highlighting for

  diff files, which is especially helpful for CodeRay development itself. The

  updated version of the scanner test suite generated .debug.diff.html files

  using this.

  Note: This is still experimental. Tokens spanning more than one line

  may get highlighted incorrectly. CodeRay tries to keep scanner states

  between the lines and changes, but the quality of the results depend on

  the scanner.

* *NEW*: Inline change highlighting, as suggested by Eric Thomas.

  See ticket "#227":http://redmine.rubychan.de/issues/227 for details.

  Use the @:inline_diff@ option to turn this feature off. It's enabled by

  default.

  For single-line changes (that is, a single deleted line followed by a single

  inserted line), this feature surrounds the changed parts with an

  @:eyecatcher@ group which appears in a more saturated background color.

  The implementation is quite complex, and highly experimental. The problem

  with multi-layer tokenizing is that the tokens have to be split into parts.

  If the inline change starts, say, in the middle of a string, then additional

  @:end_group@ and @:begin_group@ tokens must be inserted to keep the group

  nesting intact. The extended @Scanner#tokenize@ method and the new

  @Tokens#split_into_parts@ method take care of this.

* *NEW*: Highlight the file name in the change headers as @:filename@.

* *CHANGED*: Highlight unknown lines as @:comment@ instead of @:head@.

h3. @Scanners::HTML@

* *FIXED*: Closes open string groups.

h3. @Scanners::JavaScript@

* *IMPROVED*: Added @NaN@ and @Infinity@ to list of predefined constants.

h3. @Scanners::Java@

* *NEW*: Package names are highlighted as @:namespace@.

  See "#210":http://redmine.rubychan.de/issues/210.

h3. @Scanners::Plaintext@

* *IMPROVED*: Just returns the string without scanning (faster).

 This is much faster than scanning until @/\z/@ in Ruby 1.8.

h3. @Scanners::Python@

* *CHANGED*: Docstrings are highlighted as @:comment@.

  See "#190":http://redmine.rubychan.de/issues/190.

h3. *NEW*: @Scanners::Raydebug@

Copied from @Scanners::Debug@, highlights the token dump instead of importing it. It also reacts to the @.raydebug@ file name suffix now.

h3. @Scanners::Ruby@

* *ADDED* more predefined keywords (see http://murfy.de/ruby-constants).

* *IMPROVED* support for singleton method definitions.

  See "#147":http://redmine.rubychan.de/issues/147.

* *FIXED*: Don't highlight methods with a capital letter as constants

  (eg. GL.PushMatrix).

* *NEW*: Highlight buggy floats (like .5) as @:error@.

* *CLEANUP* of documentation, names of constants and variables, state handling.

* *NEW*: Complicated rule for recognition of @foo=@ style method names.

* *NEW*: Handles @:keep_state@ option (a bit; experimental).

  Actually, Ruby checks if there is @[~>=]@, but not @=>@ following the name.

h3. @Scanners::Scheme@

* *CHANGED*: Does use @:operator@ instead of @:operator_fat@ now.

h3. @Scanners::SQL@

* *IMPROVED*: Extended list of keywords and functions (thanks to Joshua Galvez).

  See "#221":http://redmine.rubychan.de/issues/221.

* *FIXED*: Closes open string groups.

h3. @Scanners::YAML@

* *FIXED*: Allow spaces before colon in mappings.

  See "#231":http://redmine.rubychan.de/issues/231.

h3. *NEW*: @Styles::Alpha@

A style that uses transparent HSLA colors as defined in CSS 3.

See "#199":http://redmine.rubychan.de/issues/199.

h3. @FileType@

* *REMOVED* @FileType#shebang@ is a protected method now.

* *NEW*: Recognizes @.gemspec@, @.rjs@, @.rpdf@ extensions and @Capfile@ as Ruby.

  Thanks to the authors of the TextMate Ruby bundle!

h3. @Plugin@

* *IMPROVED*: @register_for@ sets the @plugin_id@; it can now be a @Symbol@.

h3. Internal API changes

* *FIXED* @Encoders::HTML#token@'s second parameter is no longer optional.

h2. Changes in 0.9.3

* *FIXED*: Documentation of Tokens.

  ("#218":http://redmine.rubychan.de/issues/218)

h3. @coderay@ executable

* *NEW*: automatic TTY detection (uses @Term@ encoder)

* *NEW*: optional 3rd parameter for the filename

* *FIXED*: Converted to UNIX format.

* *FIXED*: Warn about generated files.

* *FIXED*: Ensure line break after the output (especially for LoC counter).

h3. @Scanners::JavaScript@

* *FIXED*: Don't keep state of XML scanner between calls for E4X literals.

h3. @Scanners::Java@, @Scanners::JSON@

* *FIXED*: Close unfinished strings with the correct token kind.

h2. Changes in 0.9.2

* *NEW* Basic tests and a _Rakefile_ are now included in the Gem. [Flameeyes]

  A @doc@ task is also included.

* *FIXED* Use @$CODERAY_DEBUG@ for debugging instead of @$DEBUG@. [Trans]

  ("#192":http://redmine.rubychan.de/issues/192)

* *REMOVED* @Term::Ansicolor@ was bundled under _lib/_, but not used. [Flameeyes]

  ("#205":http://redmine.rubychan.de/issues/205)

* *WORKAROUND* for Ruby bug 

  "#2745":http://redmine.ruby-lang.org/issues/show/2745

h3. @Encoders::Term@

* *FIXED* strings are closed correctly

  ("#138":http://redmine.rubychan.de/issues/138)

* *FIXED* several token kinds had no associated color

  ("#139":http://redmine.rubychan.de/issues/139)

* *NEW* alias @terminal@

  *NOTE:* This encoder will be renamed to @Encoders::Terminal@ in the next release.

h3. @Scanners::Debug@

* *FIXED* Don't close tokens that are not open. Send @:error@ token instead.

h3. @Scanners::Groovy@

* *FIXED* token kind of closing brackets is @:operator@ instead of @nil@

  ("#148":http://redmine.rubychan.de/issues/148)

h3. @Scanners::PHP@

* *FIXED* allow @\@ operator (namespace separator)

  ("#209":http://redmine.rubychan.de/issues/209)

h3. @Scanners::YAML@

* *FIXED* doesn't send debug tokens when @$DEBUG@ is true [Trans]

  ("#149":http://redmine.rubychan.de/issues/149)

h2. Changes in 0.9.1

h3. Token classes

* *NEW* token classes @:complex@, @:decorator@, @:imaginary@

  (all for Python)

* *REMOVED* token class @:procedure@

  – use @:function@ or @:method@ instead.

h3. @Tokens@

* *NEW* method @#scanner@

  Stores the scanner.

* *REMOVED* methods @.write_token@, @.read_token@, @.escape@, @.unescape@

  They were only used by the @Tokens@ encoder, which was removed also.

h3. @Encoders::Encoder@

* *REMOVED* Don't require the _stringio_ library.

* *NEW* public methods @#open_token@, @#close_token@, @#begin_line@, @#end_line@

  These methods are called automatically, like @#text_token@.

* *NEW* proteced method @#append_encoded_token_to_output@

h3. @Encoders::Tokens@

* *REMOVED* – use @Tokens#dump@ and @Tokens.load@.

h3. @Encoders::Filter@

* *NEW*

  A @Filter@ encoder has another @Tokens@ instance as output.

h3. @Encoders::TokenClassFilter@

* *NEW*

  It takes 2 options, @:exclude@ and @:include@, that specify which token classes

  to include or exclude for the output. They can be a single token class,

  an @Array@ of classes, or the value @:all@.

h3. @Encoders::CommentFilter@

* *NEW*

  Removes tokens of the @:comment@ class.

h3. @Encoders::LinesOfCode@

* *NEW*

  Counts the lines of code according to the @KINDS_NOT_LOC@ token class list

  defined by the scanner. It uses the new @TokenClassFilter@.

  Alias: @:loc@, as in @tokens.loc@.

h3. @Encoders::JSON@

* *NEW*

  Outputs tokens in a simple JSON format.

h3. @Encoders::Term@

* *NEW* (beta, by Rob Aldred)

  Outputs code highlighted for a color terminal.

h3. @Encoders::HTML@

* *NEW* option @:title@ (default value is _CodeRay output_)

  Setting this changes the title of the HTML page.

* *NEW* option @:highlight_lines@ (default: @nil@)

  Highlights the given set of line numbers.

- *REMOVED* option @:level@

  It didn't do anything. CodeRay always outputs XHTML.

h3. @Encoders::Text@

* Uses @Encoder@ interface with @super@ and @#text_token@.

h3. @Encoders::XML@

* @FIXED@ ("#94":http://redmine.rubychan.de/issues/94)

  It didn't work at all.

h3. Scanners

* *NEW* Mapped @:h@ to @:c@, @:cplusplus@ and @:'c++'@ to @:cpp@,

  @:ecma@, @:ecmascript@, @:ecma_script@ to @:java_script@,

  @:pascal@ to @:delphi@, and @:plain@ to @:plaintext@.

h3. @Scanners::Scanner@

* *NEW* constant @KINDS_NOT_LOC@

  A list of all token classes not considered in LOC count.

  Added appropriate values for scanners.

* *NEW* method @#lang@ returns the scanner's lang, which is its @plugin_id@.

* *FIXED* automatic, safe UTF-8 detection _[Ruby 1.9]_

* *FIXED* column takes care of multibyte encodings _[Ruby 1.9]_

* *FIXED* is dumpable (@Tokens@ store their scanner in an @@scanner@ variable)

h3. @Scanners::Cpp@

* *NEW* (C++)

h3. @Scanners::Groovy@

* *NEW* (beta)

h3. @Scanners::Python@

* *NEW*

h3. @Scanners::PHP@

* *NEW* (based on Stefan Walk's work)

h3. @Scanners::SQL@

* *NEW* (based on code by Josh Goebel)

h3. @Scanners::C@

* *IMPROVED* added a list of @:directive@ tokens that were @:reserved@ before

* *IMPROVED* detection of labels

* *IMPROVED* allow @1L@ and @1LL@ style literals

h3. @Scanners::CSS@

* *IMPROVED* element selectors are highlighted as @:type@ instead of @:keyword@

h3. @Scanners::Delphi@

* *IMPROVED* Don't cache tokens in CaseIgnoringWordList.

h3. @Scanners::Java@

* *IMPROVED* @assert@ is highlighted as a @:keyword@ now

* *IMPROVED* @const@ and @goto@ are highlighted as @:reserved@

* *IMPROVED* @false@, @true@, and @null@ are highlighted as @:pre_constant@

* *IMPROVED* @threadsafe@ is no longer a @:directive@

* *IMPROVED* @String@ is highlighted as a @:pre_type@

* *IMPROVED* built-in classes ending with _Error_ or _Exception_ are

  highlighted as a @:exception@ instead of @:pre_type@

h3. @Scanners::JavaScript@

* *NEW* a list of @PREDEFINED_CONSTANTS@ to be highlighted as @:pre_constant@

* *NEW* XML literals are recognized and highlighted

* *NEW* function name highlighting

* *IMPROVED* @.1@ is highlighted a number

* *FIXED* strings close with the correct kind when terminated unexpectedly

h3. @Scanners::JSON@

* *IMPROVED* constants (@true@, @false@, @nil@) are highlighted as @:value@

h3. @Scanners::Ruby@

* *IMPROVED* @Patterns::KEYWORDS_EXPECTING_VALUE@ for more accurate

  @value_expected@ detection

* *IMPROVED* handling of @\@ as a string delimiter

* *IMPROVED* handling of unicode strings; automatic switching to unicode

* *IMPROVED* highlighting of @self.method@ definitions

* *REMOVED* @Patterns::FANCY_START_SAVE@ (obsolete)

* *FIXED* encoding issues _[Ruby 1.9]_

* *FIXED* a problem in early Ruby 1.8.6 patch versions with @Regexp.escape@

h3. @Scanners::YAML@

* *IMPROVED* indentation detection

h3. @Styles::Cycnus@

* changed a few colors (exceptions, inline strings, predefined types)

h3. @Plugin@

* *NEW* method @#title@

  Set and get the plugin's title. Titles can be arbitrary strings.

* *NEW* method @#helper@ loads helpers from different plugins

  Use this syntax: @helper 'other_plugin/helper_name'@

h3. @FileType@

* *NEW* @FileType[]@ takes @Pathname@ instances

* *NEW* regonizes @.cc@, @.cpp@, @.cp@, @.cxx@, @.c++@, @.C@, @.hh@, @.hpp@, @.h++@, @.cu@ extensions (C++)

  Thanks to Sander Cox and the TextMate C bundle.

* *NEW* regonizes @.pas@, @.dpr@ extensions (Delphi)

* *NEW* regonizes @.gvy@, @.groovy@ extensions (Groovy)

* *NEW* regonizes @.php@, @.php3@, @.php4@, @.php5@ extensions (PHP)

* *NEW* regonizes @.py@, @.py3@, @.pyw@ extensions (Python)

* *NEW* regonizes @.rxml@ extension (Ruby)

* *NEW* regonizes @.sql@ extension (SQL)

* File types list was sorted alphabetically.

h3. @CaseIgnoringWordList@

* *FIXED* ("#97":http://redmine.rubychan.de/issues/97)

  The default value is no longer ignored.

h3. @ForRedCloth@

* *FIXED* for RedCloth versions 4.2.0+ ("#119":http://redmine.rubychan.de/issues/119)

h3. Cleanups

* warnings about character classes _[Ruby 1.9]_

* encoding issues _[Ruby 1.9]_

* documentation, code