Changelog » History » Version 2

« Previous - Version 2/9 (diff) - Next » - Current version
Kornelius Kalnbach, 06/11/2010 01:27 PM

CodeRay Version History¶

This files lists all changes in the CodeRay library since the 0.8.4 release.

Table of contents
CodeRay Version History

Changes in 1.0¶

Direct Streaming¶

CodeRay 1.0 introduces Direct Streaming as a faster and simpler alternative to Tokens. It means that all Scanners, Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with this version.

The benefit of this change is more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9), a simpler API, and less code.

Changes related to the new tokens handling include:

CHANGED: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a special case; Tokens just encodes the given tokens into an Array for later use.
CHANGED: The token actions (text_token, begin_group etc.) are now public methods of Encoder and Tokens.
REWRITE of all Scanners, Encoders, Filters, and Tokens.
RENAMED :open and :close actions to :begin_group and :end_group.
RENAMED open_token and close_token methods to begin_group and end_group.
NEW method #tokens allows to add several tokens to the stream. Tokens and Encoders::Encoder define this method.
CHANGED The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible with earlier versions.
REMOVED TokenStream and the Streamable API and all related features like NotStreamableError are now obsolete and have been removed.

General changes¶

IMPROVED documentation in general; additions, corrections and cleanups
FIXED some image links in the documentation
IMPROVED Ruby 1.9 support (_._ not in $LOAD_PATH)

`Tokens`¶

REMOVED methods #stream?, #each_text_token.
REMOVED #text and #text_size methods. Use the Text encoder instead.
REMOVED special implementation of #each taking a filter parameter. Use TokenKindFilter instead.
NEW methods encode_with, count, begin_group, end_group, begin_line, and end_line.

RENAMED: `Tokens::AbbreviationForKind`¶

Renamed from ClassOfKind; the term "token class" is no longer used in CodeRay. Instead, tokens have kinds.
See #122.

REMOVED token kinds :attribute_name_fat, :attribute_value_fat, :operator_fat, :tag_fat, :xml_text, :open, and :close.
ADDED token kinds :filename, :namespace, and :eyecatcher.
CHANGED: Don't raise error for unknown token kinds unless in $CODERAY_DEBUG mode.

`Encoders::CommentFilter`¶

NEW alias :remove_comments

`Encoders::Filter`¶

NEW option tokens.
CHANGED: Now it simply delegates to the output.
REMOVED include_text_token? and include_block_token? methods.

`Encoders::HTML`¶

The HTML encoder was cleaned up and simplified.

CHANGED the default style to :alpha.
NEW: HTML 5 and CSS 3 compatible, IE incompatible.
See #215.
ADDED support for :line_number_anchors.
See #208.
CHANGED: Use double click to toggle line numbers in table mode (as single
click jumps to an anchor.)
REMOVED support for :line_numbers => :list.
FIXED splitting of lines for :line_numbers => :inline, so that the line
numbers don't get colored, too.
RENAMED Output#numerize to #number, which is an actual English word.

`Encoders::LinesOfCode`¶

CHANGED: compile and finish methods are now protected.

`Encoders::Terminal`¶

RENAMED from Encoders::Term, added :term alias.
CLEANUP: Use #setup's super, don't use :procedure token class.
CHANGED: #token's second parameter is no longer optional.
REMOVED colors for obsolete token kinds.
FIXED handling of line tokens.

`Encoders::Text`¶

FIXED default behavior of stripping the trailing newline.

RENAMED: `Encoders::TokenKindFilter`¶

Renamed from TokenClassFilter.

NEW: Handles token groups.
See #223.
RENAMED include_block_token? to include_group?.

`Encoders::Statistic`¶

CHANGED: Tokens actions are counted separately.

`Encoders::YAML`¶

NEW method call for allowing code like CodeRay::Duo[:python => :yaml].(code) in Ruby 1.9.

`Scanners::Scanner`¶

REMOVED helper method String#to_unix.
REMOVED method streamable?.
NEW: The #tokenize method also takes an Array of Strings as source. The
code is highlighted as one and split into parts of the input lengths
after that using Tokens#split_into_parts.

`Scanners::CSS`¶

NEW: Rudimentary support for the attr, counter, and counters functions.
See #224.
NEW: Rudimentary support for CSS 3 colors.
CHANGED: Attribute selectors are highlighted as :attribute_name instead of :string.

`Scanners::Debug`¶

NEW: Support for line tokens (begin_line and end_line represented by [ and ].)
FIXED: Don't send :error and nil tokens for buggy input any more.
FIXED: Closes unclosed tokens at the end of scan_tokens.
IMPROVED: Highlight unknown tokens as :error.
CHANGED: Raises an error when trying to end an invalid token group.

`Scanners::Delphi`¶

FIXED: Closes open string groups.

`Scanners::Diff`¶

NEW: Highlighting of code based on file names.
See ticket #52.

Use the :highlight_code option to turn this feature off. It's enabled
by default.

This is a very original feature. It enables multi-language highlighting for
  diff files, which is especially helpful for CodeRay development itself. The
  updated version of the scanner test suite generated .debug.diff.html files
  using this.

Note: This is still experimental. Tokens spanning more than one line
  may get highlighted incorrectly. CodeRay tries to keep scanner states
  between the lines and changes, but the quality of the results depend on
  the scanner.

NEW: Inline change highlighting, as suggested by Eric Thomas.
See ticket #227 for details.

Use the :inline_diff option to turn this feature off. It's enabled by
default.

For single-line changes (that is, a single deleted line followed by a single
  inserted line), this feature surrounds the changed parts with an
  :eyecatcher group which appears in a more saturated background color.
  The implementation is quite complex, and highly experimental. The problem
  with multi-layer tokenizing is that the tokens have to be split into parts.
  If the inline change starts, say, in the middle of a string, then additional
  :end_group and :begin_group tokens must be inserted to keep the group
  nesting intact. The extended Scanner#tokenize method and the new
  Tokens#split_into_parts method take care of this.

NEW: Highlight the file name in the change headers as :filename.
CHANGED: Highlight unknown lines as :comment instead of :head.

`Scanners::HTML`¶

FIXED: Closes open string groups.

`Scanners::JavaScript`¶

IMPROVED: Added NaN and Infinity to list of predefined constants.

`Scanners::Java`¶

NEW: Package names are highlighted as :namespace.
See #210.

`Scanners::Plaintext`¶

IMPROVED: Just returns the string without scanning (faster).
This is much faster than scanning until /\z/ in Ruby 1.8.

`Scanners::Python`¶

CHANGED: Docstrings are highlighted as :comment.
See #190.

NEW: `Scanners::Raydebug`¶

Copied from Scanners::Debug, highlights the token dump instead of importing it. It also reacts to the .raydebug file name suffix now.

`Scanners::Ruby`¶

ADDED more predefined keywords (see http://murfy.de/ruby-constants).
IMPROVED support for singleton method definitions.
See #147.
FIXED: Don't highlight methods with a capital letter as constants
(eg. GL.PushMatrix).
NEW: Highlight buggy floats (like .5) as :error.
CLEANUP of documentation, names of constants and variables, state handling.
NEW: Complicated rule for recognition of foo= style method names.
NEW: Handles :keep_state option (a bit; experimental).
Actually, Ruby checks if there is [~>=], but not => following the name.

`Scanners::Scheme`¶

CHANGED: Does use :operator instead of :operator_fat now.

`Scanners::SQL`¶

IMPROVED: Extended list of keywords and functions (thanks to Joshua Galvez).
See #221.
FIXED: Closes open string groups.

`Scanners::YAML`¶

FIXED: Allow spaces before colon in mappings.
See #231.

NEW: `Styles::Alpha`¶

A style that uses transparent HSLA colors as defined in CSS 3.
See #199.

`FileType`¶

REMOVED FileType#shebang is a protected method now.
NEW: Recognizes .gemspec, .rjs, .rpdf extensions and Capfile as Ruby.
Thanks to the authors of the TextMate Ruby bundle!

`Plugin`¶

IMPROVED: register_for sets the plugin_id; it can now be a Symbol.

Internal API changes¶

FIXED Encoders::HTML#token's second parameter is no longer optional.

Changes in 0.9.3¶

FIXED: Documentation of Tokens.
(#218)

`coderay` executable¶

NEW: automatic TTY detection (uses Term encoder)
NEW: optional 3rd parameter for the filename
FIXED: Converted to UNIX format.
FIXED: Warn about generated files.
FIXED: Ensure line break after the output (especially for LoC counter).

`Scanners::JavaScript`¶

FIXED: Don't keep state of XML scanner between calls for E4X literals.

`Scanners::Java`, `Scanners::JSON`¶

FIXED: Close unfinished strings with the correct token kind.

Changes in 0.9.2¶

NEW Basic tests and a Rakefile are now included in the Gem. [Flameeyes]
A doc task is also included.
FIXED Use $CODERAY_DEBUG for debugging instead of $DEBUG. [Trans]
(#192)
REMOVED Term::Ansicolor was bundled under lib/, but not used. [Flameeyes]
(#205)
WORKAROUND for Ruby bug
#2745

`Encoders::Term`¶

FIXED strings are closed correctly
(#138)
FIXED several token kinds had no associated color
(#139)
NEW alias terminal
NOTE: This encoder will be renamed to Encoders::Terminal in the next release.

`Scanners::Debug`¶

FIXED Don't close tokens that are not open. Send :error token instead.

`Scanners::Groovy`¶

FIXED token kind of closing brackets is :operator instead of nil
(#148)

`Scanners::PHP`¶

FIXED allow \ operator (namespace separator)
(#209)

`Scanners::YAML`¶

FIXED doesn't send debug tokens when $DEBUG is true [Trans]
(#149)

Changes in 0.9.1¶

Token classes¶

NEW token classes :complex, :decorator, :imaginary
(all for Python)
REMOVED token class :procedure
– use :function or :method instead.

`Tokens`¶

NEW method #scanner
Stores the scanner.
REMOVED methods .write_token, .read_token, .escape, .unescape
They were only used by the Tokens encoder, which was removed also.

`Encoders::Encoder`¶

REMOVED Don't require the stringio library.
NEW public methods #open_token, #close_token, #begin_line, #end_line
These methods are called automatically, like #text_token.
NEW proteced method #append_encoded_token_to_output

`Encoders::Tokens`¶

REMOVED – use Tokens#dump and Tokens.load.

`Encoders::Filter`¶

NEW
A Filter encoder has another Tokens instance as output.

`Encoders::TokenClassFilter`¶

NEW
It takes 2 options, :exclude and :include, that specify which token classes
to include or exclude for the output. They can be a single token class,
an Array of classes, or the value :all.

`Encoders::CommentFilter`¶

NEW
Removes tokens of the :comment class.

`Encoders::LinesOfCode`¶

NEW
Counts the lines of code according to the KINDS_NOT_LOC token class list
defined by the scanner. It uses the new TokenClassFilter.
```
Alias: :loc, as in tokens.loc.
```

`Encoders::JSON`¶

NEW
Outputs tokens in a simple JSON format.

`Encoders::Term`¶

NEW (beta, by Rob Aldred)
Outputs code highlighted for a color terminal.

`Encoders::HTML`¶

NEW option :title (default value is CodeRay output)
Setting this changes the title of the HTML page.
NEW option :highlight_lines (default: nil)
Highlights the given set of line numbers.
- REMOVED option :level
```
It didn't do anything. CodeRay always outputs XHTML.
```

`Encoders::Text`¶

Uses Encoder interface with super and #text_token.

`Encoders::XML`¶

FIXED (#94)
It didn't work at all.

Scanners¶

NEW Mapped :h to :c, :cplusplus and :'c++' to :cpp,
:ecma, :ecmascript, :ecma_script to :java_script,
:pascal to :delphi, and :plain to :plaintext.

`Scanners::Scanner`¶

NEW constant KINDS_NOT_LOC
A list of all token classes not considered in LOC count.
Added appropriate values for scanners.
NEW method #lang returns the scanner's lang, which is its plugin_id.
FIXED automatic, safe UTF-8 detection [Ruby 1.9]
FIXED column takes care of multibyte encodings [Ruby 1.9]
FIXED is dumpable (Tokens store their scanner in an @scanner variable)

`Scanners::Cpp`¶

NEW (C++)

`Scanners::Groovy`¶

NEW (beta)

`Scanners::Python`¶

NEW

`Scanners::PHP`¶

NEW (based on Stefan Walk's work)

`Scanners::SQL`¶

NEW (based on code by Josh Goebel)

`Scanners::C`¶

IMPROVED added a list of :directive tokens that were :reserved before
IMPROVED detection of labels
IMPROVED allow 1L and 1LL style literals

`Scanners::CSS`¶

IMPROVED element selectors are highlighted as :type instead of :keyword

`Scanners::Delphi`¶

IMPROVED Don't cache tokens in CaseIgnoringWordList.

`Scanners::Java`¶

IMPROVED assert is highlighted as a :keyword now
IMPROVED const and goto are highlighted as :reserved
IMPROVED false, true, and null are highlighted as :pre_constant
IMPROVED threadsafe is no longer a :directive
IMPROVED String is highlighted as a :pre_type
IMPROVED built-in classes ending with Error or Exception are
highlighted as a :exception instead of :pre_type

`Scanners::JavaScript`¶

NEW a list of PREDEFINED_CONSTANTS to be highlighted as :pre_constant
NEW XML literals are recognized and highlighted
NEW function name highlighting
IMPROVED .1 is highlighted a number
FIXED strings close with the correct kind when terminated unexpectedly

`Scanners::JSON`¶

IMPROVED constants (true, false, nil) are highlighted as :value

`Scanners::Ruby`¶

IMPROVED Patterns::KEYWORDS_EXPECTING_VALUE for more accurate
value_expected detection
IMPROVED handling of \ as a string delimiter
IMPROVED handling of unicode strings; automatic switching to unicode
IMPROVED highlighting of self.method definitions
REMOVED Patterns::FANCY_START_SAVE (obsolete)
FIXED encoding issues [Ruby 1.9]
FIXED a problem in early Ruby 1.8.6 patch versions with Regexp.escape

`Scanners::YAML`¶

IMPROVED indentation detection

`Styles::Cycnus`¶

changed a few colors (exceptions, inline strings, predefined types)

`Plugin`¶

NEW method #title
Set and get the plugin's title. Titles can be arbitrary strings.
NEW method #helper loads helpers from different plugins
Use this syntax: helper 'other_plugin/helper_name'

`FileType`¶

NEW FileType[] takes Pathname instances
NEW regonizes .cc, .cpp, .cp, .cxx, .c++, .C, .hh, .hpp, .h++, .cu extensions (C++)
Thanks to Sander Cox and the TextMate C bundle.
NEW regonizes .pas, .dpr extensions (Delphi)
NEW regonizes .gvy, .groovy extensions (Groovy)
NEW regonizes .php, .php3, .php4, .php5 extensions (PHP)
NEW regonizes .py, .py3, .pyw extensions (Python)
NEW regonizes .rxml extension (Ruby)
NEW regonizes .sql extension (SQL)
File types list was sorted alphabetically.

`CaseIgnoringWordList`¶

FIXED (#97)
The default value is no longer ignored.

`ForRedCloth`¶

FIXED for RedCloth versions 4.2.0+ (#119)

Cleanups¶

warnings about character classes [Ruby 1.9]
encoding issues [Ruby 1.9]
documentation, code

Files (0)

Project

General

Profile

CodeRay

Wiki

Changelog » History » Version 2

CodeRay Version History¶

Changes in 1.0¶

Direct Streaming¶

General changes¶

Tokens¶

RENAMED: Tokens::AbbreviationForKind¶

Encoders::CommentFilter¶

Encoders::Filter¶

Encoders::HTML¶

Encoders::LinesOfCode¶

Encoders::Terminal¶

Encoders::Text¶

RENAMED: Encoders::TokenKindFilter¶

Encoders::Statistic¶

Encoders::YAML¶

Scanners::Scanner¶

Scanners::CSS¶

Scanners::Debug¶

Scanners::Delphi¶

Scanners::Diff¶

Scanners::HTML¶

Scanners::JavaScript¶

Scanners::Java¶

Scanners::Plaintext¶

Scanners::Python¶

NEW: Scanners::Raydebug¶

Scanners::Ruby¶

Scanners::Scheme¶

Scanners::SQL¶

Scanners::YAML¶

NEW: Styles::Alpha¶

FileType¶

Plugin¶

Internal API changes¶

Changes in 0.9.3¶

coderay executable¶

Scanners::JavaScript¶

Scanners::Java, Scanners::JSON¶

Changes in 0.9.2¶

Encoders::Term¶

Scanners::Debug¶

Scanners::Groovy¶

Scanners::PHP¶

Scanners::YAML¶

Changes in 0.9.1¶

Token classes¶

Tokens¶

Encoders::Encoder¶

Encoders::Tokens¶

Encoders::Filter¶

Encoders::TokenClassFilter¶

Encoders::CommentFilter¶

Encoders::LinesOfCode¶

Encoders::JSON¶

Encoders::Term¶

Encoders::HTML¶

Encoders::Text¶

Encoders::XML¶

Scanners¶

Scanners::Scanner¶

Scanners::Cpp¶

Scanners::Groovy¶

Scanners::Python¶

Scanners::PHP¶

Scanners::SQL¶

Scanners::C¶

Scanners::CSS¶

Scanners::Delphi¶

Scanners::Java¶

Scanners::JavaScript¶

Scanners::JSON¶

Scanners::Ruby¶

Scanners::YAML¶

Styles::Cycnus¶

`Tokens`¶

RENAMED: `Tokens::AbbreviationForKind`¶

`Encoders::CommentFilter`¶

`Encoders::Filter`¶

`Encoders::HTML`¶

`Encoders::LinesOfCode`¶

`Encoders::Terminal`¶

`Encoders::Text`¶

RENAMED: `Encoders::TokenKindFilter`¶

`Encoders::Statistic`¶

`Encoders::YAML`¶

`Scanners::Scanner`¶

`Scanners::CSS`¶

`Scanners::Debug`¶

`Scanners::Delphi`¶

`Scanners::Diff`¶

`Scanners::HTML`¶

`Scanners::JavaScript`¶

`Scanners::Java`¶

`Scanners::Plaintext`¶

`Scanners::Python`¶

NEW: `Scanners::Raydebug`¶

`Scanners::Ruby`¶

`Scanners::Scheme`¶

`Scanners::SQL`¶

`Scanners::YAML`¶

NEW: `Styles::Alpha`¶

`FileType`¶

`Plugin`¶

`coderay` executable¶

`Scanners::JavaScript`¶

`Scanners::Java`, `Scanners::JSON`¶

`Encoders::Term`¶

`Scanners::Debug`¶

`Scanners::Groovy`¶

`Scanners::PHP`¶

`Scanners::YAML`¶

`Tokens`¶

`Encoders::Encoder`¶

`Encoders::Tokens`¶

`Encoders::Filter`¶

`Encoders::TokenClassFilter`¶

`Encoders::CommentFilter`¶

`Encoders::LinesOfCode`¶

`Encoders::JSON`¶

`Encoders::Term`¶

`Encoders::HTML`¶

`Encoders::Text`¶

`Encoders::XML`¶

`Scanners::Scanner`¶

`Scanners::Cpp`¶

`Scanners::Groovy`¶

`Scanners::Python`¶

`Scanners::PHP`¶

`Scanners::SQL`¶

`Scanners::C`¶

`Scanners::CSS`¶

`Scanners::Delphi`¶

`Scanners::Java`¶

`Scanners::JavaScript`¶

`Scanners::JSON`¶

`Scanners::Ruby`¶

`Scanners::YAML`¶

`Styles::Cycnus`¶

`Plugin`¶

`FileType`¶

`CaseIgnoringWordList`¶

`ForRedCloth`¶