Project

General

Profile

Changelog » History » Version 4

« Previous - Version 4/9 (diff) - Next » - Current version
Kornelius Kalnbach, 07/04/2011 02:39 AM


CodeRay Version History

This files lists all changes in the CodeRay library since the 0.8.4 release.

Changes in 1.0

Direct Streaming

CodeRay 1.0 introduces Direct Streaming as a faster and simpler alternative to Tokens. It means that all Scanners, Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with this version.

The benefit of this change is more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9), a simpler API, and less code.

Changes related to the new tokens handling include:
  • CHANGED: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a special case; Tokens just encodes the given tokens into an Array for later use.
  • CHANGED: The token actions (text_token, begin_group etc.) are now public methods of Encoder and Tokens.
  • REWRITE of all Scanners, Encoders, Filters, and Tokens.
  • RENAMED :open and :close actions to :begin_group and :end_group.
  • RENAMED open_token and close_token methods to begin_group and end_group.
  • NEW method #tokens allows to add several tokens to the stream. Tokens and Encoders::Encoder define this method.
  • CHANGED The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible with earlier versions.
  • REMOVED TokenStream and the Streamable API and all related features like NotStreamableError are now obsolete and have been removed.

General changes

  • IMPROVED documentation in general; additions, corrections and cleanups.
  • NEW: Extended support and usage of HTML5 and CSS 3 features.
  • IMPROVED Ruby 1.9 support (_._ not in $LOAD_PATH)
  • IMPROVED speed of HTML encoding when using CSS classes.
  • IMPROVED: Faster startup by replacing require with autoload. CodeRay
    features will now only be loaded when they're needed.
  • FIXED some image links in the documentation.
  • ADDED a lot of tests.

Helpers

The helper classes were cleaned up; see below for details.

  • CHANGED Plugin API was simplified and stripped of all unnecessary features.
  • CHANGED Moved GZip and FileType libraries into CodeRay; cleaned them up.

Tokens

  • REMOVED methods #stream?, #each_text_token.
  • REMOVED #text and #text_size methods. Use the Text encoder instead.
  • REMOVED special implementation of #each taking a filter parameter. Use TokenKindFilter instead.
  • NEW methods encode_with, count, begin_group, end_group, begin_line, and end_line.

RENAMED: TokenKinds

Renamed from Tokens::ClassOfKind (was also Tokens::AbbreviationForKind for a while).
The term "token class" is no longer used in CodeRay. Instead, tokens have kinds.
See #122.

  • RENAMED :pre_constant and :pre_type to :predefined_constant and predefined_type.
  • REMOVED token kinds :attribute_name_fat, :attribute_value_fat, :operator_fat,
    :tag_fat, :xml_text, :nesting_delimiter, :open, and :close.
  • ADDED token kinds :filename, :namespace, and :eyecatcher.
  • CHANGED: Don't raise error for unknown token kinds unless in $CODERAY_DEBUG mode.
  • CHANGED the value for a token kind that is not highlighted from
    :NO_HIGHLIGHT to false.

Duo

  • NEW method call for allowing code like CodeRay::Duo[:python => :yaml].(code) in Ruby 1.9.

Encoders::CommentFilter

  • NEW alias :remove_comments

Encoders::Filter

  • NEW option tokens.
  • CHANGED: Now it simply delegates to the output.
  • REMOVED include_text_token? and include_block_token? methods.

Encoders::HTML

The HTML encoder was cleaned up and simplified.

  • CHANGED the default style to :alpha.
  • NEW: HTML5 and CSS 3 compatible.
    See #215.
  • ADDED support for :line_number_anchors.
    See #208.
  • CHANGED: Use double click to toggle line numbers in table mode (as single
    click jumps to an anchor.)
  • REMOVED support for :line_numbers => :list.
  • FIXED splitting of lines for :line_numbers => :inline, so that the line
    numbers don't get colored, too.
  • RENAMED Output#numerize to #number, which is an actual English word.

Encoders::LinesOfCode

  • CHANGED: compile and finish methods are now protected.

Encoders::Terminal

  • RENAMED from Encoders::Term, added :term alias.
  • CLEANUP: Use #setup's super, don't use :procedure token class.
  • CHANGED: #token's second parameter is no longer optional.
  • REMOVED colors for obsolete token kinds.
  • FIXED handling of line tokens.

Encoders::Text

  • FIXED default behavior of stripping the trailing newline.

RENAMED: Encoders::TokenKindFilter

Renamed from TokenClassFilter.

  • NEW: Handles token groups.
    See #223.
  • RENAMED include_block_token? to include_group?.

Encoders::Statistic

  • CHANGED: Tokens actions are counted separately.

Scanners::Scanner

  • REMOVED helper method String#to_unix.
  • REMOVED method #streamable?.
  • REMOVED #marshal_load and #marshal_dump.
  • RENAMED class method normify to normalize; it also deals with encoding now.
  • NEW methods #file_extension and #encoding.
  • NEW: The #tokenize method also takes an Array of Strings as source. The
    code is highlighted as one and split into parts of the input lengths
    after that using Tokens#split_into_parts.

NEW: Scanners::Clojure

Thanks to Licenser, CodeRay now supports the Clojure language.

Scanners::CSS

  • NEW: Rudimentary support for the attr, counter, and counters functions.
    See #224.
  • NEW: Rudimentary support for CSS 3 colors.
  • CHANGED: Attribute selectors are highlighted as :attribute_name instead of :string.
  • CHANGED: Comments are scanned as one token instead of three.

Scanners::Debug

  • NEW: Support for line tokens (begin_line and end_line represented by [ and ].)
  • FIXED: Don't send :error and nil tokens for buggy input any more.
  • FIXED: Closes unclosed tokens at the end of scan_tokens.
  • IMPROVED: Highlight unknown tokens as :error.
  • CHANGED: Raises an error when trying to end an invalid token group.

Scanners::Delphi

  • FIXED: Closes open string groups.

Scanners::Diff

  • NEW: Highlighting of code based on file names.
    See ticket #52.

    Use the :highlight_code option to turn this feature off. It's enabled
    by default.

    This is a very original feature. It enables multi-language highlighting for
    diff files, which is especially helpful for CodeRay development itself. The
    updated version of the scanner test suite generated .debug.diff.html files
    using this.
    Note: This is still experimental. Tokens spanning more than one line
    may get highlighted incorrectly. CodeRay tries to keep scanner states
    between the lines and changes, but the quality of the results depend on
    the scanner.
  • NEW: Inline change highlighting, as suggested by Eric Thomas.
    See ticket #227 for details.

    Use the :inline_diff option to turn this feature off. It's enabled by
    default.

    For single-line changes (that is, a single deleted line followed by a single
    inserted line), this feature surrounds the changed parts with an
    :eyecatcher group which appears in a more saturated background color.
    The implementation is quite complex, and highly experimental. The problem
    with multi-layer tokenizing is that the tokens have to be split into parts.
    If the inline change starts, say, in the middle of a string, then additional
    :end_group and :begin_group tokens must be inserted to keep the group
    nesting intact. The extended Scanner#tokenize method and the new
    Tokens#split_into_parts method take care of this.
  • NEW: Highlight the file name in the change headers as :filename.
  • CHANGED: Highlight unknown lines as :comment instead of :head.

Scanners::HTML

  • FIXED: Closes open string groups.

Scanners::JavaScript

  • IMPROVED: Added NaN and Infinity to list of predefined constants.
  • IMPROVED recognition of RegExp literals with leading spaces.

Scanners::Java

  • NEW: Package names are highlighted as :namespace.
    See #210.

RENAMED: Scanners::Text (was Scanners::Plaintext)

  • IMPROVED: Just returns the string without scanning (faster).

    This is much faster than scanning until /\z/ in Ruby 1.8.

Scanners::Python

  • CHANGED: Docstrings are highlighted as :comment.
    See #190.

NEW: Scanners::Raydebug

Copied from Scanners::Debug, highlights the token dump instead of importing it. It also reacts to the .raydebug file name suffix now.

Scanners::Ruby

  • ADDED more predefined keywords (see http://murfy.de/ruby-constants).
  • IMPROVED support for singleton method definitions.
    See #147.
  • FIXED: Don't highlight methods with a capital letter as constants
    (eg. GL.PushMatrix).
  • NEW: Highlight buggy floats (like .5) as :error.
  • CLEANUP of documentation, names of constants and variables, state handling.

    Moved StringState class from patterns.rb into a separate file.

  • NEW: Complicated rule for recognition of foo= style method names.
  • NEW: Handles :keep_state option (a bit; experimental).

    Actually, Ruby checks if there is [~>=], but not => following the name.

  • REMOVED EncodingError

Scanners::Scheme

  • CHANGED: Does use :operator instead of :operator_fat now.

Scanners::SQL

  • IMPROVED: Extended list of keywords and functions (thanks to
    Joshua Galvez, Etienne Massip, and others).

    See #221.

  • FIXED: Closes open string groups.
  • FIXED: Words after . are always recognized as :ident.

Scanners::YAML

  • FIXED: Allow spaces before colon in mappings.

    See #231.

NEW: Styles::Alpha

A style that uses transparent HSLA colors as defined in CSS 3.
See #199.

It also uses the CSS 3 property user-select: none to keep the user from selecting the line numbers. This is especially nice for :inline line numbers.
See #226.

WordList

Stripped down to 19 LOC.

  • REMOVED caching option because it creates memory leaks.
  • REMOVED block option.

FileType

  • REMOVED FileType#shebang is a protected method now.
  • NEW: Recognizes .gemspec, .rjs, .rpdf extensions, Gemfile, and Capfile as Ruby.

    Thanks to the authors of the TextMate Ruby bundle!

Plugin

  • IMPROVED: register_for sets the plugin_id; it can now be a Symbol.
  • ADDED PluginHost#const_missing method: Plugins are loaded automatically.
    Using Scanners::JavaScript in your code loads scanners/java_script.rb.
  • ADDED #all_plugins method to simplify getting
    information about all available plugins (suggested by bnhymn).

GZip

  • MOVED into CodeRay namespace.
  • MOVED file from gzip_simple.rb to gzip.rb.
  • REMOVED String extensions.

Internal API changes

  • FIXED Encoders::HTML#token's second parameter is no longer optional.
  • CHANGED Encoders::HTML::Output's API.
  • REMOVED lots of unused methods.

Changes in 0.9.8 "banister" [2011-05-01]

Fixes for JRuby's 1.9 mode and minor issues.

Rake tasks

  • REMOVED obsolete has_rdoc gem specification, fixing a warning.

Scanners::Scanner

  • NEW method #scan_rest replaces scan_until(/\z/), which is broken in JRuby 1.6 --1.9 mode.
    See #297.

Scanners::CSS

  • FIXED LOC counting (should be 0).
    See #296.

Scanners::Ruby

  • FIXED the IDENT pattern not to use character properties, which are broken in JRuby 1.6 --1.9 mode.
    See #297, thanks to banister for reporting!

Scanners::SQL

  • ADDED more keywords: between, databases, distinct, fields, full, having, is, prompt, tables.
    See #221, thanks to Etienne Massip again.

FileType

  • NEW regonizes ColdFusion file type extensions .cfm and .cfc as XML.
    See #298, thanks to Emidio Stani.

Changes in 0.9.7 "Etienne" [2011-01-14]

Fixes a dangerous JavaScript scanner bug, and a testing problem with Ruby 1.9.1.

Tests

  • FIXED The functional tests now load the lib directory (instead of the gem) in Ruby 1.9.1.

Scanners::JavaScript

  • FIXED KEY_CHECK_PATTERN regexp
    See #264, thanks to Etienne Massip!

Changes in 0.9.6 "WoNáDo" [2010-11-25]

Minor improvements to the Ruby scanner and a fix for Ruby 1.9.

Scanners::Ruby

  • IMPROVED handling of new hash syntax (keys are marked as :key now,
    colon is a separate :operator token, all idents can be used as keys)
    See #257, thanks to WoNáDo!
  • ADDED __ENCODING__ magic constant (Ruby 1.9)
  • FIXED: Scanner no longer tries to modify the input string on Ruby 1.9.
    See #260, thanks to Jan Lelis!

Changes in 0.9.5 "Germany.rb" [2010-09-28]

Support for Rubinius (#251), improved mutlibyte handling, Ruby 1.9 syntax, and valid HTML.

Encoders::HTML

  • FIXED: Line tokens use span with display: block instead of div, which was invalid HTML (#255).

Scanner::Scanner

  • IMPROVED handling of encodings in Ruby 1.9: UTF-8 and Windows-1252 are checked.
  • NEW: Invalid chars will be converted to ? in Ruby 1.9.
  • FIXED string= method for Rubinius. See issue 481 on their site.

Scanners::CSS

  • FIXED: Don't use non-ASCII regexps.

Scanners::Diff

  • FIXED: Highlight unexpected lines as :comment.

Scanners::PHP

  • FIXED: Use ASCII-8BIT encoding for now.

Scanners::Ruby

  • ADDED support for some Ruby 1.9 syntax (#254):
    • the -> lambda shortcut
    • new Hash syntax using colons ({ a: b })
  • FIXED: Use UTF-8 encoding.
  • IMPROVED unicode support on Ruby 1.8 (#253).
  • FIXED recognition of non-ASCII identifiers in Ruby 1.9, JRuby, and Rubinius (#253).
  • CHANGED heredoc recognition to ignore delimiters starting with a digit. This is incorrect, but causes less false positives.

Scanners::SQL

  • FIXED scanning of comments; nice catch, Rubinius!
    (#252)

Changes in 0.9.4 "Ramadan" [2010-08-31]

Updated command line interface and minor scanner fixes for the Diff, HTML, and RHTML scanners.

coderay executable

  • FIXED: Partly rewritten, simplified, fixed.
    (#244)

Scanners::Diff

  • FIXED handling of change headers with code on the same line as the @@ marker.
    (#247)

Scanners::HTML

  • FIXED a missing regexp modifier that slowed down the scanning.
    (#245)

Scanners::RHTML

  • FIXED highlighting of ERB comment blocks.
    (#246)

Changes in 0.9.3 "Eyjafjallajökull" [2010-04-18]

  • FIXED: Documentation of Tokens.
    (#218)

coderay executable

  • NEW: automatic TTY detection (uses Term encoder)
  • NEW: optional 3rd parameter for the filename
  • FIXED: Converted to UNIX format.
  • FIXED: Warn about generated files.
  • FIXED: Ensure line break after the output (especially for LoC counter).

Scanners::JavaScript

  • FIXED: Don't keep state of XML scanner between calls for E4X literals.

Scanners::Java, Scanners::JSON

  • FIXED: Close unfinished strings with the correct token kind.

Changes in 0.9.2 "Flameeyes" [2010-03-14]

  • NEW Basic tests and a Rakefile are now included in the Gem. [Flameeyes]
    A doc task is also included.
  • FIXED Use $CODERAY_DEBUG for debugging instead of $DEBUG. [Trans]
    (#192)
  • REMOVED Term::Ansicolor was bundled under lib/, but not used. [Flameeyes]
    (#205)
  • WORKAROUND for Ruby bug
    #2745

Encoders::Term

  • FIXED strings are closed correctly
    (#138)
  • FIXED several token kinds had no associated color
    (#139)
  • NEW alias terminal

    NOTE: This encoder will be renamed to Encoders::Terminal in the next release.

Scanners::Debug

  • FIXED Don't close tokens that are not open. Send :error token instead.

Scanners::Groovy

  • FIXED token kind of closing brackets is :operator instead of nil
    (#148)

Scanners::PHP

  • FIXED allow \ operator (namespace separator)
    (#209)

Scanners::YAML

  • FIXED doesn't send debug tokens when $DEBUG is true [Trans]
    (#149)

Changes in 0.9.1 [2009-12-31]

Token classes

  • NEW token classes :complex, :decorator, :imaginary
    (all for Python)
  • REMOVED token class :procedure
    – use :function or :method instead.

Tokens

  • NEW method #scanner

    Stores the scanner.

  • REMOVED methods .write_token, .read_token, .escape, .unescape

    They were only used by the Tokens encoder, which was removed also.

Encoders::Encoder

  • REMOVED Don't require the stringio library.
  • NEW public methods #open_token, #close_token, #begin_line, #end_line
    These methods are called automatically, like #text_token.
  • NEW proteced method #append_encoded_token_to_output

Encoders::Tokens

  • REMOVED – use Tokens#dump and Tokens.load.

Encoders::Filter

  • NEW
    A Filter encoder has another Tokens instance as output.

Encoders::TokenClassFilter

  • NEW

    It takes 2 options, :exclude and :include, that specify which token classes
    to include or exclude for the output. They can be a single token class,
    an Array of classes, or the value :all.

Encoders::CommentFilter

  • NEW

    Removes tokens of the :comment class.

Encoders::LinesOfCode

  • NEW

    Counts the lines of code according to the KINDS_NOT_LOC token class list
    defined by the scanner. It uses the new TokenClassFilter.

    Alias: :loc, as in tokens.loc.

Encoders::JSON

  • NEW

    Outputs tokens in a simple JSON format.

Encoders::Term

  • NEW (beta, by Rob Aldred)

    Outputs code highlighted for a color terminal.

Encoders::HTML

  • NEW option :title (default value is CodeRay output)

    Setting this changes the title of the HTML page.

  • NEW option :highlight_lines (default: nil)

    Highlights the given set of line numbers.
    - REMOVED option :level

    It didn't do anything. CodeRay always outputs XHTML.

Encoders::Text

  • Uses Encoder interface with super and #text_token.

Encoders::XML

  • FIXED (#94)

    It didn't work at all.

Scanners

  • NEW Mapped :h to :c, :cplusplus and :'c++' to :cpp,
    :ecma, :ecmascript, :ecma_script to :java_script,
    :pascal to :delphi, and :plain to :plaintext.

Scanners::Scanner

  • NEW constant KINDS_NOT_LOC

    A list of all token classes not considered in LOC count.
    Added appropriate values for scanners.

  • NEW method #lang returns the scanner's lang, which is its plugin_id.
  • FIXED automatic, safe UTF-8 detection [Ruby 1.9]
  • FIXED column takes care of multibyte encodings [Ruby 1.9]
  • FIXED is dumpable (Tokens store their scanner in an @scanner variable)

Scanners::Cpp

  • NEW (C++)

Scanners::Groovy

  • NEW (beta)

Scanners::Python

  • NEW

Scanners::PHP

  • NEW (based on Stefan Walk's work)

Scanners::SQL

  • NEW (based on code by Josh Goebel)

Scanners::C

  • IMPROVED added a list of :directive tokens that were :reserved before
  • IMPROVED detection of labels
  • IMPROVED allow 1L and 1LL style literals

Scanners::CSS

  • IMPROVED element selectors are highlighted as :type instead of :keyword

Scanners::Delphi

  • IMPROVED Don't cache tokens in CaseIgnoringWordList.

Scanners::Java

  • IMPROVED assert is highlighted as a :keyword now
  • IMPROVED const and goto are highlighted as :reserved
  • IMPROVED false, true, and null are highlighted as :pre_constant
  • IMPROVED threadsafe is no longer a :directive
  • IMPROVED String is highlighted as a :pre_type
  • IMPROVED built-in classes ending with Error or Exception are
    highlighted as a :exception instead of :pre_type

Scanners::JavaScript

  • NEW a list of PREDEFINED_CONSTANTS to be highlighted as :pre_constant
  • NEW XML literals are recognized and highlighted
  • NEW function name highlighting
  • IMPROVED .1 is highlighted a number
  • FIXED strings close with the correct kind when terminated unexpectedly

Scanners::JSON

  • IMPROVED constants (true, false, nil) are highlighted as :value

Scanners::Ruby

  • IMPROVED Patterns::KEYWORDS_EXPECTING_VALUE for more accurate
    value_expected detection
  • IMPROVED handling of \ as a string delimiter
  • IMPROVED handling of unicode strings; automatic switching to unicode
  • IMPROVED highlighting of self.method definitions
  • REMOVED Patterns::FANCY_START_SAVE (obsolete)
  • FIXED encoding issues [Ruby 1.9]
  • FIXED a problem in early Ruby 1.8.6 patch versions with Regexp.escape

Scanners::YAML

  • IMPROVED indentation detection

Styles::Cycnus

  • changed a few colors (exceptions, inline strings, predefined types)

Plugin

  • NEW method #title

    Set and get the plugin's title. Titles can be arbitrary strings.

  • NEW method #helper loads helpers from different plugins

    Use this syntax: helper 'other_plugin/helper_name'

FileType

  • NEW FileType[] takes Pathname instances
  • NEW regonizes .cc, .cpp, .cp, .cxx, .c++, .C, .hh, .hpp, .h++, .cu extensions (C++)

    Thanks to Sander Cox and the TextMate C bundle.

  • NEW regonizes .pas, .dpr extensions (Delphi)
  • NEW regonizes .gvy, .groovy extensions (Groovy)
  • NEW regonizes .php, .php3, .php4, .php5 extensions (PHP)
  • NEW regonizes .py, .py3, .pyw extensions (Python)
  • NEW regonizes .rxml extension (Ruby)
  • NEW regonizes .sql extension (SQL)
  • File types list was sorted alphabetically.

CaseIgnoringWordList

  • FIXED (#97)

    The default value is no longer ignored.

ForRedCloth

  • FIXED for RedCloth versions 4.2.0+ (#119)

Cleanups

  • warnings about character classes [Ruby 1.9]
  • encoding issues [Ruby 1.9]
  • documentation, code