Refactor fenced_code & codehilite options (#816)

* Add `language-` prefix to output when syntax highlighting is
  disabled for both codehilite and fenced_code extensions.
* Add `lang_prefix` config option to customize the prefix.
* Add a 'pygments' env to tox which runs the tests with Pygments
  installed. Pygments is locked to a specific version in the env.
* Updated codehilite to accept any Pygments options.
* Refactor fenced code attributes.
   - ID attr is defined on `pre` tag.
    - Add support for attr_list extension, which allows setting arbitrary 
      attributes. 
    - When syntax highlighting is enabled, any pygments options can
       be defined per block in the attr list.
    - For backward compatibility, continue to support `hi_lines` outside
      of an attr_list. That is the only attr other than lang which is allowed
      without the brackets (`{}`) of an attr list. Note that if the brackets
      exist, then everything, including lang and hl_lines, must be within
      them.
* Resolves #775. Resolves #334. Addresses #652.

Author: waylan

.github/workflows/tox.yml

@@ -20,7 +20,7 @@ jobs:
       fail-fast: false
       max-parallel: 4
       matrix:
-        tox-env: [py35, py36, py37, py38, pypy3]
+        tox-env: [py35, py36, py37, py38, pypy3, pygments]
         include:
           - tox-env: py35
             python-version: 3.5
@@ -32,6 +32,8 @@ jobs:
             python-version: 3.8
           - tox-env: pypy3
             python-version: pypy3
+          - tox-env: pygments
+            python-version: 3.7
 
     env:
       TOXENV: ${{ matrix.tox-env }}

.spell-dict

@@ -33,7 +33,7 @@ ElementTree
 encodings
 extendMarkdown
 Fauske
-Formatter
+formatter
 Fortin
 GitHub
 globals
@@ -77,6 +77,8 @@ PHP
 Posix
 postprocessor
 postprocessors
+prepend
+prepended
 preprocessor
 preprocessors
 Pygments

docs/change_log/index.md

@@ -3,9 +3,7 @@ title: Change Log
 Python-Markdown Change Log
 =========================
 
-version 3.3.3 (under development)
-
-* Document limitations of `attr_list` extension (#965).
+Under development: version 3.3 ([Notes](release-3.3.md)).
 
 May 8, 2020: version 3.2.2 (a bug-fix release).
 
@@ -15,7 +13,7 @@ May 8, 2020: version 3.2.2 (a bug-fix release).
 * Do not double escape entities in TOC.
 * Correctly report if an extension raises a `TypeError` (#939).
 * Raise a `KeyError` when attempting to delete a nonexistent key from the
-  extension registry (#939). 
+  extension registry (#939).
 * Remove import of `packaging` (or `pkg_resources` fallback) entirely.
 * Remove `setuptools` as a run-time dependency (`install_required`).
 

docs/change_log/release-3.3.md

@@ -7,10 +7,56 @@ PyPy3.
 
 ## Backwards-incompatible changes
 
+### The prefix `language-` is now prepended to all language classes by default on code blocks.
+
+The [HTML5 spec][spec] recommends that the class defining the language of a code block be prefixed with `language-`.
+Therefore, by default, both the [fenced_code] and [codehilite] extensions now prepend the prefix when code
+highlighting is disabled.
+
+If you have previously been including the prefix manually in your fenced code blocks, then you will not want a second
+instance of the prefix. Similarly, if you are using a third party syntax highlighting tool which does not recognize
+the prefix, or requires a different prefix, then you will want to redefine the prefix globally using the `lang_prefix`
+configuration option of either the `fenced_code` or `codehilite` extensions.
+
+For example, to configure `fenced_code` to not apply any prefix (the previous behavior), set the option to an empty string:
+
+```python
+from markdown.extensions.fenced_code import FencedCodeExtension
+
+markdown.markdown(src, extensions=[FencedCodeExtension(lang_prefix='')])
+```
+
+!!! note
+    When code highlighting is [enabled], the output from Pygments is used unaltered. Currently, Pygments does not
+    provide an option to include the language class in the output, let alone prefix it. Therefore, any language prefix
+    is only applied when syntax highlighting is disabled.
+
 ## New features
 
+The following new features have been included in the 3.3 release:
+
+* All Pygments' options are now available for syntax highlighting (#816).
+    - The [Codehilite](../extensions/code_hilite.md) extension now accepts any options
+      which Pygments supports as global configuration settings on the extension.
+    - [Fenced Code Blocks](../extensions/fenced_code_blocks.md) will accept any of the
+      same options on individual code blocks.
+    - Any of the previously supported aliases to Pygments' options continue to be
+      supported at this time. However, it is recommended that the Pygments option names
+      be used directly to ensure continued compatibility in the future.
+
+* [Fenced Code Blocks](../extensions/fenced_code_blocks.md) now work with
+  [Attribute Lists](../extensions/attr_list.md) when syntax highlighting is disabled.
+  Any random HTML attribute can be defined and set on the `<code>` tag of fenced code
+  blocks when the `attr_list` extension is enabled (#816).
+
 ## Bug fixes
 
 The following bug fixes are included in the 3.3 release:
 
 * Fix issues with complex emphasis (#979).
+* Limitations of `attr_list` extension are Documented (#965).
+
+[spec]: https://www.w3.org/TR/html5/text-level-semantics.html#the-code-element
+[fenced_code]: ../extensions/fenced_code_blocks.md
+[codehilite]: ../extensions/code_hilite.md
+[enabled]: ../extensions/fenced_code_blocks.md#enabling-syntax-highlighting

docs/extensions/code_hilite.md

@@ -185,54 +185,67 @@ configuring extensions.
 
 The following options are provided to configure the output:
 
-* **`linenums`**:
-    Use line numbers. Possible values are `True` for yes, `False` for no and
-    `None` for auto. Defaults to `None`.
+* **`linenums`**{ #linenums }:
+    An alias to Pygments' `linenos` formatter option. Possible values are `True` for yes, `False` for no and `None`
+    for auto. Defaults to `None`.
 
     Using `True` will force every code block to have line numbers, even when
     using colons (`:::`) for language identification.
 
     Using `False` will turn off all line numbers, even when using shebangs
     (`#!`) for language identification.
 
-* **`guess_lang`**:
+* **`guess_lang`**{ #guess_lang }:
     Automatic language detection. Defaults to `True`.
 
     Using `False` will prevent Pygments from guessing the language, and thus
     highlighting blocks only when you explicitly set the language.
 
-* **`css_class`**:
-    Set CSS class name for the wrapper `<div>` tag. Defaults to
+* **`css_class`**{ #css_class }:
+    An alias to Pygments `cssclass` formatter option. Set CSS class name for the wrapper `<div>` tag. Defaults to
     `codehilite`.
 
-* **`pygments_style`**:
+* **`pygments_style`**{ #pygments_style }:
     Pygments HTML Formatter Style (`ColorScheme`). Defaults to `default`.
 
     !!! Note
         This is useful only when `noclasses` is set to `True`, otherwise the
         CSS styles must be provided by the end user.
 
-* **`noclasses`**:
+* **`noclasses`**{ #noclasses }:
     Use inline styles instead of CSS classes. Defaults to `False`.
 
-* **`use_pygments`**:
+* **`use_pygments`**{ #use_pygments }:
     Specifies the use of Pygments in generating the output.
 
     If `True` (the default) and Pygments is available, CodeHilite will use
     Pygments to analyze and format the output. Additionally, if using Pygments
     &gt;= 2.4, the output will be wrapped in `<code>` tags, whereas earlier
     versions will not.
 
-    Otherwise, Pygments will not be used. If a language is defined for a code
-    block, it will be assigned to the `<code>` tag as a class in the manner
-    suggested by the [HTML5 spec][spec] (alternate output will not be
-    entertained) and may be used by a JavaScript library in the browser to
-    highlight the code block.
-    
+    Otherwise, Pygments will not be used. If a language is defined for a code block, it will be assigned to the
+    `<code>` tag as a class in the manner suggested by the [HTML5 spec][spec] and may be used by a JavaScript library
+    in the browser to highlight the code block. See the [`lang_prefix`](#lang_prefix) option to customize the prefix.
+
+* **`lang_prefix`**{ #lang_prefix }:
+    The prefix prepended to the language class assigned to the HTML `<code>` tag. Default: `language-`.
+
+    This option only applies when `use_pygments` is `False` as Pygments does not provide an option to include a
+    language prefix.
+
+
+* Any other Pygments' options:
+
+    All other options are accepted and passed on to Pygments' lexer and formatter. Therefore,
+    valid options include any options which are accepted by the [html formatter] or
+    whichever [lexer] the code's language uses. Invalid options are ignored without error.
+
 A trivial example:
 
 ```python
 markdown.markdown(some_text, extensions=['codehilite'])
 ```
 
+[html formatter]: https://pygments.org/docs/formatters/#HtmlFormatter
+[lexer]: https://pygments.org/docs/lexers/
 [spec]: https://www.w3.org/TR/html5/text-level-semantics.html#the-code-element