Don't eat "pathless shebang lines" that could be actual Rust code

[timabbott]

Howdy! The following syntax has surprising results with Python-Markdown, because the markdown processor tries to hide the "shebang line without path":

#!test

The feature causing this problem is https://python-markdown.github.io/extensions/code_hilite/#shebang-no-path, which attempts to create an empty code block. I think that structurally you should be able to represent anything you like inside a code block, including syntax that could be a pathless shebang line, and thus the correct solution is to either remove the "shebang-no-path" feature, or at least make it optional (probably default-off), which makes the above string not appear properly in code blocks.

See https://johnmacfarlane.net/babelmark2/?text=%60%60%60%0A%23!test%0A%60%60%60%0A for what other implementations do, and zulip/zulip#18591 for an example user experience with this feature (the original reproducer was actual Rust code).

[facelessuser]

I've personally never been a huge fan of the shebang notation. I'm sure this was added before things like fenced code blocks were a thing (though I'll admit I don't actually know the history here).

The fenced_code extension uses CodeHilite for the code highlighting which pulls in this feature, which I personally think is only useful in indented code blocks (though I don't like it there either). Personally, I use SuperFences for code highlighting. It does not use CodeHilite to do the highlighting so it does not pick up this side effect. Also, SuperFences allows nested fenced code under other constructs like lists which the shipped fenced_code does not handle well.

With all of that said, it may be possible to somehow have the default fenced_code extension still use CodeHilite, but circumvent this behavior. We'd have to see if @waylan disagrees with adding such functionality as I'm not sure if this functionality was intended.

[waylan]

I'm sure this was added before things like fenced code blocks were a thing

Yes it was. Back when the feature was added, it was the only way one could define a language for a code block. We later created fenced code blocks. In fact, I believe that because fenced code blocks became a thing (and became popular), other ways of defining languages for code blocks never became popular.

That said, the existing fenced code block implementation is limited in that it can only be used for code blocks at the document root. In other words, code blocks nested within list items, blockquotes, etc. must still use indented code blocks. The shebang line is the only way those code blocks can have a language defined.

And we have a lot of users with thousands of existing (and possibly many years old) documents which are relying on the existing default behavior of the extension. Therefore, we will not be changing the default.

... with one exception. Checking for shebang lines (and removing pathless shebang lines) is not supposed to happen on fenced code blocks. Apparently some recent change must have unintentionally altered that behavior. So we do have a bug in this respect alone.

Finally, I would like to point out that this is not "default" behavior. The user must explicitly enable the extension to get this behavior. If the user does not like the behavior of the extension, the user is free to use a different extension.

[facelessuser]

Cool. If I get a chance soon, I may look into this bug.

[timabbott]

Thanks for the background! I agree with the assessment that turning off this feature for fenced code blocks would resolve the issue.

[waylan]

There are two additional changes that could be made to indented code blocks:

1. We could add a new config option to the extension which optionally disabled handing of shebang lines for indented code blocks. The default behavior should remain the same, but the user would have the option to turn the feature off.
2. We could skip removal of the shebang line if the entire code block only consists of that one line. It is reasonable to believe that noone will ever intentionally want an empty code block. On the other hand, the likelihood that that would be useful in the real word seems pretty low to me.

Personally, I don't think either is worth the effort. If the first (or only) line of a code block could be mistaken for a pathless shebang line, you should include an actual shebang line before it:

    #!rust
    #!test

In fact, that is probably what you want as it ensures the code block is properly identified as rust for syntax highlighting, In the case where the block consists only of #!test, then the block would be identified as the test language, which is most certainly not what you want, regardless of whether the line is removed or not.

[facelessuser]

My main plan is simply to ensure fenced code doesn't use the shebang. I'm not seeing anyone calling out for shebang support to be removed from indented, so I'm fine with that. I don't use it as I really only use fenced code blocks, but I know there are people who do, so I don't plan to mess with that.

[facelessuser]

@timabbott So I've taken a look at this, so I may need some clarification as I cannot reproduce this issue. Consider the following script:

import markdown

text = """
```
#!test
```
"""

parser = markdown.Markdown(extensions=['fenced_code'])

print(markdown.__version__)
print(parser.convert(text))

Notice in the output things are parsed as expected:

3.3.4
<pre><code>#!test
</code></pre>

I went back and looked at your example on Babelmark 2, and I noticed you are complaining about the handling, but the fenced_code extension is not enabled, so it is treating #!test as a header, not a shebang.

I would recommend turning on fenced_code (only works at root level) or try out SuperFences (which I maintain and works under nested lists etc.).

Fenced code is not part of the original Markdown spec, so it is often implemented via plugins. Some parsers may turn on such functionality by default in modern parsers, but Python Markdown does not.

If you are still having issues, please provide more information.

[waylan]

@facelessuser my understanding is that both fenced_code and codehilite extensions need to be enabled to get the reported behavior.

>>> print(markdown.markdown(text, extensions=['fenced_code', 'codehilite']))
<table class="codehilitetable"><tr><td class="linenos"><div class="linenodiv"><pre>1</pre></div></td><td class="code"><div class="code
hilite"><pre><span></span><code>
</code></pre></div>
</td></tr></table>

Note that the code block only contains one blank line. #!test is missing.

The bug is in the codehilite extension. The extension is supposed to skip shebang line parsing when the codeblock is a fenced code block. I believe is only skips shebang line parsing if a language is predefined. In fact, with this input, the bug does not occur:

```rust
#!test
```

I suppose one could argue that this is not a bug in that the user has failed to specify the language. However, the intention was always to only parse shebang lines for indented code blocks. Regardless, I consider this a low priority as the user can always work around it by specifying a language.

[facelessuser]

Ah, thanks for the clarification. For some reason I thought Codehilite was implicitly used by fenced code. I ran through this too quickly. I'll take another look.