HTML error with video elements (after version 3.2.2)

[jorgesumle]

I was using version 3.2.2, but after the upgrade I get HTML validation errors with video elements:

'"file:/var/www/html/deponejo/Freak-Spot/output/locutus-de-gnu/index.html":321.4-321.120: error: Unclosed element "video".', '"file:/var/www/html/deponejo/Freak-Spot/output/locutus-de-gnu/index.html":325.45-325.48: error: No "p" element in scope but a "p" end tag seen.', '"file:/var/www/html/deponejo/Freak-Spot/output/locutus-de-gnu/index.html":326.1-326.8: error: Stray end tag "video".'

Input

<video controls poster="https://freakspot.net/wp-content/uploads/2017/01/Bildschirmfoto-vom-2017-02-01-00-58-04.png">
  <source src="https://b2aeaa58a57a200320db-8b65b95250e902c437b256b5abf3eac7.ssl.cf5.rackcdn.com/media_entries/12965/A0001-4030.medium.webm" type="video/webm">
  <p>Lo siento, tu navegador no soporta HTML 5. Por
  favor, cambia o actualiza tu navegador</p>
</video>

Output

<p><video controls poster="https://freakspot.net/wp-content/uploads/2017/01/Bildschirmfoto-vom-2017-02-01-00-58-04.png">
  <source src="https://b2aeaa58a57a200320db-8b65b95250e902c437b256b5abf3eac7.ssl.cf5.rackcdn.com/media_entries/12965/A0001-4030.medium.webm" type="video/webm">
<br>
<p>Lo siento, tu navegador no soporta HTML 5. Por
  favor, cambia o actualiza tu navegador</p></p>
</video>

[facelessuser]

I imagine this may be an issue caused by the fact that it is treated as an inline element but you are embedding block elements within it.

[waylan]

I'm not able to replicate this behavior.

>>> import markdown
>>> markdown.version
'3.3.2'
>>> t = """
... <video controls poster="https://freakspot.net/wp-content/uploads/2017/01/Bildschirmfoto-vom-2017-02-01-00-58-04.png">
...   <source src="https://b2aeaa58a57a200320db-8b65b95250e902c437b256b5abf3eac7.ssl.cf5.rackcdn.com/media_entries/12965/A0001-4030.medium.webm" type="video/webm">
...   <p>Lo siento, tu navegador no soporta HTML 5. Por
...   favor, cambia o actualiza tu navegador</p>
... </video>
... """
>>> print(markdown.markdown(t))
<video controls poster="https://freakspot.net/wp-content/uploads/2017/01/Bildschirmfoto-vom-2017-02-01-00-58-04.png">
  <source src="https://b2aeaa58a57a200320db-8b65b95250e902c437b256b5abf3eac7.ssl.cf5.rackcdn.com/media_entries/12965/A0001-4030.medium.webm" type="video/webm">
  <p>Lo siento, tu navegador no soporta HTML 5. Por
  favor, cambia o actualiza tu navegador</p>
</video>

[waylan]

Oh, it requires the md_in_html extension, which wasn't mentioned:

>>> print(markdown.markdown(t, extensions=['md_in_html']))
<p><video controls poster="https://freakspot.net/wp-content/uploads/2017/01/Bildschirmfoto-vom-2017-02-01-00-58-04.png">
  <source src="https://b2aeaa58a57a200320db-8b65b95250e902c437b256b5abf3eac7.ssl.cf5.rackcdn.com/media_entries/12965/A0001-4030.medium.webm" type="video/webm">
<br />
<p>Lo siento, tu navegador no soporta HTML 5. Por
  favor, cambia o actualiza tu navegador</p></p>
</video>

By the way, I just noticed that some block level elements in the core are not included in the extension. Presumably we simply need to expand the list to ensure nothing is missing. Unfortunately, we can't just reuse the list in the core as we need them broken up into subcategories. Regardless, that should be an easy fix.

[facelessuser]

Is there a problem with block tags? As far as I'm aware, video are inline in the HTML spec.

[facelessuser]

I guess, at the very least, the inconsistent handling between md_in_html and normal parsing is a bug.

[waylan]

Comparing the tags listed in markdown.util.BLOCK_LEVEL_ELEMENTS with markdown.extensions.md_in_html.block_level_tags we have the following differences:

   'address',    
+  'address',    
   'article',    
   'aside',      
   'blockquote', 
+  'body',       
   'canvas',     
+  'colgroup',   
   'dd',         
   'details',    
   'div',        
   'dl',         
   'dt',         
   'fieldset',   
   'figcaption', 
   'figure',     
   'footer',     
   'form',       
-  'group',      
   'h1',         
   'h2',         
   'h3',         
   'h4',         
   'h5',         
   'h6',         
   'header',     
   'hr',         
   'iframe',     
+  'legend',     
   'li',         
   'main',       
+  'map',        
   'math',       
   'menu',       
   'nav',        
   'noscript',   
+  'object',     
   'ol',         
-  'output',     
+  'option',     
   'p',          
   'pre',        
-  'progress',   
   'script',     
   'section',    
   'style',      
   'table',      
   'tbody',      
   'td',         
+  'textarea',   
+  'tfoot',      
   'th',         
   'thead',      
   'tr',         
   'ul',         
-  'video'

1. The address tag is included twice in md_in_html; once in span_tags and once in block_tags. Opps. But which one should it be?
2. The following tags are missing completely from md_in_html: 'group', 'output', 'video'. They need to be added, but where?
3. The following tags are listed in md_in_html but not in the core: 'body', 'colgroup', 'legend', 'map', 'object', 'option', 'testarea', 'tfoot'. Should they be added to the core? Perhaps some of them should be removed from md_in_html?

[facelessuser]

@waylan, I'm confused, as a plugin can add new items to the block tag list, shouldn't this list be shared?

[facelessuser]

Personally, I'd have the md_in_html parser just use the common one via something like self.md.block_level_tags.

[facelessuser]

Oh, there are 3 lists of block level tags now. One in util, one in core, and one in md_in_html. I would use the one in core as it has been available for some time for people to use to adjust block level tags.

EDIT: corrected, I would use the one in core.

[waylan]

@waylan, I'm confused, as a plugin can add new items to the block tag list, shouldn't this list be shared?

Normally, yes. However, block_level_tags is defined as span_tags + block_tags + raw_tags. The extension needs the tags to be defined in those three groups to work correctly.

markdown/markdown/extensions/md_in_html.py

Lines 26 to 39 in 2766698

	# Block-level tags in which the content only gets span level parsing
	span_tags = ['address', 'dd', 'dt', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'legend', 'li', 'p', 'td', 'th']
	# Block-level tags in which the content gets parsed as blocks
	block_tags = [
	'address', 'article', 'aside', 'blockquote', 'body', 'colgroup', 'details', 'div', 'dl', 'fieldset',
	'figcaption', 'figure', 'footer', 'form', 'iframe', 'header', 'hr', 'main', 'menu', 'nav', 'map',
	'noscript', 'object', 'ol', 'section', 'table', 'tbody', 'thead', 'tfoot', 'tr', 'ul'
	]
	# Block-level tags which never get their content parsed.
	raw_tags = ['canvas', 'math', 'option', 'pre', 'script', 'style', 'textarea']
	block_level_tags = span_tags + block_tags + raw_tags

For a full explanation of why that matters, the see the documentation for the differences between markdown="1", markdown="block" and markdown="span".

[facelessuser]

Then where does the core list fit into things?

[facelessuser]

https://github.com/Python-Markdown/markdown/blob/master/markdown/core.py#L75-L85

I'd hate to lose the ability to adjust this list.