cpython-lru_cache-weakref: Tools/scripts/highlight.py annotate

annotate Tools/scripts/highlight.py @ 87690:66c1c9f32567 default tip

Add use_weakref option to functools.lru_cache The use_weakref option makes lru_cache create weak references to the function arguments when creating the cache key, which prevents them from staying in the memory forever even after all other references to them are gone. This is a proof of concept for http://bugs.python.org/issue19859

author	Radomir Dopieralski <devel@sheep.art.pl>
date	Tue, 03 Dec 2013 16:02:48 +0100
parents	b127046831e2
children

rev	line source
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	1 #!/usr/bin/env python3
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	2 '''Add syntax highlighting to Python source code'''
77942 1401ff261ced Add examples to the module docstring Raymond Hettinger <python@rcn.com> parents: 77941 diff changeset	3
77903 21ca7f56e12c Set title to the source filename Raymond Hettinger <python@rcn.com> parents: 77902 diff changeset	4 __author__ = 'Raymond Hettinger'
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	5
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	6 import keyword, tokenize, cgi, re, functools
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	7 try:
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	8 import builtins
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	9 except ImportError:
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	10 import __builtin__ as builtins
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	11
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	12 #### Analyze Python Source #################################
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	13
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	14 def is_builtin(s):
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	15 'Return True if s is the name of a builtin'
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	16 return hasattr(builtins, s)
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	17
77938 1f06fe5cd4da Refactor to isolate HTML encoding step from the parsing step. Raymond Hettinger <python@rcn.com> parents: 77929 diff changeset	18 def combine_range(lines, start, end):
1f06fe5cd4da Refactor to isolate HTML encoding step from the parsing step. Raymond Hettinger <python@rcn.com> parents: 77929 diff changeset	19 'Join content from a range of lines between start and end'
77918 671894ae19a2 Do HTML escaping after the tokenization step. Raymond Hettinger <python@rcn.com> parents: 77904 diff changeset	20 (srow, scol), (erow, ecol) = start, end
671894ae19a2 Do HTML escaping after the tokenization step. Raymond Hettinger <python@rcn.com> parents: 77904 diff changeset	21 if srow == erow:
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	22 return lines[srow-1][scol:ecol], end
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	23 rows = [lines[srow-1][scol:]] + lines[srow: erow-1] + [lines[erow-1][:ecol]]
77938 1f06fe5cd4da Refactor to isolate HTML encoding step from the parsing step. Raymond Hettinger <python@rcn.com> parents: 77929 diff changeset	24 return ''.join(rows), end
77918 671894ae19a2 Do HTML escaping after the tokenization step. Raymond Hettinger <python@rcn.com> parents: 77904 diff changeset	25
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	26 def analyze_python(source):
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	27 '''Generate and classify chunks of Python for syntax highlighting.
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	28 Yields tuples in the form: (category, categorized_text).
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	29 '''
77929 d7bed032520b Fix keyword argument Raymond Hettinger <python@rcn.com> parents: 77928 diff changeset	30 lines = source.splitlines(True)
77918 671894ae19a2 Do HTML escaping after the tokenization step. Raymond Hettinger <python@rcn.com> parents: 77904 diff changeset	31 lines.append('')
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	32 readline = functools.partial(next, iter(lines), '')
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	33 kind = tok_str = ''
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	34 tok_type = tokenize.COMMENT
77918 671894ae19a2 Do HTML escaping after the tokenization step. Raymond Hettinger <python@rcn.com> parents: 77904 diff changeset	35 written = (1, 0)
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	36 for tok in tokenize.generate_tokens(readline):
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	37 prev_tok_type, prev_tok_str = tok_type, tok_str
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	38 tok_type, tok_str, (srow, scol), (erow, ecol), logical_lineno = tok
77928 4d320cd4c6fe Minor cleanups Raymond Hettinger <python@rcn.com> parents: 77920 diff changeset	39 kind = ''
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	40 if tok_type == tokenize.COMMENT:
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	41 kind = 'comment'
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	42 elif tok_type == tokenize.OP and tok_str[:1] not in '{}[](),.:;@':
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	43 kind = 'operator'
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	44 elif tok_type == tokenize.STRING:
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	45 kind = 'string'
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	46 if prev_tok_type == tokenize.INDENT or scol==0:
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	47 kind = 'docstring'
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	48 elif tok_type == tokenize.NAME:
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	49 if tok_str in ('def', 'class', 'import', 'from'):
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	50 kind = 'definition'
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	51 elif prev_tok_str in ('def', 'class'):
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	52 kind = 'defname'
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	53 elif keyword.iskeyword(tok_str):
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	54 kind = 'keyword'
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	55 elif is_builtin(tok_str) and prev_tok_str != '.':
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	56 kind = 'builtin'
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	57 if kind:
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	58 text, written = combine_range(lines, written, (srow, scol))
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	59 yield '', text
78253 b127046831e2 Simplify the LaTeX section (only three escapes are needed for alltt) Raymond Hettinger <python@rcn.com> parents: 78103 diff changeset	60 text, written = tok_str, (erow, ecol)
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	61 yield kind, text
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	62 line_upto_token, written = combine_range(lines, written, (erow, ecol))
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	63 yield '', line_upto_token
77938 1f06fe5cd4da Refactor to isolate HTML encoding step from the parsing step. Raymond Hettinger <python@rcn.com> parents: 77929 diff changeset	64
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	65 #### Raw Output ###########################################
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	66
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	67 def raw_highlight(classified_text):
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	68 'Straight text display of text classifications'
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	69 result = []
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	70 for kind, text in classified_text:
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	71 result.append('%15s: %r\n' % (kind or 'plain', text))
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	72 return ''.join(result)
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	73
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	74 #### ANSI Output ###########################################
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	75
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	76 default_ansi = {
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	77 'comment': ('\033[0;31m', '\033[0m'),
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	78 'string': ('\033[0;32m', '\033[0m'),
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	79 'docstring': ('\033[0;32m', '\033[0m'),
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	80 'keyword': ('\033[0;33m', '\033[0m'),
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	81 'builtin': ('\033[0;35m', '\033[0m'),
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	82 'definition': ('\033[0;33m', '\033[0m'),
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	83 'defname': ('\033[0;34m', '\033[0m'),
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	84 'operator': ('\033[0;33m', '\033[0m'),
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	85 }
980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	86
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	87 def ansi_highlight(classified_text, colors=default_ansi):
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	88 'Add syntax highlighting to source code using ANSI escape sequences'
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	89 # http://en.wikipedia.org/wiki/ANSI_escape_code
980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	90 result = []
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	91 for kind, text in classified_text:
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	92 opener, closer = colors.get(kind, ('', ''))
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	93 result += [opener, text, closer]
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	94 return ''.join(result)
980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	95
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	96 #### HTML Output ###########################################
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	97
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	98 def html_highlight(classified_text,opener='<pre class="python">\n', closer='</pre>\n'):
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	99 'Convert classified text to an HTML fragment'
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	100 result = [opener]
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	101 for kind, text in classified_text:
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	102 if kind:
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	103 result.append('<span class="%s">' % kind)
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	104 result.append(cgi.escape(text))
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	105 if kind:
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	106 result.append('</span>')
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	107 result.append(closer)
77918 671894ae19a2 Do HTML escaping after the tokenization step. Raymond Hettinger <python@rcn.com> parents: 77904 diff changeset	108 return ''.join(result)
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	109
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	110 default_css = {
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	111 '.comment': '{color: crimson;}',
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	112 '.string': '{color: forestgreen;}',
77938 1f06fe5cd4da Refactor to isolate HTML encoding step from the parsing step. Raymond Hettinger <python@rcn.com> parents: 77929 diff changeset	113 '.docstring': '{color: forestgreen; font-style:italic;}',
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	114 '.keyword': '{color: darkorange;}',
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	115 '.builtin': '{color: purple;}',
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	116 '.definition': '{color: darkorange; font-weight:bold;}',
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	117 '.defname': '{color: blue;}',
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	118 '.operator': '{color: brown;}',
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	119 }
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	120
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	121 default_html = '''\
77902 d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	122 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	123 "http://www.w3.org/TR/html4/strict.dtd">
d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	124 <html>
d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	125 <head>
d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	126 <meta http-equiv="Content-type" content="text/html;charset=UTF-8">
77920 2367ec12da2d Use new-style string formatting for the HTML template Raymond Hettinger <python@rcn.com> parents: 77919 diff changeset	127 <title> {title} </title>
77902 d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	128 <style type="text/css">
77920 2367ec12da2d Use new-style string formatting for the HTML template Raymond Hettinger <python@rcn.com> parents: 77919 diff changeset	129 {css}
77902 d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	130 </style>
d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	131 </head>
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	132 <body>
77920 2367ec12da2d Use new-style string formatting for the HTML template Raymond Hettinger <python@rcn.com> parents: 77919 diff changeset	133 {body}
77902 d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	134 </body>
d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	135 </html>
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	136 '''
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	137
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	138 def build_html_page(classified_text, title='python',
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	139 css=default_css, html=default_html):
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	140 'Create a complete HTML page with colorized source code'
77902 d02588a26057 Small cleanups Raymond Hettinger <python@rcn.com> parents: 77901 diff changeset	141 css_str = '\n'.join(['%s %s' % item for item in css.items()])
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	142 result = html_highlight(classified_text)
77903 21ca7f56e12c Set title to the source filename Raymond Hettinger <python@rcn.com> parents: 77902 diff changeset	143 title = cgi.escape(title)
77920 2367ec12da2d Use new-style string formatting for the HTML template Raymond Hettinger <python@rcn.com> parents: 77919 diff changeset	144 return html.format(title=title, css=css_str, body=result)
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	145
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	146 #### LaTeX Output ##########################################
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	147
78103 592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	148 default_latex_commands = {
592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	149 'comment': '{\color{red}#1}',
592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	150 'string': '{\color{ForestGreen}#1}',
592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	151 'docstring': '{\emph{\color{ForestGreen}#1}}',
592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	152 'keyword': '{\color{orange}#1}',
592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	153 'builtin': '{\color{purple}#1}',
592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	154 'definition': '{\color{orange}#1}',
592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	155 'defname': '{\color{blue}#1}',
592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	156 'operator': '{\color{brown}#1}',
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	157 }
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	158
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	159 default_latex_document = r'''
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	160 \documentclass{article}
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	161 \usepackage{alltt}
78103 592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	162 \usepackage{upquote}
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	163 \usepackage{color}
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	164 \usepackage[usenames,dvipsnames]{xcolor}
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	165 \usepackage[cm]{fullpage}
78103 592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	166 %(macros)s
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	167 \begin{document}
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	168 \center{\LARGE{%(title)s}}
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	169 \begin{alltt}
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	170 %(body)s
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	171 \end{alltt}
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	172 \end{document}
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	173 '''
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	174
78253 b127046831e2 Simplify the LaTeX section (only three escapes are needed for alltt) Raymond Hettinger <python@rcn.com> parents: 78103 diff changeset	175 def alltt_escape(s):
b127046831e2 Simplify the LaTeX section (only three escapes are needed for alltt) Raymond Hettinger <python@rcn.com> parents: 78103 diff changeset	176 'Replace backslash and braces with their escaped equivalents'
b127046831e2 Simplify the LaTeX section (only three escapes are needed for alltt) Raymond Hettinger <python@rcn.com> parents: 78103 diff changeset	177 xlat = {'{': r'\{', '}': r'\}', '\\': r'\textbackslash{}'}
b127046831e2 Simplify the LaTeX section (only three escapes are needed for alltt) Raymond Hettinger <python@rcn.com> parents: 78103 diff changeset	178 return re.sub(r'[\\{}]', lambda mo: xlat[mo.group()], s)
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	179
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	180 def latex_highlight(classified_text, title = 'python',
78103 592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	181 commands = default_latex_commands,
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	182 document = default_latex_document):
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	183 'Create a complete LaTeX document with colorized source code'
78103 592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	184 macros = '\n'.join(r'\newcommand{\py%s}[1]{%s}' % c for c in commands.items())
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	185 result = []
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	186 for kind, text in classified_text:
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	187 if kind:
78103 592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	188 result.append(r'\py%s{' % kind)
78253 b127046831e2 Simplify the LaTeX section (only three escapes are needed for alltt) Raymond Hettinger <python@rcn.com> parents: 78103 diff changeset	189 result.append(alltt_escape(text))
78079 425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	190 if kind:
425d5e89fa25 Fix builtin test and simplify the classified text tuple. Raymond Hettinger <python@rcn.com> parents: 78049 diff changeset	191 result.append('}')
78103 592307606f51 Fix-up latex styling Raymond Hettinger <python@rcn.com> parents: 78079 diff changeset	192 return default_latex_document % dict(title=title, macros=macros, body=''.join(result))
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	193
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	194
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	195 if __name__ == '__main__':
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	196 import sys, argparse, webbrowser, os, textwrap
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	197
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	198 parser = argparse.ArgumentParser(
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	199 description = 'Add syntax highlighting to Python source code',
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	200 formatter_class=argparse.RawDescriptionHelpFormatter,
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	201 epilog = textwrap.dedent('''
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	202 examples:
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	203
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	204 # Show syntax highlighted code in the terminal window
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	205 $ ./highlight.py myfile.py
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	206
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	207 # Colorize myfile.py and display in a browser
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	208 $ ./highlight.py -b myfile.py
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	209
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	210 # Create an HTML section to embed in an existing webpage
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	211 ./highlight.py -s myfile.py
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	212
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	213 # Create a complete HTML file
77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	214 $ ./highlight.py -c myfile.py > myfile.html
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	215
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	216 # Create a PDF using LaTeX
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	217 $ ./highlight.py -l myfile.py \| pdflatex
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	218
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	219 '''))
77928 4d320cd4c6fe Minor cleanups Raymond Hettinger <python@rcn.com> parents: 77920 diff changeset	220 parser.add_argument('sourcefile', metavar = 'SOURCEFILE',
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	221 help = 'file containing Python sourcecode')
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	222 parser.add_argument('-b', '--browser', action = 'store_true',
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	223 help = 'launch a browser to show results')
77943 9fe720fe6057 Make ANSI the default output style Raymond Hettinger <python@rcn.com> parents: 77942 diff changeset	224 parser.add_argument('-c', '--complete', action = 'store_true',
9fe720fe6057 Make ANSI the default output style Raymond Hettinger <python@rcn.com> parents: 77942 diff changeset	225 help = 'build a complete html webpage')
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	226 parser.add_argument('-l', '--latex', action = 'store_true',
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	227 help = 'build a LaTeX document')
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	228 parser.add_argument('-r', '--raw', action = 'store_true',
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	229 help = 'raw parse of categorized text')
77928 4d320cd4c6fe Minor cleanups Raymond Hettinger <python@rcn.com> parents: 77920 diff changeset	230 parser.add_argument('-s', '--section', action = 'store_true',
4d320cd4c6fe Minor cleanups Raymond Hettinger <python@rcn.com> parents: 77920 diff changeset	231 help = 'show an HTML section rather than a complete webpage')
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	232 args = parser.parse_args()
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	233
77943 9fe720fe6057 Make ANSI the default output style Raymond Hettinger <python@rcn.com> parents: 77942 diff changeset	234 if args.section and (args.browser or args.complete):
77928 4d320cd4c6fe Minor cleanups Raymond Hettinger <python@rcn.com> parents: 77920 diff changeset	235 parser.error('The -s/--section option is incompatible with '
77943 9fe720fe6057 Make ANSI the default output style Raymond Hettinger <python@rcn.com> parents: 77942 diff changeset	236 'the -b/--browser or -c/--complete options')
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	237
77928 4d320cd4c6fe Minor cleanups Raymond Hettinger <python@rcn.com> parents: 77920 diff changeset	238 sourcefile = args.sourcefile
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	239 with open(sourcefile) as f:
77942 1401ff261ced Add examples to the module docstring Raymond Hettinger <python@rcn.com> parents: 77941 diff changeset	240 source = f.read()
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	241 classified_text = analyze_python(source)
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	242
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	243 if args.raw:
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	244 encoded = raw_highlight(classified_text)
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	245 elif args.complete or args.browser:
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	246 encoded = build_html_page(classified_text, title=sourcefile)
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	247 elif args.section:
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	248 encoded = html_highlight(classified_text)
78049 08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	249 elif args.latex:
08bfd88027bb Add simple LaTeX markup Raymond Hettinger <python@rcn.com> parents: 78021 diff changeset	250 encoded = latex_highlight(classified_text, title=sourcefile)
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	251 else:
78014 77ca62751877 Minor algorithmic tweaks, better factoring, improved docstrings and variable names Raymond Hettinger <python@rcn.com> parents: 77944 diff changeset	252 encoded = ansi_highlight(classified_text)
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	253
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	254 if args.browser:
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	255 htmlfile = os.path.splitext(os.path.basename(sourcefile))[0] + '.html'
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	256 with open(htmlfile, 'w') as f:
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	257 f.write(encoded)
77899 da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	258 webbrowser.open('file://' + os.path.abspath(htmlfile))
da4dd603030b Add syntax highlighter tool Raymond Hettinger <python@rcn.com> parents: diff changeset	259 else:
77940 980de5b47d0d Add support for ANSI escape sequences Raymond Hettinger <python@rcn.com> parents: 77938 diff changeset	260 sys.stdout.write(encoded)

Mercurial > cpython-lru_cache-weakref

annotate Tools/scripts/highlight.py @ 87690:66c1c9f32567 default tip