Fixed #34140 blacken-docs #16496

jvzammit · 2023-01-23T21:28:49Z

Two commits

Apply changes to have blacken-docs run without errors
Automated changes applied by blacken-docs

Files manually modified in this PR for blacken-docs --rst-literal-blocks to run without issue:

ref/applications.txt
ref/clickjacking.txt
ref/databases.txt
ref/django-admin.txt
ref/logging.txt
ref/request-response.txt
ref/settings.txt
ref/signals.txt
ref/template-response.txt
ref/unicode.txt
ref/urlresolvers.txt
ref/utils.txt

I know this is not the complete thing. But it's still time consuming enough to justify creating a PR. And get some reviews on some manual decisions I have taken in getting the documentation files able to be handled by blacken-docs without errors.

For completeness' sake, these are the "previous" errors before commit 1 in this PR:

docs/ref/urlresolvers.txt:39: code block parse error Cannot parse: 1:0: >>> reverse('admin:app_list', kwargs={'app_label': 'auth'})
docs/ref/urlresolvers.txt:68: code block parse error Cannot parse: 1:0: >>> reverse('cities', args=['Orléans'])
docs/ref/clickjacking.txt:58: code block parse error Cannot parse: 3:4:     'django.middleware.clickjacking.XFrameOptionsMiddleware',
docs/ref/signals.txt:330: code block parse error Cannot parse: 1:0: >>> p = Pizza.objects.create(...)
docs/ref/signals.txt:360: code block parse error Cannot parse: 1:0: >>> t.pizza_set.remove(p)
docs/ref/unicode.txt:177: code block parse error Cannot parse: 1:0: >>> from urllib.parse import quote
docs/ref/unicode.txt:196: code block parse error Cannot parse: 1:0: >>> from django.utils.encoding import uri_to_iri
docs/ref/unicode.txt:312: code block parse error Cannot parse: 1:7: export LANG="en_US.UTF-8"
docs/ref/settings.txt:1719: code block parse error Cannot parse: 1:7: mysite/
docs/ref/applications.txt:12: code block parse error Cannot parse: 1:0: >>> from django.apps import apps
docs/ref/applications.txt:74: code block parse error Cannot parse: 3:4:     'polls.apps.PollsAppConfig',
docs/ref/utils.txt:301: code block parse error Cannot parse: 1:0: >>> from django.utils import feedgenerator
docs/ref/utils.txt:541: code block parse error Cannot parse: 3:30: def fancy_utility_function(s, ...):
docs/ref/utils.txt:571: code block parse error Cannot parse: 5:30: def fancy_utility_function(s, ...):
docs/ref/utils.txt:663: code block parse error Cannot parse: 1:0: >> json_script({"hello": "world"}, element_id="hello-data")
docs/ref/utils.txt:800: code block parse error Cannot parse: 1:0: >>> mystr = '<b>Hello World</b>   '
docs/ref/utils.txt:846: code block parse error Cannot parse: 1:0: >>> slugify(' Joel is a slug ')
docs/ref/utils.txt:852: code block parse error Cannot parse: 1:0: >>> slugify('你好 World', allow_unicode=True)
docs/ref/request-response.txt:182: code block parse error Cannot parse: 1:0: >>> request.headers
docs/ref/request-response.txt:384: code block parse error Cannot parse: 1:0: >>> request.get_signed_cookie('name')
docs/ref/request-response.txt:414: code block parse error Cannot parse: 1:0: >>> request.accepts('text/html')
docs/ref/request-response.txt:495: code block parse error Cannot parse: 1:0: >>> QueryDict.fromkeys(['a', 'a', 'b'], value='val')
docs/ref/request-response.txt:533: code block parse error Cannot parse: 1:0: >>> q = QueryDict('a=1', mutable=True)
docs/ref/request-response.txt:546: code block parse error Cannot parse: 1:0: >>> q = QueryDict('a=1&a=2&a=3')
docs/ref/request-response.txt:556: code block parse error Cannot parse: 1:0: >>> q = QueryDict('a=1&a=2&a=3')
docs/ref/request-response.txt:591: code block parse error Cannot parse: 1:0: >>> q = QueryDict('a=1&a=2&a=3')
docs/ref/request-response.txt:600: code block parse error Cannot parse: 1:0: >>> q = QueryDict('a=1&a=2&a=3', mutable=True)
docs/ref/request-response.txt:611: code block parse error Cannot parse: 1:0: >>> q = QueryDict('a=1&a=2&a=3', mutable=True)
docs/ref/request-response.txt:621: code block parse error Cannot parse: 1:0: >>> q = QueryDict('a=1&a=3&a=5')
docs/ref/request-response.txt:629: code block parse error Cannot parse: 1:0: >>> q = QueryDict('a=2&b=3&b=5')
docs/ref/request-response.txt:636: code block parse error Cannot parse: 1:0: >>> q = QueryDict(mutable=True)
docs/ref/request-response.txt:662: code block parse error Cannot parse: 1:0: >>> from django.http import HttpResponse
docs/ref/request-response.txt:671: code block parse error Cannot parse: 1:0: >>> response = HttpResponse()
docs/ref/request-response.txt:694: code block parse error Cannot parse: 1:0: >>> response = HttpResponse()
docs/ref/request-response.txt:700: code block parse error Cannot parse: 1:0: >>> response = HttpResponse()
docs/ref/request-response.txt:712: code block parse error Cannot parse: 1:0: >>> response = HttpResponse(headers={'Age': 120})
docs/ref/request-response.txt:731: code block parse error Cannot parse: 1:0: >>> response = HttpResponse(my_data, headers={
docs/ref/request-response.txt:1072: code block parse error Cannot parse: 1:0: >>> from django.http import JsonResponse
docs/ref/request-response.txt:1083: code block parse error Cannot parse: 1:0: >>> response = JsonResponse([1, 2, 3], safe=False)
docs/ref/request-response.txt:1107: code block parse error Cannot parse: 1:0: >>> response = JsonResponse(data, encoder=MyJSONEncoder)
docs/ref/request-response.txt:1209: code block parse error Cannot parse: 1:0: >>> from django.http import FileResponse
docs/ref/template-response.txt:221: code block parse error Cannot parse: 2:0: >>> from django.template.response import TemplateResponse
docs/ref/logging.txt:229: code block parse error Cannot parse: 5:1: },
docs/ref/logging.txt:281: code block parse error Cannot parse: 7:1: },
docs/ref/logging.txt:296: code block parse error Cannot parse: 7:1: },
docs/ref/logging.txt:312: code block parse error Cannot parse: 8:1: },
docs/ref/logging.txt:355: code block parse error Cannot parse: 6:1: },
docs/ref/logging.txt:377: code block parse error Cannot parse: 5:1: },
docs/ref/django-admin.txt:68: code block parse error Cannot parse: 2:3: 1.4a1
docs/ref/django-admin.txt:94: code block parse error Cannot parse: 1:13: django-admin check auth admin myapp
docs/ref/django-admin.txt:103: code block parse error Cannot parse: 1:13: django-admin check --tag models --tag compatibility
docs/ref/django-admin.txt:109: code block parse error Cannot parse: 1:13: django-admin check --database default --database other
docs/ref/django-admin.txt:127: code block parse error Cannot parse: 1:13: django-admin check --deploy --settings=production_settings
docs/ref/django-admin.txt:161: code block parse error Cannot parse: 1:13: django-admin compilemessages --locale=pt_BR
docs/ref/django-admin.txt:179: code block parse error Cannot parse: 1:13: django-admin compilemessages --ignore=cache --ignore=outdated/*/locale
docs/ref/django-admin.txt:332: code block parse error Cannot parse: 1:13: django-admin dumpdata --exclude=auth --exclude=contenttypes
docs/ref/django-admin.txt:373: code block parse error Cannot parse: 1:13: django-admin dumpdata -o mydata.json.gz
docs/ref/django-admin.txt:523: code block parse error Cannot parse: 1:13: django-admin loaddata --format=json -
docs/ref/django-admin.txt:532: code block parse error Cannot parse: 1:13: django-admin dumpdata --format=json --database=test app_label.ModelName | django-admin loaddata --format=json --database=prod -
docs/ref/django-admin.txt:567: code block parse error Cannot parse: 1:13: django-admin makemessages --locale=de --extension xhtml
docs/ref/django-admin.txt:572: code block parse error Cannot parse: 1:13: django-admin makemessages --locale=de --extension=html,txt --extension xml
docs/ref/django-admin.txt:585: code block parse error Cannot parse: 1:13: django-admin makemessages --locale=pt_BR
docs/ref/django-admin.txt:607: code block parse error Cannot parse: 1:13: django-admin makemessages --locale=de --symlinks
docs/ref/django-admin.txt:618: code block parse error Cannot parse: 1:13: django-admin makemessages --locale=en_US --ignore=apps/* --ignore=secret/*.html
docs/ref/django-admin.txt:928: code block parse error Cannot parse: 1:13: django-admin runserver
docs/ref/django-admin.txt:932: code block parse error Cannot parse: 1:13: django-admin runserver 1.2.3.4:8000
docs/ref/django-admin.txt:936: code block parse error Cannot parse: 1:13: django-admin runserver 7000
docs/ref/django-admin.txt:940: code block parse error Cannot parse: 1:13: django-admin runserver 1.2.3.4:7000
docs/ref/django-admin.txt:944: code block parse error Cannot parse: 1:13: django-admin runserver -6
docs/ref/django-admin.txt:948: code block parse error Cannot parse: 1:13: django-admin runserver -6 7000
docs/ref/django-admin.txt:952: code block parse error Cannot parse: 1:13: django-admin runserver [2001:0db8:1234:5678::9]:7000
docs/ref/django-admin.txt:956: code block parse error Cannot parse: 1:13: django-admin runserver localhost:8000
docs/ref/django-admin.txt:960: code block parse error Cannot parse: 1:13: django-admin runserver -6 localhost:8000
docs/ref/django-admin.txt:978: code block parse error Cannot parse: 1:13: django-admin sendtestemail foo@example.com bar@example.com
docs/ref/django-admin.txt:1007: code block parse error Cannot parse: 1:13: django-admin shell -i ipython
docs/ref/django-admin.txt:1011: code block parse error Cannot parse: 1:13: django-admin shell -i bpython
docs/ref/django-admin.txt:1016: code block parse error Cannot parse: 1:13: django-admin shell -i python
docs/ref/django-admin.txt:1031: code block parse error Cannot parse: 1:13: django-admin shell --command="import django; print(django.__version__)"
docs/ref/django-admin.txt:1180: code block parse error Cannot parse: 1:13: django-admin startapp myapp /Users/jezdez/Code/myapp
docs/ref/django-admin.txt:1194: code block parse error Cannot parse: 1:13: django-admin startapp --template=/Users/jezdez/Code/my_app_template myapp
docs/ref/django-admin.txt:1203: code block parse error Cannot parse: 1:13: django-admin startapp --template=https://github.com/githubuser/django-app-template/archive/main.zip myapp
docs/ref/django-admin.txt:1286: code block parse error Cannot parse: 1:13: django-admin startproject myproject /Users/jezdez/Code/myproject_repo
docs/ref/django-admin.txt:1495: code block parse error Cannot parse: 1:13: django-admin testserver mydata.json
docs/ref/django-admin.txt:1533: code block parse error Cannot parse: 1:13: django-admin testserver --addrport 7000 fixture1 fixture2
docs/ref/django-admin.txt:1542: code block parse error Cannot parse: 1:13: django-admin testserver --addrport 1.2.3.4:7000 test
docs/ref/django-admin.txt:1579: code block parse error Cannot parse: 1:13: django-admin changepassword ringo
docs/ref/django-admin.txt:1733: code block parse error Cannot parse: 1:13: django-admin migrate --pythonpath='/home/djangoprojects/myproject'
docs/ref/django-admin.txt:1749: code block parse error Cannot parse: 1:13: django-admin migrate --settings=mysite.settings
docs/ref/django-admin.txt:1761: code block parse error Cannot parse: 1:13: django-admin migrate --traceback
docs/ref/django-admin.txt:1777: code block parse error Cannot parse: 1:13: django-admin migrate --verbosity 2
docs/ref/django-admin.txt:1787: code block parse error Cannot parse: 1:13: django-admin runserver --no-color
docs/ref/django-admin.txt:1804: code block parse error Cannot parse: 1:13: django-admin migrate --skip-checks
docs/ref/django-admin.txt:1837: code block parse error Cannot parse: 1:3: ...\> py -m pip install colorama
docs/ref/django-admin.txt:1872: code block parse error Cannot parse: 1:7: export DJANGO_COLORS="light"
docs/ref/django-admin.txt:1928: code block parse error Cannot parse: 1:7: export DJANGO_COLORS="error=yellow/blue,blink;notice=magenta"
docs/ref/django-admin.txt:1938: code block parse error Cannot parse: 1:7: export DJANGO_COLORS="light;error=yellow/blue,blink;notice=magenta"
docs/ref/django-admin.txt:1972: code block parse error Cannot parse: 1:22: PATH=path/to/venv/bin django-admin makemigrations
docs/ref/django-admin.txt:1976: code block parse error Cannot parse: 1:13: django-admin inspectdb | black -
docs/ref/databases.txt:298: code block parse error Cannot parse: 1:0: >>> from django.contrib.auth.models import User
docs/ref/databases.txt:443: code block parse error Cannot parse: 1:7: CREATE DATABASE <dbname> CHARACTER SET utf8;
docs/ref/databases.txt:593: code block parse error Cannot parse: 1:6: ALTER TABLE <tablename> ENGINE=INNODB;
docs/ref/databases.txt:673: code block parse error Cannot parse: 1:6: ALTER TABLE `your_table` MODIFY `your_datetime_column` DATETIME(6)
docs/ref/databases.txt:957: code block parse error Cannot parse: 1:33: 'NAME': 'localhost:1521/orclpdb1',
docs/ref/databases.txt:961: code block parse error Cannot parse: 4:1: ),
docs/ref/databases.txt:973: code block parse error Cannot parse: 3:1: },
docs/ref/databases.txt:989: code block parse error Cannot parse: 3:1: },
docs/ref/databases.txt:1069: code block parse error Cannot parse: 1:7: mysite/
docs/ref/databases.txt:1095: code block parse error Cannot parse: 5:4:     },

jvzammit · 2023-01-23T22:03:49Z

P.S. I couldn't get blacken-docs to run without the subsequent Unexpected indentation errors.

Help on that would be appreciated. I'm referring to this specific issue; I changed several instance of this:

'key': 'value'

to this:

{
    ...: ...,
    'key': 'value',
    ...: ...,
}

to get blacken-docs to change the ' to ".

Any help on this, perhaps an obvious solution which I'm missing, would be appreciated!

pauloxnet

I tried to write down some suggestions.

pauloxnet · 2023-01-23T23:00:12Z

docs/ref/databases.txt

@@ -577,7 +580,7 @@ Here's a sample configuration which uses a MySQL option file::
    database = NAME
    user = USER
    password = PASSWORD
-    default-character-set = utf8
+    default - character - set = utf8


I think this code a different code block from the previous.

Fixed by marking it setting it to have ini file pygment:

.. code-block:: ini

pauloxnet · 2023-01-23T23:03:25Z

docs/ref/databases.txt

+        ...: ...,
+        "NAME": "localhost:1521/orclpdb1",
+        ...: ...,


Suggested change

...: ...,

"NAME": "localhost:1521/orclpdb1",

...: ...,

# ...

"NAME": "localhost:1521/orclpdb1",

# ...

As used before maybe can work something like that?

This works, thanks @pauloxnet !!

pauloxnet · 2023-01-23T23:05:29Z

docs/ref/databases.txt

+        ...: ...,
+        "NAME": (
+            "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))"
+            "(CONNECT_DATA=(SERVICE_NAME=orclpdb1)))"
+        ),
+        ...: ...,


Suggested change

...: ...,

"NAME": (

"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))"

"(CONNECT_DATA=(SERVICE_NAME=orclpdb1)))"

),

...: ...,

# ...

"NAME": (

"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521))"

"(CONNECT_DATA=(SERVICE_NAME=orclpdb1)))"

),

# ...

See above

Thanks @pauloxnet - fixed

pauloxnet · 2023-01-23T23:06:26Z

docs/ref/databases.txt

+        ...: ...,
+        "OPTIONS": {
+            "threaded": True,
+        },
+        ...: ...,


Suggested change

...: ...,

"OPTIONS": {

"threaded": True,

},

...: ...,

# ...

"OPTIONS": {

"threaded": True,

},

# ...

See above

Thanks @pauloxnet - fixed

pauloxnet · 2023-01-23T23:07:02Z

docs/ref/databases.txt

+        ...: ...,
+        "OPTIONS": {
+            "use_returning_into": False,
+        },
+        ...: ...,


Suggested change

...: ...,

"OPTIONS": {

"use_returning_into": False,

},

...: ...,

# ...

"OPTIONS": {

"use_returning_into": False,

},

# ...

Thanks @pauloxnet - fixed

pauloxnet · 2023-01-23T23:09:39Z

docs/ref/logging.txt

-                'include_html': True,
-                'reporter_class': 'somepackage.error_reporter.CustomErrorReporter',
+        {
+            ...: ...,


Suggested change

...: ...,

# ...

Thanks @pauloxnet - fixed

pauloxnet · 2023-01-23T23:09:59Z

docs/ref/logging.txt

            },
-        },
+            ...: ...,


Suggested change

...: ...,

# ...

Thanks @pauloxnet - fixed

pauloxnet · 2023-01-23T23:10:25Z

docs/ref/logging.txt

-                '()': 'django.utils.log.CallbackFilter',
-                'callback': skip_unreadable_post,
+        {
+            ...: ...,


Suggested change

...: ...,

# ...

pauloxnet · 2023-01-23T23:10:40Z

docs/ref/logging.txt

            },
-        },
+            ...: ...,


Suggested change

...: ...,

# ...

Thanks @pauloxnet - fixed

pauloxnet · 2023-01-23T23:13:28Z

docs/ref/request-response.txt

        Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6)

    For use in, for example, Django templates, headers can also be looked up
    using underscores in place of hyphens::

-        {{ request.headers.user_agent }}
+        {{request.headers.user_agent}}


This isn't a python code block

When passing --rst-literal-blocks to blacken-docs code blocks are treated as Python code blocks by default. So, should this be changed?

I think in that code we can use .. code-block:: html+django

Thanks @pauloxnet, forgot to apply this, applied now. Rebased and force pushed.

carltongibson

I know this is not the complete thing. But it's still time consuming enough to justify creating a PR.

Hey @jvzammit — thanks for pushing this — I know it's fiddly.

Can I suggest you extract the code-block changes into a single commit.

:mod:`django.apps`:

 .. code-block:: pycon

Refs #34140 -- Added missing Sphinx code-block directives for non-Python samples.

That's the most of this (maybe) and it's easy for us to just merge, thereby reducing the scope of the rest of it.

(The other point quickly visible here is that I think making the ... into explicit comments may be better than the quasi-dict ...: ... — which is a bit weird.)

carltongibson · 2023-01-24T09:17:40Z

docs/ref/databases.txt

-            ...
+            ...: ...,


In these cases, is using a comment available:

# ...

Update: I see a0b3803#r1084661525 makes the same point.

Yep changed that, thanks.

The issue is that after changing this I still get these errors on make html:

docs/ref/logging.txt:20: ERROR: Unexpected indentation. docs/ref/logging.txt:290: ERROR: Unexpected indentation. docs/ref/logging.txt:26: WARNING: Definition list ends without a blank line; unexpected unindent. docs/ref/logging.txt:27: WARNING: Block quote ends without a blank line; unexpected unindent. docs/ref/logging.txt:29: WARNING: Block quote ends without a blank line; unexpected unindent. docs/ref/logging.txt:39: ERROR: Unexpected indentation. docs/ref/logging.txt:309: ERROR: Unexpected indentation. docs/ref/logging.txt:45: WARNING: Definition list ends without a blank line; unexpected unindent. docs/ref/logging.txt:46: WARNING: Block quote ends without a blank line; unexpected unindent. docs/ref/logging.txt:48: WARNING: Block quote ends without a blank line; unexpected unindent. docs/ref/settings.txt:1696: ERROR: Unexpected indentation.

cc @pauloxnet

OK, we can look at that 👍

(Let's see if we can get the code-block adjustments merged.)

I "fixed" the ref/settings.txt issue.

But for the logging.txt ones, I can't spot the solution. But I'm no expert, so it might be an obvious mistake I'm making.

Btw, I'm trying to get the "first" part, i.e. have blacken-docs work without errors for any of these files.

We can see if it makes sense to have those changes merged. I.e. without the blacken-docs changes.

That commit would include more than the code-block changes. But not much more.

If we need a "clean" code-block changes only commit, I'll create a new PR with that commit only. But first I need to fix the errors (possibly only one error causing in the rest) in ref/logging.txt 👀

@jvzammit — get it as close as you can, with the energy/time/enthusiasm you have (leaving a buffer! 😅) and I can have a look at any lingerers.

I tried it locally and I confirm the same errors/warnings related to docs/ref/logging.txt but I can't spot the solution either.

Thanks @pauloxnet the indentation changes you suggested fixed the errors 🎉

@carltongibson @pauloxnet I pushed the changes and "a second commit" with the automatic changes by running:

blacken-docs --rst-literal-blocks ref/clickjacking.txt blacken-docs --rst-literal-blocks ref/databases.txt blacken-docs --rst-literal-blocks ref/django-admin.txt blacken-docs --rst-literal-blocks ref/logging.txt blacken-docs --rst-literal-blocks ref/request-response.txt blacken-docs --rst-literal-blocks ref/settings.txt blacken-docs --rst-literal-blocks ref/signals.txt blacken-docs --rst-literal-blocks ref/template-response.txt blacken-docs --rst-literal-blocks ref/unicode.txt blacken-docs --rst-literal-blocks ref/urlresolvers.txt blacken-docs --rst-literal-blocks ref/utils.txt

I can drop that second commit if it expedites merging the first commit 🚀
After all we can just run the command later on (in a subsequent PR).

I think the second commit is ok. I only add one suggestion.

Thanks @pauloxnet I dropped the second commit, apply your correct suggestion, fixup and force push. That puts all "manual" changes prior to running the command in the first commit.

Then ran the command again, and put the changes by the command in the second commit.

How is it looking now?

pauloxnet

@jvzammit I think I fixed the docs/ref/logging.txt file errors. Can you try to accept my suggestions to trigger the pipeline?

docs/ref/logging.txt

pauloxnet

I think I've found the correct code block for the Django template.

docs/ref/request-response.txt

pauloxnet · 2023-01-24T17:05:41Z

LGTM

jvzammit · 2023-01-31T10:53:07Z

@carltongibson I'm afraid of this falling through the cracks. Do we need to do anything else on it? Thanks again!

carltongibson · 2023-01-31T11:02:27Z

Hey @jvzammit — I didn't get the chance to sit down with it yet. (Currently working on tomorrow's releases... 🤹‍♀️)

One query I had last time I looked was whether we could apply the .. code-block changes globally? 🤔
(Likely answer is It's a total pain, and no — but wanted to look at it.)

To help me in the shell, what and the steps you take to run & reset?

Thanks.

jvzammit · 2023-01-31T11:16:03Z

Thanks for the immediate response @carltongibson !!

Currently working on tomorrow's releases...

That takes precedence, thanks for looking into this.

whether we could apply the .. code-block changes globally

I'm going through each one-by-one. Why? In some cases it's not pycon, but one of:

shell
sql
ini
html+django
[etc etc depending on snippet]

To help me in the shell, what and the steps you take to run & reset?

The first commit is all manual. The second commit is the result of running blacken-docs --rst-literal-blocks [file], in this case:

blacken-docs --rst-literal-blocks ref/clickjacking.txt
blacken-docs --rst-literal-blocks ref/databases.txt
blacken-docs --rst-literal-blocks ref/django-admin.txt
blacken-docs --rst-literal-blocks ref/logging.txt
blacken-docs --rst-literal-blocks ref/request-response.txt
blacken-docs --rst-literal-blocks ref/settings.txt
blacken-docs --rst-literal-blocks ref/signals.txt
blacken-docs --rst-literal-blocks ref/template-response.txt
blacken-docs --rst-literal-blocks ref/unicode.txt
blacken-docs --rst-literal-blocks ref/urlresolvers.txt
blacken-docs --rst-literal-blocks ref/utils.txt

I know it's far from automated, but it's a good first, small step. With the benefit of actually improving the snippets in current docs.

felixxm · 2023-02-15T11:12:59Z

@jvzammit Can you rebase after 534ac48?

jvzammit · 2023-02-16T09:53:37Z

@felixxm Thank you! It's not done yet, this is only a few files to test the approach. Will continue working on the rest of the files. Should that be part of this PR? Or should I create new PR(s) with the remaining changes?

felixxm · 2023-02-16T09:59:57Z

@felixxm Thank you! It's not done yet, this is only a few files to test the approach. Will continue working on the rest of the files. Should that be part of this PR? Or should I create new PR(s) with the remaining changes?

I'd do this in a single PR with 3 commits (see #15387):

cleanups to run blacken-docs with no errors,
add GitHub action & docs changes,
changes applied by blacken-docs (assign to django-bot).

carltongibson

OK, so this looks much nicer now I think — It's clear exactly what's at stake in blackening bit.

I wonder if you can drum up a team to divide and conquer the docs @jvzammit? (I see lots of keen people 😜) In theory if we can get all the cleanups in place, we can run blacken-docs at a single pass. 🤔 (Likely it's loads more effort than that sounded like writing it 😅)

jvzammit · 2023-02-16T16:36:50Z

Thanks @felixxm I like the plan you suggested.

I will need help with "add GitHub action & docs changes" step. Can you please provide some pointers on how to achieve that?

An issue I have is that I have not found a way to run blacken-docs --rst-literal-blocks on all docs files. Right now I have a script (locally) to run the above command for all files under docs except for files under docs/_build and docs/releases.

Running it for a directory raises IsADirectoryError error. For example:

$ blacken-docs --rst-literal-blocks ref/*
Traceback (most recent call last):
   ...
IsADirectoryError: [Errno 21] Is a directory: 'ref/class-based-views'

@carltongibson I'm "on it" and hope to have something "decent" soon. From what I see after your commit there are a lot let errors to address 🤞

So I will focus on the first commit hoping the other two are straightforward (third one should be fully-automated).

Thanks again!

jvzammit · 2023-02-16T19:50:45Z

@felixxm @carltongibson

I manually applied all the changes required for blacken-docs --rst-literal-blocks to run for all files except those under docs/_build and docs/releases.

The manual changes are in the first commit.

The second commit is a "throwaway" commit because it was generated by running blacken-docs --rst-literal-blocks for each file. I used this script (I know, it looks like written by a two-year-old, but it's easy to read):

import os

import black
from blacken_docs import format_file


DJANGO_DOCS_DIRECTORY = "<repo root>/django/docs"


def get_txt_filename_list(directory, file_extension):
    build_dir = directory + "/_build"
    releases_dir = directory + "/releases"
    paths = []
    for root, _, filenames in os.walk(directory):
        for filenames in filenames:
            if root.startswith(build_dir):
                continue  # skip build files
            if root.startswith(releases_dir):
                continue  # skip releases files
            if filenames.lower().endswith(file_extension.lower()):
                paths.append(os.path.join(root, filenames))
    return paths


if __name__ == "__main__":
    black_mode = black.FileMode()
    retv = 0
    for filename in get_txt_filename_list(DJANGO_DOCS_DIRECTORY, "txt"):
        retv |= format_file(
            filename,
            black_mode,
            skip_errors=True,
            rst_literal_blocks=True,
        )
    print(retv)

This leads me to these questions:

How to run something built-into blacken-docs to achieve this result? I.e. to replace of "the script" above.
Would that be located in the Build docs step within .github/workflows/docs.yml?

I guess that would be "step 2" as @felixxm described above. Right?

One more question 👉 with regards to the build error, should I add the "offender words" to spelling_wordlist? Or is that overkill? I think it is. Is there a way to avoid running the spellcheck for these words? Or should I just use a different lexer? Like html+django rather than html?

WARNING: ref/templates/builtins.txt:1123: : Spell check: José: <h1>José Mourinho</h1>.
WARNING: ref/templates/builtins.txt:1123: : Spell check: Mourinho: <h1>José Mourinho</h1>.
WARNING: ref/templates/builtins.txt:1124: : Spell check: Thibaut: <p class="odd">Thibaut Courtois</p>.
WARNING: ref/templates/builtins.txt:1124: : Spell check: Courtois: <p class="odd">Thibaut Courtois</p>.
WARNING: ref/templates/builtins.txt:1128: : Spell check: Ancelotti: <h1>Carlo Ancelotti</h1>.
WARNING: ref/templates/builtins.txt:1129: : Spell check: Neuer: <p class="odd">Manuel Neuer</p>.
WARNING: ref/templates/builtins.txt:1[13](https://github.com/django/django/actions/runs/4197603542/jobs/7280137109#step:5:14)0: : Spell check: Müller: <p class="even">Thomas Müller</p>.
WARNING: Found 7 misspelled words

Update: just added the missing words in a separate (the third) commit.

Thanks again!

adamchainz · 2023-02-19T21:16:32Z

We could make pre-commit the only way to run linters, as I started in #15178... This would save duplication, and we wouldn't need to set up extra GitHub Actions config, nor would we need to consider a wrapper script or further blacken-docs features to target particular files.

jvzammit · 2023-02-19T21:28:58Z

@adamchainz That's a sound approach. I'll wait for feedback from @carltongibson & @felixxm on how to proceed this. I.e. if your PR is merged soon-ish I could rebase and use that instead of Github actions.

carltongibson · 2023-02-21T11:29:15Z

TBH neither @felixxm nor I are that keen on moving everything into pre-commit. We already have Jenkins, we already have GHA — let's keep using those. (Personally, I also use the tox environments and would be sad to see them go... 🫠)

felixxm · 2023-02-21T11:36:54Z

Would that be located in the Build docs step within .github/workflows/docs.yml?
I guess that would be "step 2" as @felixxm described above. Right?

I'd add it to the black action. Don't worry about this I can add it to this PR.

jvzammit · 2023-02-22T17:29:12Z

@felixxm This works locally:

find docs -name '*.txt' -not -path '*/_build/*' -not -path '*/releases/*.txt' -exec blacken-docs --rst-literal-blocks {} \;

Downsides of the above:

Could not make the exclusion of docs/_build and docs/releases more restrictive -- i.e. I don't like the */directory_name/* matcher.
Is it inefficient? Because it executes blacken-docs each time. Rather than passing a list of files to blacken-docs. I could not find a way to do the latter.

P.S. Based on @adamchainz 's suggestion here: adamchainz/blacken-docs#224 (comment)

felixxm · 2023-02-22T17:44:29Z

@felixxm This works locally:

find docs -name '*.txt' -not -path '*/_build/*' -not -path '*/releases/*.txt' -exec blacken-docs --rst-literal-blocks {} \;

Downsides of the above:

* Could not make the exclusion of `docs/_build` and `docs/releases` more restrictive -- i.e. I don't like the `*/directory_name/*` matcher.

* Is it inefficient? Because it executes `blacken-docs` each time. Rather than passing a list of files to `blacken-docs`. I could not find a way to do the latter.

P.S. Based on @adamchainz 's suggestion here: adamchainz/blacken-docs#224 (comment)

I'm using:

$ find -name "*.txt" -not -path "./_build/*" -not -path "./_theme/*" | xargs blacken-docs --rst-literal-blocks

jvzammit · 2023-02-22T17:52:32Z

@felixxm The -path arguments are more restrictive in that one 👍

xargs is more elegant as well 👍

Note that running that will fail for the contents of releases directory as @pauloxnet had suggested in #16487 (comment)

Are we to exclude the completely? Or should we exclude only versions for, for example, Django 3 and below? Should be doable via regex.

felixxm · 2023-02-23T04:53:07Z

Are we to exclude the completely? Or should we exclude only versions for, for example, Django 3 and below? Should be doable via regex.

There are not many errors in release notes so I would fix them. One thing I noticed is blacken-docs crashes on ellipsis "…" in code snippets, maybe we can fix this in blacken-docs 🤔 (\cc @adamchainz).

…or "releases" doc files

…er "releases" directory

jvzammit · 2023-02-27T19:05:54Z

@felixxm I pushed changes so that files under releases have their snippets processed by blacken-docs as well. So no need to skip it in the CI command. What else is needed in this PR? Your addition to the Github action config?

P.S. I just changed the ellipsis to ... to move forward. It's not a big deal IMHO.

cc @pauloxnet -- we're including releases files -- I think going forward this change has more pros than cons.

felixxm

@jvzammit I pushed changes (with small edits) required to run blacken-docs to the #16605.

felixxm · 2023-02-28T09:23:30Z

docs/howto/logging.txt

-            'class': 'logging.FileHandler',
-            'filename': 'general.log',
-            'formatter': 'verbose',
+    {


emphasize-lines should be updated.

@felixxm Right, thanks for spotting that!
Where should I push changes to? The new PR you opened? Or still this one?

I've already applied edits to #16605.

So, what's next? Close this PR in favour of #16605 ?

I take it #16605 only includes the manual changes required for blacken-docs to run without any errors. Correct?

I don't see the addition to run blacken-docs as part of CI though.

I will merge and backport #16605, then apply blacken-docs and add a GitHub action as second step.

OK thanks. What happens to this PR? Should I rebase or close in favour of another PR?

You can close it for now, thanks 👍

Thank you @felixxm 👍

jvzammit · 2023-02-28T12:13:35Z

Closing as the changes conflict with the backport PR containing the manual changes from this PR: #16605

jvzammit force-pushed the ticket_34140_aggregator branch 2 times, most recently from 6796ff3 to 2a2cf4b Compare January 23, 2023 21:34

jvzammit mentioned this pull request Jan 23, 2023

Fixed #34140 -- Format python code blocks in documentation files under docs/releases #16487

Closed

jvzammit changed the title ~~Ticket 34140 Blacken docs~~ Fixed #34140 Blacken docs Jan 23, 2023

jvzammit force-pushed the ticket_34140_aggregator branch 2 times, most recently from cefd915 to 221ceb7 Compare January 23, 2023 21:55

pauloxnet reviewed Jan 23, 2023

View reviewed changes

carltongibson reviewed Jan 24, 2023

View reviewed changes

jvzammit force-pushed the ticket_34140_aggregator branch 4 times, most recently from 2d85191 to 4367767 Compare January 24, 2023 09:57

pauloxnet suggested changes Jan 24, 2023

View reviewed changes

docs/ref/logging.txt Outdated Show resolved Hide resolved

docs/ref/logging.txt Outdated Show resolved Hide resolved

jvzammit force-pushed the ticket_34140_aggregator branch from 0d6ac2d to 3855005 Compare January 24, 2023 14:11

jvzammit changed the title ~~Fixed #34140 Blacken docs~~ Fixed #34140 Blacken docs - part 1 of X Jan 24, 2023

pauloxnet suggested changes Jan 24, 2023

View reviewed changes

docs/ref/request-response.txt Outdated Show resolved Hide resolved

jvzammit force-pushed the ticket_34140_aggregator branch from 8369eb9 to ff01267 Compare January 24, 2023 15:28

pauloxnet approved these changes Jan 24, 2023

View reviewed changes

jvzammit mentioned this pull request Jan 24, 2023

Fixed #34140 Blacken docs - part 2 of X jvzammit/django#1

Closed

emarhin approved these changes Jan 25, 2023

View reviewed changes

carltongibson mentioned this pull request Feb 9, 2023

Refs #34140 -- Applied rst code-block to non-Python examples. #16537

Merged

jvzammit force-pushed the ticket_34140_aggregator branch 3 times, most recently from 2432ead to 6a53a03 Compare February 15, 2023 14:04

carltongibson reviewed Feb 16, 2023

View reviewed changes

jvzammit force-pushed the ticket_34140_aggregator branch from a6c1084 to e748340 Compare February 16, 2023 19:30

Ticket 34140: Apply changes to have blacken-docs run without errors

0d759c8

jvzammit force-pushed the ticket_34140_aggregator branch from e748340 to 0d759c8 Compare February 16, 2023 19:39

Ticket 34140: Apply changes made by blacken-docs command

3eef3a7

Ticket 34140: Add new words to spelling_wordlist

3a0afbe

jvzammit mentioned this pull request Feb 17, 2023

Issue 223 -- Remove outdated instruction from README adamchainz/blacken-docs#224

Closed

jvzammit changed the title ~~Fixed #34140 Blacken docs - part 1 of X~~ Fixed #34140 Blacken docs Feb 19, 2023

jvzammit changed the title ~~Fixed #34140 Blacken docs~~ Fixed #34140 blacken-docs Feb 19, 2023

jvzammit mentioned this pull request Feb 22, 2023

Fixed #34140 -- Format python code blocks in documentation files #16261

Closed

jvzammit added 2 commits February 27, 2023 19:57

Ticket 34140: Apply changes to have blacken-docs run without errors f…

828c14a

…or "releases" doc files

Ticket 34140: Apply changes made by blacken-docs command to files und…

432124a

…er "releases" directory

jvzammit force-pushed the ticket_34140_aggregator branch from a094db6 to 432124a Compare February 27, 2023 18:57

felixxm reviewed Feb 28, 2023

View reviewed changes

jvzammit closed this Feb 28, 2023

felixxm mentioned this pull request Jun 21, 2023

Made pre-commit the only way to run Flake8 and isort. #15178

Closed

Fixed #34140 blacken-docs #16496

Fixed #34140 blacken-docs #16496

Conversation

jvzammit commented Jan 23, 2023 • edited

Two commits

jvzammit commented Jan 23, 2023 • edited

pauloxnet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carltongibson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pauloxnet left a comment

Choose a reason for hiding this comment

pauloxnet left a comment

Choose a reason for hiding this comment

pauloxnet commented Jan 24, 2023

jvzammit commented Jan 31, 2023

carltongibson commented Jan 31, 2023

jvzammit commented Jan 31, 2023

felixxm commented Feb 15, 2023

jvzammit commented Feb 16, 2023

felixxm commented Feb 16, 2023

carltongibson left a comment

Choose a reason for hiding this comment

jvzammit commented Feb 16, 2023 • edited

jvzammit commented Feb 16, 2023 • edited

adamchainz commented Feb 19, 2023

jvzammit commented Feb 19, 2023

carltongibson commented Feb 21, 2023

felixxm commented Feb 21, 2023

jvzammit commented Feb 22, 2023

felixxm commented Feb 22, 2023

jvzammit commented Feb 22, 2023

felixxm commented Feb 23, 2023

jvzammit commented Feb 27, 2023 • edited

felixxm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvzammit commented Feb 28, 2023

jvzammit commented Jan 23, 2023 •

edited

jvzammit commented Jan 23, 2023 •

edited

jvzammit commented Feb 16, 2023 •

edited

jvzammit commented Feb 16, 2023 •

edited

jvzammit commented Feb 27, 2023 •

edited