Tornado5

[scottbelden]

Resolves #8913

Testing Plan: Ran ./tools/test-all

A few notes:

• Asyncio's loop doesn't seem to have any public functions for seeing what all is running so in https://github.com/scottbelden/zulip/blob/346aa2e9d0f854e71cb8c01e0b3c080445f27ae7/tools/run-dev.py#L398 I just changed to to stop. It might be possible to use asyncio.Task.all_tasks to get this information?

• I moved a lot of the logic to the __init__.py file. If I used the ioloop_logging.py file then there was a test that was failing because our Policy wasn't getting set. See below:

Running API tests...
2018-05-17 17:42:25.094 INFO [] Tornado   0.0% busy over the past  0.3 seconds
2018-05-17 17:42:25.246 WARN [django.server] "POST /api/v1/users HTTP/1.1" 400 67
2018-05-17 17:42:25.500 INFO [process_queue] Not using RabbitMQ queue workers in the test suite.
2018-05-17 17:42:25.904 ERR  [] Traceback (most recent call last):
  File "/srv/zulip-py3-venv/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/zulip-py3-venv/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/srv/zulip/zerver/lib/rest.py", line 131, in rest_dispatch
    return target_function(request, **kwargs)
  File "/srv/zulip/zerver/decorator.py", line 607, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "/srv/zulip-py3-venv/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/srv/zulip/zerver/decorator.py", line 577, in _wrapped_func_arguments
    raise err
  File "/srv/zulip/zerver/decorator.py", line 570, in _wrapped_func_arguments
    return rate_limit()(view_func)(request, profile, *args, **kwargs)
  File "/srv/zulip/zerver/decorator.py", line 755, in wrapped_func
    return func(request, *args, **kwargs)
  File "/srv/zulip/zerver/lib/request.py", line 187, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "/srv/zulip/zerver/views/streams.py", line 249, in remove_subscriptions_backend
    acting_user=user_profile)
  File "/srv/zulip/zerver/lib/actions.py", line 2634, in bulk_remove_subscriptions
    queue_json_publish("deferred_work", event)
  File "/srv/zulip/zerver/lib/queue.py", line 317, in queue_json_publish
    from zerver.worker.queue_processors import get_worker
  File "/srv/zulip/zerver/worker/queue_processors.py", line 36, in <module>
    from zerver.tornado.socket import req_redis_key, respond_send_message
  File "/srv/zulip/zerver/tornado/socket.py", line 277, in <module>
    'disabled_transports': ['eventsource', 'htmlfile']})
  File "/srv/zulip-py3-venv/lib/python3.4/site-packages/sockjs/tornado/router.py", line 94, in __init__
    self.io_loop = io_loop or ioloop.IOLoop.instance()
  File "/srv/zulip-py3-venv/lib/python3.4/site-packages/tornado/ioloop.py", line 221, in instance
    return IOLoop.current()
  File "/srv/zulip-py3-venv/lib/python3.4/site-packages/tornado/ioloop.py", line 282, in current
    loop = asyncio.get_event_loop()
  File "/usr/lib/python3.4/asyncio/events.py", line 576, in get_event_loop
    return get_event_loop_policy().get_event_loop()
  File "/usr/lib/python3.4/asyncio/events.py", line 522, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-22'.

• This doesn't subclass or do anything to Tornado's AsyncIOLoop. Their docs (https://github.com/tornadoweb/tornado/blob/branch5.0/tornado/platform/asyncio.py#L167-L237) seem to imply that referencing those classes is deprecated. Instead we use the Policy to make sure that when tornado calls asyncio.get_event_loop() it returns our custom event loop. One way I verified this manually was to modify the tornado source at tornado.platform.asyncio to add this print statement:

class BaseAsyncIOLoop(IOLoop):
    def initialize(self, asyncio_loop, **kwargs):
        self.asyncio_loop = asyncio_loop
        print('--- BaseAsyncIOLoop: init asyncio loop: {}'.format(asyncio_loop))

NOTE: If you have that print in and run the dev script, the first print will still be the standard Unix event loop rather than our custom one. This is because that script starts the ioloop without setting the policy: https://github.com/zulip/zulip/blob/master/tools/run-dev.py#L417-L422.

@timabbott Let me know if there are any changes you would like. Hopefully this all makes sense...

[zulipbot]

Hello @zulip/server-dependencies members, this pull request was labeled with the "area: dependencies" label, so you may want to check it out!

[scottbelden]

Is the python 3.5 test failure due to the changes here or was that just something that periodically fails?

[timabbott]

I haven't seen that one fail nondeterministically; let me try to reproduce.

[timabbott]

It looks like it's having a problem with installing this: sinwar/sockjs-tornado@b31696b

@sinwar did you do anything with that branch recently?

[zulipbot]

Heads up @scottbelden, we just merged some commits that conflict with the changes your made in this pull request! You can review this repository's recent commits to see where the conflicts occur. Please rebase your feature branch against the upstream/master branch and resolve your pull request's merge conflicts accordingly.

[andersk]

Closed in favor of

• Upgrade to Tornado 6 #21469