Course search not working

[mjfrey]

After setting up courses and refreshing course meta-data in catalog service, trying to go to
/courses in LMS yields the following error:

edx.devstack.edxapp | 2017-01-25 10:07:57,926 ERROR 52 [search.views] views.py:226 - Search view exception when searching for for user 2: ConnectionError('N/A', "HTTPConnectionPool(host=u'port', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)", MaxRetryError("HTTPConnectionPool(host=u'port', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)",))
edx.devstack.edxapp | Traceback (most recent call last):
edx.devstack.edxapp | File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/search/views.py", line 195, in course_discovery
edx.devstack.edxapp | field_dictionary=field_dictionary,
edx.devstack.edxapp | File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/search/api.py", line 83, in course_discovery_search
edx.devstack.edxapp | searcher = SearchEngine.get_search_engine(getattr(settings, "COURSEWARE_INDEX_NAME", "courseware_index"))
edx.devstack.edxapp | File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/search/search_engine_base.py", line 50, in get_search_engine
edx.devstack.edxapp | return search_engine_class(index=index) if search_engine_class else None
edx.devstack.edxapp | File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/search/elastic.py", line 275, in init
edx.devstack.edxapp | if not self._es.indices.exists(index=self.index_name):
edx.devstack.edxapp | File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 70, in _wrapped
edx.devstack.edxapp | return func(*args, params=params, **kwargs)
edx.devstack.edxapp | File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 149, in exists
edx.devstack.edxapp | self.transport.perform_request('HEAD', _make_path(index), params=params)
edx.devstack.edxapp | File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 274, in perform_request
edx.devstack.edxapp | status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore)
edx.devstack.edxapp | File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 51, in perform_request
edx.devstack.edxapp | raise ConnectionError('N/A', str(e), e)
edx.devstack.edxapp | ConnectionError: ConnectionError(HTTPConnectionPool(host=u'port', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)) caused by: MaxRetryError(HTTPConnectionPool(host=u'port', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known))
edx.devstack.edxapp |
edx.devstack.edxapp | ==> /edx/var/log/nginx/access.log <==
edx.devstack.edxapp | - - 172.19.0.1 - - [25/Jan/2017:15:07:57 +0000] "POST /search/course_discovery/ HTTP/1.1" 500 65 0.276 "http://localhost:18000/courses" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36"

[clintonb]

Fixing this will most likely require us to update the edx/edx-search package. A version supporting ES 1.x has not yet been published; although, the code is completed. Additionally, the package will need to be updated on LMS.

[clintonb]

I did some experimenting today, and I have no clue how this ever made it to production. The simple act of creating the index is not documented anywhere (specifically http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/configuration/edx_search.html)! After creating the indices manually, and upgrading to edx-search 1.0.1, I was unable to reindex the courses.

curl -XPUT 'edx.devstack.elasticsearch:9200/courseware_index'
./manage.py cms reindex_course --all --settings=devstack_docker --setup

I have disabled this feature for the time being: https://github.com/edx/edx-platform/pull/14651. While I would love to fix it, doing so should be done by someone who knows the magic incantations to make it work!

[jibsheet]

@clintonb edx-search is only used on openedx installs. As such, it's relatively unmaintained. The person who wrote it no longer works here. I think Andy or Robert are the closer you have to a SME if you want thumbs on that PR.

[clintonb]

@andy-armstrong @robrap I am merging https://github.com/edx/edx-platform/pull/14651 to get us moving along. Please advise on fixing this if it is a priority.

[andy-armstrong]

Thanks for fixing this, and for the heads up, @clintonb. As @jibsheet said, course-search is not something that anyone has looked at since Marty left. It is on @marcotuts's radar to try to enable at some point in the future, but there's no timeline. @robrap has been doing the necessary work to upgrade the ElasticSearch usage so he may have more to say.

[robrap]

@clintonb I'm not really clear on what makes this issue appear. edx/edx-search already has a 1.0.1 release version which supports 1.x ES, but it is not yet in Production. This is the PR that upgrades edx-platform: https://github.com/edx/edx-platform/pull/14463/files.

When we tested on a sandbox, course search seemed to work on that sandbox. @dianakhuang did the configuration, so should could explain more if needed.

Is this a devstack specific issue? I don't think we have tested it on devstack yet, and it would be good to know if we have a problem coming soon. However, one solution would be to disable this by default on all devstacks, and either the community can help resolve, or let it wait until we (edX) makes course search a priority.

[clintonb]

@robrap this issue is specific to Docker-based devstack. I haven't tried with Vagrant-based devstack. I suspect it's a configuration error, but there isn't enough documentation around this functionality to figure out where the error is.

@dianakhuang if you can share what values need to be set where, please do. No rush, however, as fixing this is not a high priority for anyone (just a nice-to-have).

[robrap]

@dianakhuang is out. I think she just used the Read the Docs page you had posted:
http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/configuration/edx_search.html#supported-flags

[clintonb]

I did some digging this morning, and discovered that the index is installed at the moment the SearchEngine wrapper class is instantiated: https://github.com/edx/edx-search/blob/master/search/elastic.py#L275-L276.

The paver devstack task is responsible for calling a management command that attempts to reindex the courses, and subsequently creates the index: https://github.com/edx/edx-platform/blob/d3b873c7e9ec85247b901bf988e295dbfc816903/pavelib/servers.py#L141-L141.

Additional settings: https://github.com/edx/edx-platform/blob/480a3ca60b30f9a4970094c9e84316fce8c5b631/cms/envs/aws.py#L452-L454.

Ensuring the port is an integer will solve the connection issues; however, that is not easily done with Ansible. See https://github.com/edx/configuration/pull/3887 for details.

The simplest solution may be to pass the combined host and port (e.g Elasticsearch(host='edx.devstack.elasticsearch:9200') instead of Elasticsearch(host='edx.devstack.elasticsearch' port=9200)). This format should be supported: https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.

[macdiesel]

https://openedx.atlassian.net/browse/PLAT-1668