data: currencies, engine descriptions and osm_keys_tags: use SQLite instead of JSON

[dalf]

What does this PR do?

To reduce memory usage, use a SQLite database to store the engine descriptions, currencies and OSM keys/tags.

Dump of the databases are stored in Git to facilitate maintenance, especially the pull requests made automatically every month.

With this PR searx.data provides some functions to access the data:

def fetch_engine_descriptions(language) -> Dict[str, List[str]]:
  """Return engine description and source for each engine name"""

def fetch_iso4217_from_user(name: str) -> Optional[str]:
  """Currency: get the ISO4217 name from the user input"""

def fetch_name_from_iso4217(iso4217: str, language: str) -> Optional[str]:
  """Currency: get the localized name from the ISO4217"""

def fetch_osm_key_label(key_name: str, language: str) -> Optional[str]:
  """Get the OSM key label from the key id"""

def fetch_osm_tag_label(tag_key: str, tag_value: str, language: str) -> Optional[str]:
  """Get the OSM tag label from the tag key and value"""

The function names starts with fetch instead of get to emphasis the fact the data are fetch from the databases.

With these functions are part of the code or engines can access the data without weird import like in the apple map engine:

searxng/searx/engines/apple_maps.py

Line 9 in dbed8da

from searx.engines.openstreetmap import get_key_label

Why is this change important?

It spares about 20MB per worker similar to #3443, but the memory remains low even after some queries using OSM (for example).

SQLite is going to cache some pages, but as far I understand this is kernel cache:

• it is shared between the processes
• the kernel discards the cache entries when the memory is low

About load time: it takes 10ms to load useragents.json, external_urls.json, wikidata_units.json, external_bangs.json, engine_traits.json and locales.json on my AMD 5750GE. Even ten time slower is still reasonable IMO: the HTTP requests during the initialization are way slower than that.

How to test this PR locally?

Author's checklist

Related issues

Related to

• searx.data : use sqlite to reduce the memory footprint #2633
• [mod] lower memory footprint by lazy loading JSON data #3443

[return42]

@dalf can we merge this PR or are you waiting for more test results?

[dalf]

After some test on Paul's instance, the memory increases to nearly its original value after few days.

---

I have updated the code:

• It is rebased on the last master branch
• SQL connections are shared between the threads (it seems Python documentation is misleading: the connections can be shared, a safety belt can be added)
• the cache is reduced to 512KB instead of 2MB
• sql_connection is generator, so if the connection has to be closed, it can be done in one place (idea from @return42 )

@mrpaulblack can you try the last update?