Stormcrawler-docker and OpenSearch

[annikajohannimloh]

Hi all,
I am brand new to the topic of search engines, crawlers and docker. My goal is to implement a search engine with OpenSearch. I saw your talk at OpenSearch from February this year, so I decided to use Stormcrawler.
In the docker-compose.yml it says how I want to run elasticSearch instead of URLFrontier. How exactly does this work with OpenSearch? I have seen that there is a crawler frontier for OpenSearch which is based on URLFrontier (https://github.com/PresearchOfficial/opensearch-frontier).
I wanted to ask you, as experts, how I manage to combine these two things. I am grateful for any tip :)

[jnioche]

Hi I have just realised that the stormcrawler-docker had no discussions section, I have created one. Your question would be better there but I'll start answering here for now.

the purpose of the stormcrawler-docker setup is mostly to show to run the Storm services in Docker. Understanding how Storm runs is a difficulty for new users and the stormcrawler-docker resources try to help with that.

I want to run elasticSearch instead of URLFrontier. How exactly does this work with OpenSearch?

Of course when it comes to StormCrawler, you are not limited to a particular backend, you can use Elasticsearch, SOLR, OpenSearch or URLFrontier. You don't need more than one. In your case, you can remove the sections about URLFrontier, ElasticSearch, Kibana and just add service definitions for OpenSearch.

I have seen that there is a crawler frontier for OpenSearch which is based on URLFrontier

This is the implementation of the URLFrontier service running on Opensearch. This would be used instead of using the spouts and statusupdater bolts from the Opensearch module in SC. It won't help you with indexing the documents into OpenSearch or storing metrics.

It is probably easier to start with the Opensearch module instead of URLFrontier.

One you have the services running in Docker (the Storm ones + OpenSearch), have a look at the opensearch module to see how to use the Maven archetype.

Of course you don't have to use Docker and could install OpenSearch and Apache Storm on your computer.

I hope this helps

[rzo1]

Any other logs? ;-)
Index creation worked?

[rzo1]

Ah I see it... @annikajohannimloh You aren't using the shaded JAR, thus the ClassNotFoundException for flux.

You need to use the shaded JAR (120MB+). Sorry for the comment yesterday (I run low on coffee lol).

[rzo1]

The shade should be created after running mvn clean install:

You need to use the JAR including all of its dependencies.

[annikajohannimloh]

All good. Now I can execute both commands without errors, thanks for your help! :)

I am very new to the subject and do not know quite how to proceed. I have already looked at some tutorials, but nowhere is so really described how to proceed. I want to implement a search for a website with OpenSearch. The pages are crawled, are they already indexed or is that where I need to start? How exactly can I use or access this data for my search now?

[rzo1]

The crawled pages are indexed withing OpenSearch. You can run queries against the related _content index: https://opensearch.org/docs/latest/query-dsl

Note: It might be required to do some tweaking regarding language (on index definition time) as described here: https://opensearch.org/docs/latest/query-dsl/analyzers/language-analyzers/

Note: If you are crawling a dynamic web page, you might need to adjust your crawling process to do headless rendering / dynamic scraping.

[jnioche]

A simpler approach to get you unstuck in the short term would be to connect to the supervisor node and run the topology locally instead of doing so in distributed mode. Simply replace storm jar with storm local.

[annikajohannimloh]

Exactly, with storm local it works. At least I can see the results in the OpenSearch dashboard. However, this process always takes almost 1 hour.

[jnioche]

this process always takes almost 1 hour

The topology will run for as long as you tell it to. If you specify a time with 'local' it will run for that amount of time. In distributed mode, it will run forever. SC doesn't stop when it runs out of URLs, that's a difference with other crawlers. In practice it doesn't matter much because a crawl tends to generate more URLs that it can cope with.

You should see the URLs appearing in the OpenSearch indices pretty much as soon as they have been processed.