Skip to content
/ phinde Public

Self-hosted search engine for your static blog

License

Notifications You must be signed in to change notification settings

cweiske/phinde

Repository files navigation

phinde - generic web search engine

Self-hosted search engine you can use for your static blog or about any other website you want search functionality for.

My live instance is at http://search.cweiske.de/ and indexes my website, blog and all linked URLs.

Features

  • Crawler and indexer with the ability to run many in parallel
  • Shows and highlights text that contains search words
  • Boolean search queries:
    • foo bar searches for foo AND bar
    • foo OR bar
    • title:foo searches for foo only in the page title
  • Facets for tag, domain, language and type
  • Date search:
    • before:2016-08-30 - modification date before that day
    • after:2016-08-30 - modified after that day
    • date::2016-08-30 - exact modification day match
  • Site search
    • Query: foo bar site:example.org/dir/
    • or use the site GET parameter: /?q=foo&site=example.org/dir
  • OpenSearch support with HTML and Atom result lists
  • Instant indexing with WebSub (formerly PubSubHubbub)

Dependencies

  • PHP 8.x
  • Elasticsearch 2.0
  • MySQL or MariaDB for WebSub subscriptions
  • Gearman (Debian 9: gearman-job-server, not gearman-server)
  • gearadmin command line tool (gearman-tools package)
  • PHP Gearman extension
  • Some PHP libraries that get installed with composer

Setup

  1. Install and run Elasticsearch and Gearman

  2. Install php-gearman and gearman-tools

  3. Get a local copy of the code:

    $ git clone https://git.cweiske.de/phinde.git phinde
    
  4. Install dependencies via composer:

    $ composer install --no-dev
    
  5. Point your webserver's document root to phinde's www directory

  6. Copy data/config.php.dist to data/config.php and adjust it. Make sure your add your domain to the crawl whitelist.

  7. Create a MySQL database and import the schema from data/schema.sql

  8. Run bin/setup.php which sets up the Elasticsearch schema

  9. Put your homepage into the queue:

    $ ./bin/process.php http://example.org/
    
  10. Start at least one worker to process the crawl+index queue:

    $ ./bin/phinde-worker.php
    
  11. Check phinde's status page in your browser. The number of open tasks should be > 0, the number of workers also.

Re-index when your site changes

When your site changed, the search engine needs to re-crawl and re-index the pages.

Simply tell phinde that something changed by running:

$ ./bin/process.php http://example.org/foo.htm

phinde supports HTML pages and Atom feeds, so if your blog has a feed it's enough to let phinde reindex that one. It will find all linked pages automatically.

Website integration

Adding a simple search form to your website is easy. It needs two things:

  • <form> tag with an action that points to the phinde instance
  • Search text field with name of q.

Example:

<form method="get" action="http://phinde.example.org">
  <input type="text" name="q" placeholder="Search text"/>
  <button type="submit">Search</button>
</form>

System service

When using systemd, you can let it run multiple worker instances when the system boots up:

  1. Copy files data/systemd/phinde*.service into /etc/systemd/system/

  2. Adjust user and group names, and the work directories

  3. Enable three worker processes:

    $ systemctl daemon-reload
    $ systemctl enable phinde@1
    $ systemctl enable phinde@2
    $ systemctl enable phinde@3
    $ systemctl enable phinde
    $ systemctl start phinde
    
  4. Now three workers are running. Restarting the phinde service also restarts the workers.

Cron job

Run bin/renew-subscriptions.php once a day with cron. It will renew the WebSub subscriptions.

Howto

Delete index data from one domain:

$ curl -iv -XDELETE -H 'Content-Type: application/json' -d '{"query":{"term":{"domain":"example.org"}}}' http://127.0.0.1:9200/phinde/_query

That's delete-by-query 2.0, see https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/delete-by-query-usage.html

Subscribe to a website/feed

Phinde supports WebSub to get subscribe to changes of a website. When phinde gets notified by the website's hub about changes, it will immediately crawl and index the changed pages.

Subscribe to a website's feed:

$ php bin/subscribe.php http://example.org/feed.atom

Phinde will determine the website's hub and send a registration request to it.

The status page will show the number of working, and the number of open subscriptions.

Unsubscribing also happens on command line:

$ php bin/unsubscribe.php http://example.org/feed.atom

About phinde

Source code

phinde's source code is available from http://git.cweiske.de/phinde.git or the mirror on github.

License

phinde is licensed under the AGPL v3 or later.

Author

phinde was written by Christian Weiske.