Skip to content

Commit

Permalink
Quickwit service run several services
Browse files Browse the repository at this point in the history
Closes #1160
  • Loading branch information
fulmicoton committed Mar 28, 2022
1 parent 0db764c commit 1f88a29
Show file tree
Hide file tree
Showing 38 changed files with 1,082 additions and 709 deletions.
158 changes: 146 additions & 12 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 9 additions & 9 deletions docs/get-started/installation.md
Expand Up @@ -4,7 +4,7 @@ sidebar_position: 2
---

Quickwit compiles to a single binary, we provide different methods to install it.
We notably provide musl builds to provide static binaries with no dependencies.
We notably provide musl builds to provide static binaries with no dependencies.


## Download
Expand All @@ -25,8 +25,8 @@ Checkout all builds on [github](https://github.com/quickwit-oss/quickwit/release

Quickwit depends on the following external libraries to work correctly:
- `libpq`: the Postgres client library.
- `libssl`: the industry defacto cryptography library.
These libraries can be installed on your system using the native package manager.
- `libssl`: the industry defacto cryptography library.
These libraries can be installed on your system using the native package manager.
On Ubuntu for instance, you can install these dependencies using the following command:

```bash
Expand All @@ -35,7 +35,7 @@ apt-get -y update && apt-get -y install libpq-dev libssl-dev

:::note

Quickwit static binary packages are also provided as `musl` builds. These packages don't require you to install any external library and can be automatically picked during installation on your system if the required libc version is not present. You can also download and manually install a static binary package.
Quickwit static binary packages are also provided as `musl` builds. These packages don't require you to install any external library and can be automatically picked during installation on your system if the required libc version is not present. You can also download and manually install a static binary package.

:::

Expand Down Expand Up @@ -65,14 +65,14 @@ quickwit-{version}
- `config/quickwit.yaml`: is the default configuration file.
- `LICENSE_AGPLv3.0.txt`: the license file.
- `quickwit`: the quickwit executable binary.
- `qwdata/`: the default data directory.
- `qwdata/`: the default data directory.


## Use the docker image

If you use docker, this might be one of the quickest way to get going.
If you use docker, this might be one of the quickest way to get going.
The following command will pull the image from [dockerhub](https://hub.docker.com/r/quickwit/quickwit)
and gets you right in the shell of the running container ready to execute Quickwit commands.
and gets you right in the shell of the running container ready to execute Quickwit commands.
Note that we are also mounting the working directory as volume. This is useful when you already have your dataset ready on your machine and want to work with Quickwit docker image.

```bash
Expand All @@ -90,14 +90,14 @@ mkdir data && cd data
curl -o wikipedia_index_config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/wikipedia/index-config.yaml
curl -o wiki-articles-10000.json https://quickwit-datasets-public.s3.amazonaws.com/wiki-articles-10000.json

# create, index and search using the container
# create, index and search using the container
docker run -v "$(pwd)":"/quickwit/qwdata" quickwit/quickwit index create --index-config ./qwdata/wikipedia_index_config.yaml

docker run -v "$(pwd)":"/quickwit/qwdata" quickwit/quickwit index ingest --index wikipedia --input-path ./qwdata/wiki-articles-10000.json

docker run -v "$(pwd)":"/quickwit/qwdata" quickwit/quickwit index search --index wikipedia --query "barack obama"

docker run -v "$(pwd)":"/quickwit/qwdata" --expose 7280 -p 7280:7280 quickwit/quickwit service run searcher
docker run -v "$(pwd)":"/quickwit/qwdata" --expose 7280 -p 7280:7280 quickwit/quickwit run --service searcher
```

Now you can make HTTP requests to the searcher service API.
Expand Down
2 changes: 1 addition & 1 deletion docs/get-started/quickstart.md
Expand Up @@ -114,7 +114,7 @@ It should return 10 hits. Now you're ready to serve our search API.
Quickwit provides a search [REST API](../reference/rest-api.md) that can be started using the `service` subcommand.

```bash
./quickwit service run searcher
./quickwit run --service searcher
```

Check it's working with a simple GET request in the browser or via cURL:
Expand Down
14 changes: 7 additions & 7 deletions docs/guides/add-full-text-search-to-your-olap-db.md
Expand Up @@ -4,7 +4,7 @@ sidebar_position: 2
---


This guide will help you add full-text search to a well-known OLAP database, Clickhouse, using the Quickwit search streaming feature. Indeed Quickwit exposes a REST endpoint that streams ids or whatever attributes matching a search query **extremely fast** (up to 50 million in 1 second), and Clickhouse can easily use them with joins queries.
This guide will help you add full-text search to a well-known OLAP database, Clickhouse, using the Quickwit search streaming feature. Indeed Quickwit exposes a REST endpoint that streams ids or whatever attributes matching a search query **extremely fast** (up to 50 million in 1 second), and Clickhouse can easily use them with joins queries.

We will take the [Github archive dataset](https://www.gharchive.org/), which gathers more than 3 billion Github events: `WatchEvent`, `PullRequestEvent`, `IssuesEvent`... You can dive into this [great analysis](https://ghe.clickhouse.tech/) made by Clickhouse to have a good understanding of the dataset. We also took strong inspiration from this work, and we are very grateful to them for sharing this.

Expand Down Expand Up @@ -97,7 +97,7 @@ You can check it's working by using the `search` command and looking for `tantiv
## Start a searcher

```bash
./quickwit service run searcher
./quickwit run --service searcher
```

This command will start an HTTP server with a [REST API](../reference/rest-api.md). We are now
Expand Down Expand Up @@ -163,10 +163,10 @@ gunzip -c gh-archive-2021-12.json.gz | clickhouse-client -d gh-archive --query="
Let's check it's working:
```SQL
# Top repositories by stars
SELECT repo_name, count() AS stars
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY repo_name
SELECT repo_name, count() AS stars
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY repo_name
ORDER BY stars DESC LIMIT 5

┌─repo_name────────────┬─stars─┐
Expand All @@ -181,7 +181,7 @@ ORDER BY stars DESC LIMIT 5
### Use Quickwit search inside Clickhouse

Clickhouse has an exciting feature called [URL Table Engine](https://clickhouse.com/docs/en/engines/table-engines/special/url/) that queries data from a remote HTTP/HTTPS server.
This is precisely what we need: by creating a table pointing to Quickwit search stream endpoint, we will fetch ids that match a query from Clickhouse.
This is precisely what we need: by creating a table pointing to Quickwit search stream endpoint, we will fetch ids that match a query from Clickhouse.

```SQL
SELECT count(*) FROM url('http://127.0.0.1:7280/api/v1/gh-archive/search/stream?query=log4j+OR+log4shell&fastField=id&outputFormat=clickHouseRowBinary', RowBinary, 'id UInt64')
Expand Down
Expand Up @@ -163,7 +163,7 @@ Now let's launch a searcher node for this instance.

```bash
# Then start the http server search service.
./quickwit service run searcher --config ./config.yaml
./quickwit run --service searcher --config ./config.yaml
```

You will see in the terminal the confirmation that the instance has created a new cluster. Example of such a log:
Expand Down

0 comments on commit 1f88a29

Please sign in to comment.