Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index templates support #1343

Merged
merged 3 commits into from Jan 31, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/index.md
Expand Up @@ -9,6 +9,7 @@ Available documentation for FOSElasticaBundle
* [Using a Serializer](serializer.md)
* [Types](types.md)
* [Provider](provider.md)
* [Index templates](templates.md)

Cookbook Entries
----------------
Expand Down
47 changes: 47 additions & 0 deletions doc/templates.md
@@ -0,0 +1,47 @@
Index templates
================

Index templates allow you to define templates that will automatically be applied when new indices are created
(see more in [official documentation](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/indices-templates.html)).
Index Templates widely is used to create historically indexes:

* storing logs (Kibana) to prevent index growing because of incorrectly mapped fields
* metrics (Marvel) to archive or delete old information

Here's example on how to configure index templates:

```yaml
# app/config/config.yml

fos_elastica:
index_templates:
<name>:
client: default
template_name: <template name>
template: some_index_*
settings:
number_of_shards: 1
number_of_replicas: 0
types:
auto_suggest:
mappings:
<field name>: <params>
...
```

Index template is similar to index configuration and has the same fields like `types`, `settings`, `client`, etc. with additional fields:

1. `template_name` - template name. If omitted then used key (`<name>`) of `index_templates` section. Example: `template_1`
2. `template` - template pattern. Example: `te*` or `bar*`

To apply templates changes, you should run `fos:elastica:reset-templates` command:

* `--index` - index template name to reset. If no index template name specified than all templates will be reset
* `--force-delete` - will delete all indexes that match index templates patterns. Aware that pattern may match various indexes.

You must run the following command to sync templates configuration on ES server with YAML configurations:
```bash
php bin/console fos:elastica:reset-templates
```

You can build-in this command into the deployment process to automate template configuration sync.
90 changes: 90 additions & 0 deletions src/Command/ResetTemplatesCommand.php
@@ -0,0 +1,90 @@
<?php

/*
* This file is part of the FOSElasticaBundle package.
*
* (c) FriendsOfSymfony <http://friendsofsymfony.github.com/>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/

namespace FOS\ElasticaBundle\Command;

use FOS\ElasticaBundle\Index\TemplateResetter;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Question\ConfirmationQuestion;

/**
* Reset search indexes templates.
*/
final class ResetTemplatesCommand extends Command
{
protected static $defaultName = 'fos:elastica:reset-templates';

/** @var TemplateResetter */
private $resetter;

public function __construct(
TemplateResetter $resetter
) {
parent::__construct();

$this->resetter = $resetter;
}

/**
* {@inheritdoc}
*/
protected function configure()
dbalabka marked this conversation as resolved.
Show resolved Hide resolved
{
$this
->setName('fos:elastica:reset-templates')
->addOption(
'index',
null,
InputOption::VALUE_REQUIRED,
'The index template to reset. If no index template name specified than all templates will be reset'
)
->addOption(
'force-delete',
null,
InputOption::VALUE_NONE,
'Delete all indexes that matches index templates patterns. ' .
'Aware that pattern may match various indexes.'
)
->setDescription('Reset search indexes templates')
;
}

/**
* {@inheritdoc}
*/
protected function execute(InputInterface $input, OutputInterface $output)
dbalabka marked this conversation as resolved.
Show resolved Hide resolved
{
$indexTemplate = $input->getOption('index');
$deleteByPattern = $input->getOption('force-delete');

if ($deleteByPattern) {
$helper = $this->getHelper('question');
$question = new ConfirmationQuestion('You are going to remove all template indexes. Are you sure?', false);
dbalabka marked this conversation as resolved.
Show resolved Hide resolved

if (!$helper->ask($input, $output, $question)) {
return 1;
}
}

if (null !== $indexTemplate) {
$output->writeln(sprintf('<info>Resetting template</info> <comment>%s</comment>', $indexTemplate));
$this->resetter->resetIndex($indexTemplate, $deleteByPattern);
} else {
$output->writeln('<info>Resetting all templates</info>');
$this->resetter->resetAllIndexes($deleteByPattern);
}

return 0;
}
}
79 changes: 2 additions & 77 deletions src/Configuration/IndexConfig.php
Expand Up @@ -20,36 +20,9 @@

namespace FOS\ElasticaBundle\Configuration;

class IndexConfig
class IndexConfig implements IndexConfigInterface
{
/**
* The name of the index for ElasticSearch.
*
* @var string
*/
private $elasticSearchName;

/**
* The internal name of the index. May not be the same as the name used in ElasticSearch,
* especially if aliases are enabled.
*
* @var string
*/
private $name;

/**
* An array of settings sent to ElasticSearch when creating the index.
*
* @var array
*/
private $settings;

/**
* All types that belong to this index.
*
* @var TypeConfig[]
*/
private $types;
use IndexConfigTrait;

/**
* Indicates if the index should use an alias, allowing an index repopulation to occur
Expand All @@ -75,54 +48,6 @@ public function __construct($name, array $types, array $config)
$this->useAlias = isset($config['useAlias']) ? $config['useAlias'] : false;
}

/**
* @return string
*/
public function getElasticSearchName()
{
return $this->elasticSearchName;
}

/**
* @return string
*/
public function getName()
{
return $this->name;
}

/**
* @return array
*/
public function getSettings()
{
return $this->settings;
}

/**
* @param string $typeName
*
* @return TypeConfig
*
* @throws \InvalidArgumentException
*/
public function getType($typeName)
{
if (!array_key_exists($typeName, $this->types)) {
throw new \InvalidArgumentException(sprintf('Type "%s" does not exist on index "%s"', $typeName, $this->name));
}

return $this->types[$typeName];
}

/**
* @return \FOS\ElasticaBundle\Configuration\TypeConfig[]
*/
public function getTypes()
{
return $this->types;
}

/**
* @return bool
*/
Expand Down
40 changes: 40 additions & 0 deletions src/Configuration/IndexConfigInterface.php
@@ -0,0 +1,40 @@
<?php

namespace FOS\ElasticaBundle\Configuration;

/**
* Interface Index config interface
*
* @author Dmitry Balabka <dmitry.balabka@intexsys.lv>
*/
interface IndexConfigInterface
{
/**
* @return string
*/
public function getElasticSearchName();

/**
* @return string
*/
public function getName();

/**
* @return array
*/
public function getSettings();

/**
* @param string $typeName
*
* @return TypeConfig
*
* @throws \InvalidArgumentException
*/
public function getType($typeName);

/**
* @return \FOS\ElasticaBundle\Configuration\TypeConfig[]
*/
public function getTypes();
}
96 changes: 96 additions & 0 deletions src/Configuration/IndexConfigTrait.php
@@ -0,0 +1,96 @@
<?php
/*
* This file is part of the OpCart software.
*
* (c) 2015, OpticsPlanet, Inc
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/

namespace FOS\ElasticaBundle\Configuration;

/**
* Index configuration trait class
*
* @author Dmitry Balabka <dmitry.balabka@intexsys.lv>
*/
trait IndexConfigTrait
{
/**
* The name of the index for ElasticSearch.
*
* @var string
*/
private $elasticSearchName;

/**
* The internal name of the index. May not be the same as the name used in ElasticSearch,
* especially if aliases are enabled.
*
* @var string
*/
private $name;

/**
* An array of settings sent to ElasticSearch when creating the index.
*
* @var array
*/
private $settings;

/**
* All types that belong to this index.
*
* @var TypeConfig[]
*/
private $types;

/**
* @return string
*/
public function getElasticSearchName()
{
return $this->elasticSearchName;
}

/**
* @return string
*/
public function getName()
{
return $this->name;
}

/**
* @return array
*/
public function getSettings()
{
return $this->settings;
}

/**
* @param string $typeName
*
* @return TypeConfig
*
* @throws \InvalidArgumentException
*/
public function getType($typeName)
{
if (!array_key_exists($typeName, $this->types)) {
throw new \InvalidArgumentException(sprintf('Type "%s" does not exist on index "%s"', $typeName, $this->name));
}

return $this->types[$typeName];
}

/**
* @return \FOS\ElasticaBundle\Configuration\TypeConfig[]
*/
public function getTypes()
{
return $this->types;
}
}