Skip to content

Commit

Permalink
refactor(hadoop): Update Hadoop resource and datasource (#420)
Browse files Browse the repository at this point in the history 
* refactor(hadoop): Add attributes to the Hadoop Data Source

* refactor(hadoop): Update Hadoop resource

* docs(hadoop): Update use_data_catalog

* review(hadoop): Fix typo

* review(hadoop): Remove comment and description
  • Loading branch information
@youngmn
youngmn committed Apr 22, 2024
1 parent a05185a commit b96ef7c
Show file tree
Hide file tree
Showing 8 changed files with 657 additions and 532 deletions.
16 changes: 14 additions & 2 deletions docs/data-sources/hadoop.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,26 @@ The following arguments are required:
This data source exports the following attributes in addition to the arguments above:

* `region_code` - Region code.
* `vpc_no` - The ID of the associated VPC.
* `vpc_no` - The ID of the associated VPC.
* `edge_node_subnet_no` - The subnet ID of the associated edge node.
* `master_node_subnet_no` - The subnet ID of the associated master node.
* `worker_node_subnet_no` - The subnet ID of the associated worker node.
* `master_node_data_storage_type` - Data storage type of master node. There are 2-Options(`SSD`, `HDD`).
* `worker_node_data_storage_type` - Data storage type of master node. There are 2-Options(`SSD`, `HDD`).
* `master_node_data_storage_size` - Data Storage size of master node. Must be between 100(GBi) and 2000(GBi). 4000(GBi) and 6000(GBi) also available.
* `worker_node_data_storage_size` - Data Storage size of master node. Must be between 100(GBi) and 2000(GBi). 4000(GBi) and 6000(GBi) also available.
* `image_product_code` - The image product code of the Hadoop instance.
* `edge_node_product_code` - Edge server product code.
* `master_node_product_code` - Master server product code.
* `worker_node_product_code` - Worker server product code.
* `worker_node_count` - Number of worker server.
* `cluster_type_code` - The cluster type code.
* `version` - The version of Hadoop.
* `ambari_server_host` - The name of ambari host.
* `cluster_direct_access_account` - Account name with direct access to the cluster.
* `login_key` - The login key name.
* `object_storage_bucket` - The name of object storage bucket.
* `bucket_name` - The name of object storage bucket.
* `use_kdc` - Whether to use Kerberos authentication configuration.
* `kdc_realm` - Realm information of kerberos authentication.
* `domain` - Domain.
* `is_ha` - Whether using high availability of the specific Hadoop.
Expand Down
169 changes: 97 additions & 72 deletions docs/resources/hadoop.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,116 +7,141 @@ subcategory: "Hadoop"

Provides a Hadoop instance resource.

~> **NOTE:** This resource only supports VPC environment.

## Example Usage

#### Basic (Vpc)
```hcl
resource "ncloud_login_key" "loginKey" {
```terraform
resource "ncloud_login_key" "login_key" {
key_name = "hadoop-key"
}
resource "ncloud_vpc" "vpc" {
name = "hadoop-vpc"
name = "hadoop-vpc"
ipv4_cidr_block = "10.5.0.0/16"
}
resource "ncloud_subnet" "master_node_subnet" {
vpc_no = ncloud_vpc.vpc.vpc_no
name = "master-node-subnet"
subnet = "10.5.64.0/19"
zone = "KR-2"
subnet_type = "PUBLIC"
network_acl_no = ncloud_vpc.vpc.default_network_acl_no
resource "ncloud_subnet" "edge_node_subnet" {
vpc_no = ncloud_vpc.vpc.vpc_no
name = "edge-node-subnet"
subnet = "10.5.0.0/18"
zone = "KR-2"
subnet_type = "PUBLIC"
network_acl_no = ncloud_vpc.vpc.default_network_acl_no
}
resource "ncloud_subnet" "edge_node_subnet" {
vpc_no = ncloud_vpc.vpc.vpc_no
name = "edge-node-subnet"
subnet = "10.5.0.0/18"
zone = "KR-2"
subnet_type = "PUBLIC"
network_acl_no = ncloud_vpc.vpc.default_network_acl_no
resource "ncloud_subnet" "master_node_subnet" {
vpc_no = ncloud_vpc.vpc.vpc_no
name = "master-node-subnet"
subnet = "10.5.64.0/19"
zone = "KR-2"
subnet_type = "PUBLIC"
network_acl_no = ncloud_vpc.vpc.default_network_acl_no
}
resource "ncloud_subnet" "worker_node_subnet" {
vpc_no = ncloud_vpc.vpc.vpc_no
name = "worker-node-subnet"
subnet = "10.5.96.0/20"
zone = "KR-2"
subnet_type = "PRIVATE"
network_acl_no = ncloud_vpc.vpc.default_network_acl_no
vpc_no = ncloud_vpc.vpc.vpc_no
name = "worker-node-subnet"
subnet = "10.5.96.0/20"
zone = "KR-2"
subnet_type = "PRIVATE"
network_acl_no = ncloud_vpc.vpc.default_network_acl_no
}
resource "ncloud_hadoop" "hadoop" {
vpc_no = "49956"
cluster_name = "hadoopName"
cluster_type_code = "CORE_HADOOP_WITH_SPARK"
admin_user_name = "admin-test"
admin_user_password = "Admin!2Admin"
login_key_name = loginKey.key_name
master_node_subnet_no = ncloud_subnet.master_node_subnet.id
edge_node_subnet_no = ncloud_subnet.edge_node_subnet.id
worker_node_subnet_no = ncloud_subnet.worker_node_subnet.id
bucket_name = "bucket_name"
vpc_no = ncloud_vpc.vpc.vpc_no
cluster_name = "hadoopName"
cluster_type_code = "CORE_HADOOP_WITH_SPARK"
admin_user_name = "admin-test"
admin_user_password = "Admin!2Admin"
login_key_name = ncloud_login_key.login_key.key_name
edge_node_subnet_no = ncloud_subnet.edge_node_subnet.id
master_node_subnet_no = ncloud_subnet.master_node_subnet.id
worker_node_subnet_no = ncloud_subnet.worker_node_subnet.id
bucket_name = "bucket_name"
master_node_data_storage_type = "SSD"
worker_node_data_storage_type = "SSD"
master_node_data_storage_size = 100
worker_node_data_storage_size = 100
}
```


## Argument Reference

The following arguments are supported:

* `image_product_code` -
*
* `vpc_no` - (Required) The ID of the VPC where you want to place the Hadoop Instance.
* TODO: 아래 프로덕트 코드들 -> 문서보고 더 추가할것
* `master_node_product_code` - (Optional) Master node product code to determin the master node server specification to create. Default: Selected as minimum specification. The minimum standards are 1. memory 2. CPU
* `edge_node_product_code` - (Optional) Edge node product code to determin the edge node server specification to create. Default: Selected as minimum specification. The minimum standards are 1. memory 2. CPU
* `worker_node_product_code` - (Optional) Worker node product code to determin the worker node server specification to create. Default: Selected as minimum specification. The minimum standards are 1. memory 2. CPU
* `cluster_name` - (Required) Cluster name to create.
* `cluster_type_code` - (Required) Cluster type code to determin the cluster type to create.
* `add_on_code_list` - (Optional) Hadoop add-on list. There are 4-options(`PRESTO`, `HBASE`, `IMPALA` and `KUDU`). This argument can only be used in Cloud Hadoop version 1.5 or higher.
* `admin_user_name` - (Required) Admin user name of cluster to create. It is the administrator account required to access the Ambari management console.
* `admin_user_password` - (Required) Admin user password of cluster to create.
* `login_key_name` - (Required) Login key name to set the SSH authentication key required when connecting directly to the node.
* `master_node_subnet_no` - (Required) The Subnet ID of master node.
* `edge_node_subnet_no` - (Required) The Subnet ID of edge node.
* `cluster_name` - (Required) Cluster name to create. Can only enter English letters, numbers, and dashes (-), and Korean letters. Must start and end with an English letter (lowercase) or a number. Min: 3, Max: 15
* `cluster_type_code` - (Required) Cluster type code to determine the cluster type to create. Options: CORE_HADOOP_WITH_SPARK
* `admin_user_name` - (Required) Admin user name of cluster to create. It is the administrator account required to access the Ambari management console. Can only be composed of English letters (lowercase), numbers, and dashes (-). Must start and end with an English letter (lowercase) or a number. Min: 3, Max: 15
* `admin_user_password` - (Required) Admin user password of cluster to create. Must include at least 1 alphabetical character (capital letter), special character, and number. Special characters, such as single quotations ('), double quotations ("), the KRW symbol (₩), slashes (/), ampersands (&), back quotes (`), and spaces cannot be included. Min: 8, Max: 20
* `login_key` - (Required) Login key name to set the SSH authentication key required when connecting directly to the node.
* `edge_node_subnet_no` - (Required) The Subnet ID of edge node. Can select a subnet that will locate the edge node. Edge nodes are located in private/public subnets.
* `master_node_subnet_no` - (Required) The Subnet ID of master node. Can select a subnet that will locate the master node. Master nodes are located in private/public subnets
* `worker_node_subnet_no` - (Required) The Subnet ID of worker node. Must be located in Private Subnet.
* `bucket_name` - (Required) Bucket name to space for storing data in Object Storage.
* `master_node_data_storage_type` - (Required) Data storage type of master node. It does not change atfer installation. There are 2-Options(`SSD`, `HDD`). Default: SSD.
* `worker_node_data_storage_type` - (Required) Data Storage type of worker node. It does not change atfer installation. There are 2-Options(`SSD`, `HDD`). Default: SSD.
* `master_node_data_storage_size` - (Required) Data Storage size of master node. Must be between 100(GB) and 2000(GB) in 10(GB) increaments. 4000(GB) and 6000(GB) also available.
* `worker_node_data_storage_size` - (Required) Data Storage size of worker node. Must be between 100(GB) and 2000(GB) in 10(GB) increaments. 4000(GB) and 6000(GB) also available.
* `worker_node_count` - (Optional) Count of worker node. Must be between 2 and 8. Default: 2
* `master_node_data_storage_size` - (Required) Data Storage size of master node. Must be between 100(GBi) and 2000(GBi) in 10(GBi) increaments. 4000(GBi) and 6000(GBi) also available.
* `worker_node_data_storage_size` - (Required) Data Storage size of worker node. Must be between 100(GBi) and 2000(GBi) in 10(GBi) increaments. 4000(GBi) and 6000(GBi) also available.
* `image_product_code` - (Optional) Image product code to determine the Hadoop instance server image specification to create. If not entered, the instance is created for default value. Default: Cloud Hadoop's latest version. It can be obtained through [`ncloud_hadoop_image_products` data source](../data-sources/hadoop_image_products.md)
* `edge_node_product_code` - (Optional, Changeable) Edge node product code to determine the edge node server specification to create. The specification upgrade will be performed after a full service stop, so please stop work in advance. Upgrading to a server with more memory than the current specification is only possible and incurs an additional fee. Default: Selected as minimum specification. The minimum standards are 1. memory 2. CPU. It can be obtained through [`ncloud_hadoop_products` data source](../data-source/hadoop_products.md).
* `master_node_product_code` - (Optional, Changeable) Master node product code to determine the master node server specification to create. The specification upgrade will be performed after a full service stop, so please stop work in advance. Upgrading to a server with more memory than the current specification is only possible and incurs an additional fee. Default: Selected as minimum specification. The minimum standards are 1. memory 2. CPU. It can be obtained through [`ncloud_hadoop_products` data source](../data-sources/hadoop_products.md).
* `worker_node_product_code` - (Optional, Changeable) Worker node product code to determine the worker node server specification to create. The specification upgrade will be performed after a full service stop, so please stop work in advance. Upgrading to a server with more memory than the current specification is only possible and incurs an additional fee. Default: Selected as minimum specification. The minimum standards are 1. memory 2. CPU. It can be obtained through [`ncloud_hadoop_products` data source](../data-sources/hadoop_products.md).
* `add_on_code_list` - (Optional) Hadoop add-on list. This argument can only be used in Cloud Hadoop version 1.5 or higher. Options: PRESTO | HBASE | IMPALA | KUDU | TRINO | NIFI
* `worker_node_count` - (Optional, Changeable) Number of worker server. You can only select between 2 and 8 worker nodes when you first create. The minimum number of worker nodes is 2, and the number of nodes that can be changed at once is 10.
* `use_kdc` - (Optional) Whether to use KDC(Kerberos Distribute Center). Default: false
* `kdc_realm` - (Required if `use_kdc` is provided) Only domain rules of type Realm are allowed.
* `kdc_password` - (Required if `use_kdc` is provided) Password of KDC.
* `kdc_realm` - (Required if `use_kdc` is provided) KDC's Realm information. Can be entered only if useKdc is true. Only realm-format domain rules are allowed. Only uppercase letters (A-Z) are allowed and up to 15 digits are allowed. Only one dot(.) is allowed (ex. EXAMPLE.COM).
* `kdc_password` - (Required if `use_kdc` is provided) Password of KDC. Can be entered only if useKdc is true. Must include at least 1 alphabetical character (capital letter), special character, and number. Special characters, such as single quotations ('), double quotations ("), the KRW symbol (₩), slashes (/), ampersands (&), back quotes (`), and spaces cannot be included. Min: 8, Max: 20
* `use_bootstrap_script` - (Optional) Whether to use bootstrap script. Default: false.
* `bootstrap_script` - (Required if `use_kdc` is provided) Bootstrap script. Script can only be performed with buckets linked to Cloud Hadoop. Requires entering folder and file names excluding bucket name.
* `use_data_catalog` - (Optional) Whether to use data catalog. It is provided by using the Cloud Hadoop Hive Metastore as the catalog for the Data Catalog service. Integration is possible only when the catalog status of the Data Catalog service is normal. Intergration is possible only with Cloud Hadoop version 2.0 or higher.
* `bootstrap_script` - (Required if `use_kdc` is provided) Bootstrap script. Script can only be performed with buckets linked to Cloud Hadoop. Requires entering folder and file names excluding bucket name. Only English is supported. Cannot use spaces or special characters. Available up to 1024 bytes.
* `use_data_catalog` - (Optional) Whether to use data catalog. Available only `public` site. It is provided by using the Cloud Hadoop Hive Metastore as the catalog for the Data Catalog service. Integration is possible only when the catalog status of the Data Catalog service is normal. Intergration is possible only with Cloud Hadoop version 2.0 or higher. Default: false

## Attributes Reference

* `ID` - The ID of hadoop instance.
* `login_key` - The login key of hadoop instance.
* `object_storage_bucket` - The Object storage.
In addition to all arguments above, the following attributes are exported

* `id` - The ID of hadoop instance.
* `region_code` - Region code.
* `ambari_server_host` - Ambari server host.
* `cluster_direct_access_account` - Account to access the cluster directly.
* `is_ha` - Whether is High Availability or not.
* `version` - The version of Hadoop.
* `is_ha` - Whether using high availability of the specific Hadoop.
* `domain` - Domain.
* `access_control_group_no_list` - Access control group number list.
* `hadoop_server_instance_list` - Server instance list of the hadoop instance.
* `hadoop_server_name` - Name of the hadoop server instance.
* `hadoop_server_role` - Role of the hadoop server instance.
* `hadoop_product_code` - Product code of the hadoop server instance.
* `region_code` - Region code of the hadoop server instance.
* `zone_code` - Zone code of the hadoop server instance.
* `vpc_no` - Vpc no of the hadoop server instance.
* `subnet_no` - Subnet no of the hadoop server instance.
* `is_public_subnet` - Whether is Public Subnet or Private Subnet for the hadoop server instance.
* `data_storage_size` - Data storage size of the hadoop server instance.
* `cpu_count` - Cpu count of the hadoop server instance.
* `memory_size` - Memory size of the hadoop server instance.
*
* `hadoop_server_list` The list of Hadoop server instance.
* `server_instance_no` - Server instance number.
* `server_name` - Name of the server.
* `server_role` - Server role code. ex) M(Master), H(Standby Master)
* `zone_code` - Zone code.
* `subnet_no` - The ID of the associated Subnet.
* `product_code` - Product code.
* `is_public_subnet` - Public subnet status.
* `cpu_count` - the number of the virtual CPU.
* `memory_size` - Memory size.
* `data_storage_type` - The type of data storage.
* `data_storage_size` - Data storage size.
* `uptime` - Running start time.
* `create_date` - Server create date.

## Import

### `terraform import` command

* Hadoop can be imported using the `id`. For example:

```console
$ terraform import ncloud_hadoop.rsc_name 12345
```

### `import` block

* In Terraform v1.5.0 and later, use an [`import` block](https://developer.hashicorp.com/terraform/language/import) to import Hadoop using the `id`. For example:

```terraform
import {
to = ncloud_hadoop.rsc_name
id = "12345"
}
```
12 changes: 8 additions & 4 deletions examples/hadoop/main.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ provider "ncloud" {
region = var.region
}

resource "ncloud_login_key" "login_key" {
key_name = "hadoop-key"
}

resource "ncloud_vpc" "vpc" {
name = var.vpc_name
ipv4_cidr_block = "10.5.0.0/16"
Expand Down Expand Up @@ -43,12 +47,12 @@ resource "ncloud_subnet" "worker_subnet" {
resource "ncloud_hadoop" "hadoop" {
vpc_no = ncloud_vpc.vpc.vpc_no
cluster_name = var.hadoop_cluster_name
cluster_type_code = var.cluster_type_code
cluster_type_code = "CORE_HADOOP_WITH_SPARK"
admin_user_name = var.admin_user_name
admin_user_password = var.admin_user_password
login_key_name = var.login_key_name
master_node_subnet_no = ncloud_subnet.master_subnet.subnet_no
login_key = ncloud_login_key.login_key.key_name
edge_node_subnet_no = ncloud_subnet.edge_subnet.subnet_no
master_node_subnet_no = ncloud_subnet.master_subnet.subnet_no
worker_node_subnet_no = ncloud_subnet.worker_subnet.subnet_no
bucket_name = var.bucket_name
master_node_data_storage_type = "SSD"
Expand All @@ -59,4 +63,4 @@ resource "ncloud_hadoop" "hadoop" {

data "ncloud_hadoop" "by_cluster_name" {
cluster_name = ncloud_hadoop.hadoop.cluster_name
}
}
8 changes: 0 additions & 8 deletions examples/hadoop/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,6 @@ variable "admin_user_password" {
default = "admin_user_password"
}

variable "login_key_name" {
default = "login_key_name"
}

variable "cluster_type_code" {
default = "CORE_HADOOP_WITH_SPARK"
}

variable "bucket_name" {
default = "bucket_name"
}

0 comments on commit b96ef7c

Please sign in to comment.