Terraform多云Kafka实例部署方案详解#

一、Terraform基础知识#

1.1 什么是Terraform？#

Terraform是HashiCorp公司开发的开源基础设施即代码（Infrastructure as Code）工具，允许用户通过声明式配置文件定义和管理云基础设施。它支持多种云服务提供商，包括阿里云、腾讯云、AWS、Azure等。

1.2 核心概念#

Provider：云服务提供商的插件，负责与云API交互
Resource：基础设施的组件，如VM、网络、存储等
Module：可重用的配置块，封装了一组相关资源
State：存储基础设施当前状态的文件
Backend：存储状态文件的位置，可以是本地或远程存储

1.3 工作流程#

编写配置：创建.tf文件定义基础设施
初始化：运行terraform init下载依赖和初始化backend
计划：运行terraform plan预览变更
应用：运行terraform apply应用变更
销毁：运行terraform destroy销毁资源

二、项目结构设计#

2.1 整体结构#

1
├── aliyun/                  # 阿里云实例文件
2
│   └── aliyun_alikafka_demo/  # 阿里云Kafka示例
3
│       └── aliyun_china_pt_7111/  # 阿里云账号
4
│           └── alikafka_demo_202502061910/  # 实例
5
│               ├── backend.tf  # 后端配置
6
│               └── main.tf     # 主配置
7
├── tcloud/                  # 腾讯云实例文件
8
│   └── tcloud_ckafka_demo/    # 腾讯云Kafka示例
9
│       └── tcloud_china_game_x6/  # 腾讯云账号
10
│           └── ckafaka_demo_202501221738/  # 实例
11
│               ├── backend.tf  # 后端配置
12
│               └── main.tf     # 主配置
13
└── modules/                 # 模块文件
14
    ├── aliyun/              # 阿里云模块
15
    │   └── aliyun_alikafka_mod_demo/  # 阿里云Kafka模块
16
    │       ├── main.tf       # 主配置
17
    │       └── variables.tf  # 变量定义
18
    └── tcloud/              # 腾讯云模块
19
        └── tcloud_ckafka_mod_demo/  # 腾讯云Kafka模块
20
            ├── main.tf       # 主配置
21
            ├── outputs.tf    # 输出定义
22
            └── variables.tf  # 变量定义

2.2 结构说明#

云厂商目录：按云厂商分类，如aliyun、tcloud
项目目录：按项目或用途分类，如alikafka_demo、ckafka_demo
账号目录：按云账号分类，如aliyun_china_pt_7111、tcloud_china_game_x6
实例目录：按具体实例分类，包含时间戳便于区分版本
模块目录：按云厂商和资源类型分类，封装可重用的配置

三、模块定义详解#

3.1 阿里云Kafka模块#

3.1.1 主配置文件 (`/modules/aliyun/aliyun_alikafka_mod_demo/main.tf`)#

1
terraform {
2
  required_providers {
3
    alicloud = {
4
      source  = "aliyun/alicloud"
5
      version = "1.225.0"
6
    }
7
  }
8
}
9

10
provider "alicloud" {
11
  region = var.region
12
}
13

14
resource "alicloud_alikafka_instance" "this" {
15
  name          = var.instance_name
16
  deploy_type   = var.deploy_type
17
  disk_size     = var.disk_size
18
  disk_type     = var.disk_type
19
  vswitch_id    = var.vswitch_id
20
  partition_num = var.partition_num
21
  io_max        = var.io_max
22
}

3.1.2 变量定义 (`/modules/aliyun/aliyun_alikafka_mod_demo/variables.tf`)#

1
variable "region" {
2
  description = "地域"
3
  type        = string
4
}
5

6
variable "instance_name" {
7
  description = "kafka实例名"
8
  type        = string
9
}
10

11
variable "deploy_type" {
12
  // - 4: eip/vpc instance；- 5: vpc instance.
13
  description = "部署类型"
14
  type        = number
15
  default = 5
16
  validation {
17
    condition     = contains([4, 5], var.deploy_type)
18
    error_message = "The deploy_type must be one of 4, 5"
19
  }
20
}
21

22
variable "disk_size" {
23
  description = "kafka实例磁盘规格"
24
  type = number
25
  validation {
26
    condition     = contains([500, 1000], var.disk_size)
27
    error_message = "The disk_size must be one of 200, 400"
28
  }
29
}
30

31
variable "disk_type" {
32
  # 0: efficient cloud disk , 1: SSD.
33
  description = "磁盘类型"
34
  type = number
35
  default = 1
36
  validation {
37
    condition     = contains([0, 1], var.disk_type)
38
    error_message = "The disk_type must be empty, or one of 0, 1"
39
  }
40
}
41

42
variable "vswitch_id" {
43
  description = "绑定虚拟交换机id"
44
  type = string
45
}
46

47
variable "partition_num" {
48
  description = "分区数量"
49
  type = number
50
}
51

52
variable "io_max" {
53
  description = "io的最大值"
54
  type = number
55
}

3.2 腾讯云Kafka模块#

3.2.1 主配置文件 (`/modules/tcloud/tcloud_ckafka_mod_demo/main.tf`)#

1
terraform {
2
  required_providers {
3
    tencentcloud = {
4
      source = "tencentcloudstack/tencentcloud"
5
    }
6
  }
7
}
8

9
provider "tencentcloud" {
10
  region = var.region
11
}
12

13
data "tencentcloud_availability_zones_by_product" "zone" {
14
  name    = var.availability_zone
15
  product = "ckafka"
16
}
17

18
resource "tencentcloud_ckafka_instance" "this" {
19
  instance_name      = var.instance_name
20
  zone_id            = data.tencentcloud_availability_zones_by_product.zone.zones[0].id
21
  vpc_id             = var.vpc_id
22
  subnet_id          = var.vswitch_id
23
  msg_retention_time = var.msg_retention_time
24
  kafka_version      = var.kafka_version
25
  disk_size          = var.disk_size
26
  band_width         = var.band_width
27
  disk_type          = var.disk_type
28
  partition          = var.partition
29
  charge_type        = var.charge_type
30

31
  config {
32
    auto_create_topic_enable   = var.auto_create_topic_enable
33
    default_num_partitions     = var.num_partitions
34
    default_replication_factor = var.replication_factor
35
  }
36

37
  dynamic_retention_config {
38
    enable = var.dynamic_retention_config_enable
39
  }
40

41
}

3.2.2 变量定义 (`/modules/tcloud/tcloud_ckafka_mod_demo/variables.tf`)#

1
variable "region" {
2
  description = "地域"
3
  type        = string
4
  default     = "ap-shanghai"
5
}
6

7
variable "instance_name" {
8
  description = "kafka实例名"
9
  type        = string
10
}
11

12
variable "availability_zone" {
13
  # 如ap-shanghai-2
14
  description = "可用区"
15
  type = string
16
}
17

18
variable "charge_type" {
19
  # PREPAID(预付费), POSTPAID_BY_HOUR(按量付费)
20
  description = "kafka实例计费方式"
21
  type = string
22
  validation {
23
    condition     = contains(["PREPAID", "POSTPAID_BY_HOUR"], var.charge_type)
24
    error_message = "The charge_type must be one of PREPAID, POSTPAID_BY_HOUR"
25
  }
26
}
27

28
variable "kafka_version" {
29
  # 0.10.2/1.1.1/2.4.1/2.8.1
30
  description = "kafka实例版本"
31
  type = string
32
  validation {
33
    condition     = contains(["0.10.2", "1.1.1", "2.4.1", "2.4.2", "2.8.1"], var.kafka_version)
34
    error_message = "The kafka_version must be one of 0.10.2, 1.1.1, 2.4.1, 2.4.2, 2.8.1"
35
  }
36
}
37

38
variable "vswitch_id" {
39
  // 阿里云vswitch_id -> 腾讯云subnet_id
40
  description = "绑定子网id"
41
  type = string
42
}
43

44
variable "vpc_id" {
45
  description = "绑定vpc_id"
46
  type = string
47
}
48

49
variable "disk_size" {
50
  # 需满足当前实例的计费规格，此处预设200和400，可以根据需要修改
51
  description = "kafka实例磁盘规格"
52
  type = number
53
  validation {
54
    condition     = contains([200, 400], var.disk_size)
55
    error_message = "The disk_size must be one of 200, 400"
56
  }
57
}
58

59
variable "disk_type" {
60
  # 专业版实例磁盘类型，标准版实例不需要填写，CLOUD_SSD(SSD云硬盘), CLOUD_BASIC(高性能云硬盘)
61
  description = "kafka专业版实例磁盘类型"
62
  type = string
63
  default = ""
64
  validation {
65
    condition     = contains(["", "CLOUD_SSD", "CLOUD_BASIC"], var.disk_type)
66
    error_message = "The disk_type must be empty, or one of CLOUD_SSD, CLOUD_BASIC"
67
  }
68
}
69

70
variable "band_width" {
71
  # 单位为MBps.
72
  description = "kafka实例带宽"
73
  type = number
74
}
75

76
variable "auto_create_topic_enable" {
77
  description = "是否自动创建topic"
78
  type = bool
79
  default = true
80
  validation {
81
    condition     = contains([true, false], var.auto_create_topic_enable)
82
    error_message = "The auto_create_topic_enable must be one of true, false"
83
  }
84
}
85

86
variable "num_partitions" {
87
  description = "kafka实例默认分区数"
88
  type = number
89
  default = 3
90
}
91

92
variable "replication_factor" {
93
  description = "kafka实例默认副本数"
94
  type = number
95
  default = 2
96
}
97

98
variable "dynamic_retention_config_enable" {
99
  description = "是否启用动态消息保留时间配置"
100
  type = number
101
  default = 0
102
  validation {
103
    condition     = contains([0, 1], var.dynamic_retention_config_enable)
104
    error_message = "The dynamic_retention_config_enable must be one of 0, 1"
105
  }
106
}
107

108
variable "msg_retention_time" {
109
  # 以分钟为单位
110
  description = "kafka实例日志的最大保留时间"
111
  type = number
112
  default = 10080
113
}
114

115
variable "partition" {
116
  description = "kafka实例分区大小"
117
  type = number
118
  default = 3
119
}

3.2.3 输出定义 (`/modules/tcloud/tcloud_ckafka_mod_demo/outputs.tf`)#

1
output "ckafka_instance_id" {
2
  value = tencentcloud_ckafka_instance.this.id
3
}

四、实例使用方法#

4.1 阿里云Kafka实例#

4.1.1 后端配置 (`/aliyun/aliyun_alikafka_demo/aliyun_china_pt_7111/alikafka_demo_202502061910/backend.tf`)#

1
terraform {
2
  backend "oss" {
3
    endpoint = "oss-cn-hangzhou.aliyuncs.com"
4
    bucket   = "dz-devops" # 替换为你的 OSS Bucket 名称
5
    prefix = "terraform_state//aliyun/aliyun_alikafka_demo/aliyun_china_pt_7111/alikafka_demo_202502061910"
6
    key      = "terraform.tfstate" # 存储状态文件的路径和名称
7
    region   = "cn-hangzhou"       # OSS 的地域（根据你的实际情况调整）
8
  }
9
}

4.1.2 主配置 (`/aliyun/aliyun_alikafka_demo/aliyun_china_pt_7111/alikafka_demo_202502061910/main.tf`)#

1
module "alikafka_instance" {
2
  source        = "../../../../modules/aliyun/aliyun_alikafka_mod_demo"
3
  region        = "cn-hangzhou"
4
  disk_size     = 500
5
  instance_name = "alikafka-testV2"
6
  deploy_type   = 5
7
  vswitch_id    = "vsw-bp1co65f3q2s0bis9yfkg"
8
  partition_num = 50
9
  io_max        = 20
10
}

4.2 腾讯云Kafka实例#

4.2.1 后端配置 (`/tcloud/tcloud_ckafka_demo/tcloud_china_game_x6/ckafaka_demo_202501221738/backend.tf`)#

1
terraform {
2
  backend "oss" {
3
    endpoint = "oss-cn-hangzhou.aliyuncs.com"
4
    bucket   = "dz-devops" # 替换为你的 OSS Bucket 名称
5
    prefix = "terraform_state/tcloud/tcloud_ckafka_demo/tcloud_china_game_x6/ckafka_demo_202501221738"
6
    key      = "terraform.tfstate" # 存储状态文件的路径和名称
7
    region   = "cn-hangzhou"       # OSS 的地域（根据你的实际情况调整）
8
  }
9
}

4.2.2 主配置 (`/tcloud/tcloud_ckafka_demo/tcloud_china_game_x6/ckafaka_demo_202501221738/main.tf`)#

1
module "ckafka_instance" {
2
  source              = "../../../../modules/tcloud/tcloud_ckafka_mod_demo"
3
  region              = "ap-shanghai"
4
  instance_name       = "ckafka-test"
5
  availability_zone   = "ap-shanghai-2"
6
  charge_type         = "POSTPAID_BY_HOUR"
7
  kafka_version       = "2.4.2"
8
  vpc_id              = "vpc-4tkroxts"
9
  vswitch_id          = "subnet-oy1pqvzv"
10
  disk_size           = 200
11
  # disk_type           = "CLOUD_BASIC"
12
  band_width          = 20
13
  auto_create_topic_enable = true
14
  num_partitions     = 3
15
  replication_factor = 3
16
  dynamic_retention_config_enable = 1
17
  msg_retention_time = 1300
18
  partition = 400
19
}

五、运行测试步骤#

5.1 准备工作#

5.1.1 获取云服务提供商的API密钥#

阿里云：登录阿里云控制台，在「访问控制」中创建AccessKey
腾讯云：登录腾讯云控制台，在「访问管理」中创建SecretId和SecretKey

5.1.2 设置环境变量#

阿里云：

1
export ALICLOUD_ACCESS_KEY=your_access_key
2
export ALICLOUD_SECRET_KEY=your_secret_key

腾讯云：

1
export TENCENTCLOUD_SECRET_ID=your_secret_id
2
export TENCENTCLOUD_SECRET_KEY=your_secret_key

5.2 运行Terraform命令#

5.2.1 初始化项目#

1
# 进入实例目录
2
cd ./aliyun/aliyun_alikafka_demo/aliyun_china_pt_7111/alikafka_demo_202502061910
3

4
# 初始化项目（替换为实际的backend AK/SK）
5
terraform init -backend-config="access_key=xxx" -backend-config="secret_key=yyy"

5.2.2 预览变更#

1
terraform plan

5.2.3 应用变更#

1
terraform apply

5.2.4 销毁资源#

1
terraform destroy

5.3 验证结果#

阿里云：登录阿里云控制台，在「消息服务Kafka」中查看实例
腾讯云：登录腾讯云控制台，在「消息队列CKafka」中查看实例

六、总结#

6.1 方案优势#

标准化：使用Terraform定义基础设施，实现配置标准化
模块化：封装可重用的模块，提高代码复用性和可维护性
多云支持：支持阿里云和腾讯云，实现多云管理统一
自动化：自动化基础设施部署和管理，减少人为错误
可追溯：使用Git管理配置，实现基础设施变更的可追溯性

6.2 适用场景#

多环境部署：开发、测试、生产环境的标准化部署
多云管理：同时管理多个云服务提供商的资源
大规模部署：需要批量创建和管理大量资源的场景
DevOps实践：将基础设施纳入CI/CD流程，实现持续部署

花日の博客

Terraform多云Kafka实例部署方案详解#

一、Terraform基础知识#

1.1 什么是Terraform？#

1.2 核心概念#

1.3 工作流程#

二、项目结构设计#

2.1 整体结构#

2.2 结构说明#

三、模块定义详解#

3.1 阿里云Kafka模块#

3.1.1 主配置文件 (/modules/aliyun/aliyun_alikafka_mod_demo/main.tf)#

3.1.2 变量定义 (/modules/aliyun/aliyun_alikafka_mod_demo/variables.tf)#

3.2 腾讯云Kafka模块#

3.2.1 主配置文件 (/modules/tcloud/tcloud_ckafka_mod_demo/main.tf)#

3.2.2 变量定义 (/modules/tcloud/tcloud_ckafka_mod_demo/variables.tf)#

3.2.3 输出定义 (/modules/tcloud/tcloud_ckafka_mod_demo/outputs.tf)#

四、实例使用方法#

4.1 阿里云Kafka实例#

4.1.1 后端配置 (/aliyun/aliyun_alikafka_demo/aliyun_china_pt_7111/alikafka_demo_202502061910/backend.tf)#

4.1.2 主配置 (/aliyun/aliyun_alikafka_demo/aliyun_china_pt_7111/alikafka_demo_202502061910/main.tf)#

4.2 腾讯云Kafka实例#

4.2.1 后端配置 (/tcloud/tcloud_ckafka_demo/tcloud_china_game_x6/ckafaka_demo_202501221738/backend.tf)#

4.2.2 主配置 (/tcloud/tcloud_ckafka_demo/tcloud_china_game_x6/ckafaka_demo_202501221738/main.tf)#

五、运行测试步骤#

5.1 准备工作#

5.1.1 获取云服务提供商的API密钥#

5.1.2 设置环境变量#

5.2 运行Terraform命令#

5.2.1 初始化项目#

5.2.2 预览变更#

5.2.3 应用变更#

5.2.4 销毁资源#

5.3 验证结果#

六、总结#

6.1 方案优势#

6.2 适用场景#

3.1.1 主配置文件 (`/modules/aliyun/aliyun_alikafka_mod_demo/main.tf`)#

3.1.2 变量定义 (`/modules/aliyun/aliyun_alikafka_mod_demo/variables.tf`)#

3.2.1 主配置文件 (`/modules/tcloud/tcloud_ckafka_mod_demo/main.tf`)#

3.2.2 变量定义 (`/modules/tcloud/tcloud_ckafka_mod_demo/variables.tf`)#

3.2.3 输出定义 (`/modules/tcloud/tcloud_ckafka_mod_demo/outputs.tf`)#

4.1.1 后端配置 (`/aliyun/aliyun_alikafka_demo/aliyun_china_pt_7111/alikafka_demo_202502061910/backend.tf`)#

4.1.2 主配置 (`/aliyun/aliyun_alikafka_demo/aliyun_china_pt_7111/alikafka_demo_202502061910/main.tf`)#

4.2.1 后端配置 (`/tcloud/tcloud_ckafka_demo/tcloud_china_game_x6/ckafaka_demo_202501221738/backend.tf`)#

4.2.2 主配置 (`/tcloud/tcloud_ckafka_demo/tcloud_china_game_x6/ckafaka_demo_202501221738/main.tf`)#