Elasticsearch7.6 集群部署、集群认证及使用

博主： hdy
发布时间：2020 年 02 月 20 日
19508 次浏览
14921字数
分类：架构后端运维

Elasticsearch7.6 集群部署、集群认证及使用

window版本测试部署，centos使用docker-compose部署，springboot连接使用，集群配置ssl和CA证书配置

配置参考
Elasticsearch的一些概念和配置参数说明
Elasticsearch集群内部配置TLS加密通信及身份验证

window 环境部署集群

elasticsearch 下载地址
kibana 下载地址

注意：window下载解压elasticsearch一定需要解压多次。例如搭建的3节点的，需要解压3次，防止生成 cluster UUID 一致导致只能看到一个节点

1、elasticsearch.yml配置：

node.name区别：elastic_node1、 elastic_node2、 elastic_node3

cluster.name: elastic_cluster

node.name: elastic_node1

node.master: true
node.data: true

#path.data: /usr/local/elastic_node1/data
#path.logs: /usr/local/elastic_node1/logs

bootstrap.memory_lock: true

network.host: 127.0.0.1
network.tcp.no_delay: true
network.tcp.keep_alive: true
network.tcp.reuse_address: true
network.tcp.send_buffer_size: 256mb
network.tcp.receive_buffer_size: 256mb

transport.tcp.port: 9301
transport.tcp.compress: true

http.max_content_length: 200mb
http.cors.enabled: true
http.cors.allow-origin: "*"
http.port: 9201

discovery.seed_hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
cluster.initial_master_nodes: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
cluster.fault_detection.leader_check.interval: 15s
discovery.cluster_formation_warning_timeout: 30s
cluster.join.timeout: 30s
cluster.publish.timeout: 90s
cluster.routing.allocation.cluster_concurrent_rebalance: 16
cluster.routing.allocation.node_concurrent_recoveries: 16
cluster.routing.allocation.node_initial_primaries_recoveries: 16

2、依次运行生成集群

浏览器打开：http://127.0.0.1:9201/_cat/nodes?v

ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1           19          64  37                          dilm      -      elastic_node3
127.0.0.1           13          64  47                          dilm      -      elastic_node1
127.0.0.1           22          64  50                          dilm      *      elastic_node2

3、生成证书

es集群通过证书来安全的组成集群

运行

bin/elasticsearch-certutil cert

注意：密码后面需要单独设置，这里是集群安全认证，建议密码不设置，成功后生成的证书默认在es的config目录里面 elastic-certificates.p12；分别copy一份到其他节点的config里面（默认目录）

在elasticsearch.yml配置添加

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

4、给认证的集群创建用户密码

bin/elasticsearch-setup-passwords interactive

elastic 账号：拥有 superuser 角色，是内置的超级用户。
kibana 账号：拥有 kibana_system 角色，用户 kibana 用来连接 elasticsearch 并与之通信。Kibana 服务器以该用户身份提交请求以访问集群监视 API 和 .kibana 索引。不能访问 index。
logstash_system 账号：拥有 logstash_system 角色。用户 Logstash 在 Elasticsearch 中存储监控信息时使用。
beats_system账号：拥有 beats_system 角色。用户 Beats 在 Elasticsearch 中存储监控信息时使用。

elastic是超级用户

5、配置kibana认证

elasticsearch.username: "kibana"
elasticsearch.password: "123456"

完整的elasticsearch.yml配置，注意不同节点node.name区别

cluster.name: elastic_cluster
node.name: elastic_node1
node.master: true
node.data: true

#path.data: /usr/local/elastic_node1/data
#path.logs: /usr/local/elastic_node1/logs

bootstrap.memory_lock: true

network.host: 127.0.0.1
network.tcp.no_delay: true
network.tcp.keep_alive: true
network.tcp.reuse_address: true
network.tcp.send_buffer_size: 256mb
network.tcp.receive_buffer_size: 256mb

transport.tcp.port: 9302
transport.tcp.compress: true

http.max_content_length: 200mb
http.cors.enabled: true
http.cors.allow-origin: "*"
http.port: 9202

discovery.seed_hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
cluster.initial_master_nodes: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]
cluster.fault_detection.leader_check.interval: 15s
discovery.cluster_formation_warning_timeout: 30s
cluster.join.timeout: 30s
cluster.publish.timeout: 90s
cluster.routing.allocation.cluster_concurrent_rebalance: 16
cluster.routing.allocation.node_concurrent_recoveries: 16
cluster.routing.allocation.node_initial_primaries_recoveries: 16

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

centos(docker-compose) 环境部署集群

1、docker-compose.yml配置

version: '2.2'
services:
  es01:
    image: elasticsearch:7.6.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=192.168.43.128:9300
      - cluster.initial_master_nodes=es01,192.168.43.128:9300
      - cluster.fault_detection.leader_check.interval=15s
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - network.host=0.0.0.0
      - network.publish_host=192.168.43.129
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./mnt/data:/usr/share/elasticsearch/data
      - ./mnt/logs:/usr/share/elasticsearch/logs
    ports:
      - 9200:9200
      - 9300:9300
    networks:
      - elastic
  cerebro:
    image: lmenezes/cerebro:0.8.3
    container_name: cerebro
    ports:
      - "9000:9000"
    command:
      - -Dhosts.0.host=http://es01:9200
    networks:
      - elastic
volumes:
  mnt:
    driver: local

networks:
  elastic:
    driver: bridge

权限问题执行 chmod -R 777 mnt/*

2、生成证书文件创建密码

进入容器 docker exec -it 5144d3b1dd56 /bin/bash
生成证书 bin/elasticsearch-certutil cert
复制证书并cp到其他节点 docker cp 09f57b6067e0:/usr/share/elasticsearch/elastic-certificates.p12 .

3、修改配置&&动态添加测试

version: '2.2'
services:
  es01:
    image: elasticsearch:7.6.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=192.168.43.128:9300
      - cluster.initial_master_nodes=es01,192.168.43.128:9300
      - cluster.fault_detection.leader_check.interval=15s
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - network.host=0.0.0.0
      - network.publish_host=192.168.43.129
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.license.self_generated.type=basic
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=elastic-certificates.p12
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./mnt/data:/usr/share/elasticsearch/data
      - ./mnt/logs:/usr/share/elasticsearch/logs
      - ./mnt/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
    ports:
      - 9200:9200
      - 9300:9300
    networks:
      - elastic
  cerebro:
    image: lmenezes/cerebro:0.8.3
    container_name: cerebro
    ports:
      - "9000:9000"
    command:
      - -Dhosts.0.host=http://es01:9200
    networks:
      - elastic
volumes:
  mnt:
    driver: local

networks:
  elastic:
    driver: bridge

注意证书的位置，给权限 chmod -R 777 mnt/*

设置密码(建议进入主节点容器中) bin/elasticsearch-setup-passwords interactive -u 'http://es01:9200'
通用配置与window类似

springboot使用测试

1、引入pom

  <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.6.0</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>7.6.0</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.6.0</version>
        </dependency>

2、代码

EsConfiguration.class

/**
 * @author hdy
 */
@Configuration
public class EsConfiguration {
    /**
     * 集群地址
     */
    private static String hosts = "192.168.43.128";
    private static String hosts1 = "192.168.43.129";
    private static String hosts2 = "192.168.43.130";
    /**
     * 使用的端口号
     */
    private static int port = 9200;
    /**
     * // 使用的协议
     */
    private static String schema = "http";
    private static ArrayList<HttpHost> hostList = null;
    /**
     * 连接超时时间
     */
    private static int connectTimeOut = 1000;
    /**
     * 连接超时时间
     */
    private static int socketTimeOut = 30000;
    /**
     * 获取连接的超时时间
     */
    private static int connectionRequestTimeOut = 500;
    /**
     * 最大连接数
     */
    private static int maxConnectNum = 100;
    /**
     * 最大路由连接数
     */
    private static int maxConnectPerRoute = 100;

    private RestClientBuilder builder;

    private final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();

    static {
        hostList = new ArrayList<>();
        hostList.add(new HttpHost(hosts, port, schema));
        hostList.add(new HttpHost(hosts1, port, schema));
        hostList.add(new HttpHost(hosts2, port, schema));
    }

    @Bean("restHighLevelClient")
    public RestHighLevelClient client() {
        builder = RestClient.builder(hostList.toArray(new HttpHost[0]));
        setConnectTimeOutConfig();
        setMutiConnectConfig();
        return new RestHighLevelClient(builder);
    }

    /**
     * 异步httpclient的连接延时配置
     */
    private void setConnectTimeOutConfig() {
        builder.setRequestConfigCallback(requestConfigBuilder -> {
            requestConfigBuilder.setConnectTimeout(connectTimeOut);
            requestConfigBuilder.setSocketTimeout(socketTimeOut);
            requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeOut);
            return requestConfigBuilder;
        });
    }

    /**
     * 异步httpclient的连接数配置
     */
    private void setMutiConnectConfig() {
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("elastic", "123456"));
        builder.setHttpClientConfigCallback(httpClientBuilder -> {
            httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            httpClientBuilder.setMaxConnTotal(maxConnectNum);
            httpClientBuilder.setMaxConnPerRoute(maxConnectPerRoute);
            return httpClientBuilder;
        });
    }

}

ElasticsearchApplicationTests.class

@Log4j2
@RunWith(SpringRunner.class)
@SpringBootTest
public class ElasticsearchApplicationTests {

    @Autowired
    RestHighLevelClient restHighLevelClient;

    @Test
    public void contextLoads() {

        for (int i = 1000000; i >= 0; i--) {
            Map<String, Object> jsonMap = new HashMap<>();
            jsonMap.put("name", "测试" + i);
            jsonMap.put("age", "" + i);
            jsonMap.put("des", "啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦拉了拉");
            jsonMap.put("des1", "des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1");
            jsonMap.put("des2", "des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2");
            jsonMap.put("des3", "des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3");
            jsonMap.put("des4", "des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4");
            jsonMap.put("postDate", new Date());
            jsonMap.put("message", "trying out Elasticsearch");
            IndexRequest indexRequest = new IndexRequest("test").id("" + i).source(jsonMap);
            try {
                IndexResponse response = null;
                try {
                    response = restHighLevelClient.index(indexRequest, RequestOptions.DEFAULT);
                } catch (IOException e) {
                    e.printStackTrace();
                }
                log.info(response.toString());
            } catch (ElasticsearchException e) {
                if (e.status() == RestStatus.CONFLICT) {
                    System.out.println("e = " + e);
                }
            }
        }

        GetRequest getRequest = new GetRequest("posts", "2");
        GetResponse response = null;
        try {
            response = restHighLevelClient.get(getRequest, RequestOptions.DEFAULT);
            log.info(response.toString());
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

}

错误信息,少引入elasticsearch-rest-client pom包

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'restHighLevelClient' defined in class path resource [com/dy/client/EsConfiguration.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [org.elasticsearch.client.RestHighLevelClient] from ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:570) ~[spring-beans-5.1.13.RELEASE.jar:5.1.13.RELEASE]
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) [junit-rt.jar:na]
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) [junit-rt.jar:na]
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.elasticsearch.client.RestHighLevelClient] from ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:507) ~[spring-core-5.1.13.RELEASE.jar:5.1.13.RELEASE]
    ... 38 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/client/Cancellable
    at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_191]
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:489) ~[spring-core-5.1.13.RELEASE.jar:5.1.13.RELEASE]
    ... 45 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.client.Cancellable
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_191]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_191]
    ... 49 common frames omitted

最后修改：2020 年 02 月 20 日

如果觉得我的文章对你有用，请随意赞赏

Elasticsearch7.6 集群部署、集群认证及使用

hdy • 2020 年 02 月 20 日

<h1>Elasticsearch7.6 集群部署、集群认证及使用</h1><blockquote>window版本测试部署，centos使用docker-compose部署，springboot连接使用，集群配置ssl和CA证书配置</blockquote><ul><li>配置参考<br><span class="external-link"><a class="no-external-link" href="http://blog.yrclubs.com/archives/51/" target="_blank"><i data-feather="external-link"></i>Elasticsearch的一些概念和配置参数说明</a></span><br><span class="external-link"><a class="no-external-link" href="http://blog.yrclubs.com/archives/50/" target="_blank"><i data-feather="external-link"></i>Elasticsearch集群内部配置TLS加密通信及身份验证</a></span></li></ul><h2>window 环境部署集群</h2><p><span class="external-link"><a class="no-external-link" href="https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.0-windows-x86_64.zip" target="_blank"><i data-feather="external-link"></i>elasticsearch 下载地址</a></span><br><span class="external-link"><a class="no-external-link" href="https://artifacts.elastic.co/downloads/kibana/kibana-7.6.0-windows-x86_64.zip" target="_blank"><i data-feather="external-link"></i>kibana  下载地址</a></span></p><blockquote>注意：window下载解压elasticsearch一定需要解压多次。例如搭建的3节点的，需要解压3次，防止生成 cluster UUID 一致导致只能看到一个节点</blockquote><h4>1、elasticsearch.yml配置：</h4><blockquote>node.name区别：elastic_node1、 elastic_node2、 elastic_node3</blockquote><pre><code>cluster.name: elastic_cluster

node.name: elastic_node1

node.master: true
node.data: true

#path.data: /usr/local/elastic_node1/data
#path.logs: /usr/local/elastic_node1/logs

bootstrap.memory_lock: true

network.host: 127.0.0.1
network.tcp.no_delay: true
network.tcp.keep_alive: true
network.tcp.reuse_address: true
network.tcp.send_buffer_size: 256mb
network.tcp.receive_buffer_size: 256mb

transport.tcp.port: 9301
transport.tcp.compress: true

http.max_content_length: 200mb
http.cors.enabled: true
http.cors.allow-origin: &quot;*&quot;
http.port: 9201

discovery.seed_hosts: [&quot;127.0.0.1:9301&quot;,&quot;127.0.0.1:9302&quot;,&quot;127.0.0.1:9303&quot;]
cluster.initial_master_nodes: [&quot;127.0.0.1:9301&quot;,&quot;127.0.0.1:9302&quot;,&quot;127.0.0.1:9303&quot;]
cluster.fault_detection.leader_check.interval: 15s
discovery.cluster_formation_warning_timeout: 30s
cluster.join.timeout: 30s
cluster.publish.timeout: 90s
cluster.routing.allocation.cluster_concurrent_rebalance: 16
cluster.routing.allocation.node_concurrent_recoveries: 16
cluster.routing.allocation.node_initial_primaries_recoveries: 16
</code></pre><h4>2、依次运行生成集群</h4><p>浏览器打开：<span class="external-link"><a class="no-external-link" href="http://127.0.0.1:9201/_cat/nodes?v" target="_blank"><i data-feather="external-link"></i>http://127.0.0.1:9201/_cat/nodes?v</a></span></p><pre><code>ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1           19          64  37                          dilm      -      elastic_node3
127.0.0.1           13          64  47                          dilm      -      elastic_node1
127.0.0.1           22          64  50                          dilm      *      elastic_node2</code></pre><h4>3、生成证书</h4><p><strong>es集群通过证书来安全的组成集群</strong></p><ul><li>运行</li></ul><pre><code>bin/elasticsearch-certutil cert</code></pre><blockquote>注意： 密码后面需要单独设置，这里是集群安全认证，建议密码不设置，成功后生成的证书默认在es的config目录里面 elastic-certificates.p12；分别copy一份到其他节点的config里面（默认目录）</blockquote><p>在elasticsearch.yml配置添加</p><pre><code>xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12</code></pre><h4>4、给认证的集群创建用户密码</h4><pre><code>bin/elasticsearch-setup-passwords interactive</code></pre><ul><li>elastic 账号：拥有 superuser 角色，是内置的超级用户。</li><li>kibana 账号：拥有 kibana_system 角色，用户 kibana 用来连接 elasticsearch 并与之通信。Kibana 服务器以该用户身份提交请求以访问集群监视 API 和 .kibana 索引。不能访问 index。</li><li>logstash_system 账号：拥有 logstash_system 角色。用户 Logstash 在 Elasticsearch 中存储监控信息时使用。</li><li>beats_system账号：拥有 beats_system 角色。用户 Beats 在 Elasticsearch 中存储监控信息时使用。</li></ul><blockquote>elastic是超级用户</blockquote><h4>5、配置kibana认证</h4><pre><code>elasticsearch.username: &quot;kibana&quot;
elasticsearch.password: &quot;123456&quot;</code></pre><ul><li>完整的elasticsearch.yml配置，注意不同节点node.name区别</li></ul><pre><code>cluster.name: elastic_cluster
node.name: elastic_node1
node.master: true
node.data: true

#path.data: /usr/local/elastic_node1/data
#path.logs: /usr/local/elastic_node1/logs

bootstrap.memory_lock: true

network.host: 127.0.0.1
network.tcp.no_delay: true
network.tcp.keep_alive: true
network.tcp.reuse_address: true
network.tcp.send_buffer_size: 256mb
network.tcp.receive_buffer_size: 256mb

transport.tcp.port: 9302
transport.tcp.compress: true

http.max_content_length: 200mb
http.cors.enabled: true
http.cors.allow-origin: &quot;*&quot;
http.port: 9202

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12</code></pre><h2>centos(docker-compose) 环境部署集群</h2><h4>1、docker-compose.yml配置</h4><pre><code class="lang-yaml">version: '2.2'
services:
  es01:
    image: elasticsearch:7.6.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=192.168.43.128:9300
      - cluster.initial_master_nodes=es01,192.168.43.128:9300
      - cluster.fault_detection.leader_check.interval=15s
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - network.host=0.0.0.0
      - network.publish_host=192.168.43.129
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - &quot;ES_JAVA_OPTS=-Xms256m -Xmx256m&quot;
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./mnt/data:/usr/share/elasticsearch/data
      - ./mnt/logs:/usr/share/elasticsearch/logs
    ports:
      - 9200:9200
      - 9300:9300
    networks:
      - elastic
  cerebro:
    image: lmenezes/cerebro:0.8.3
    container_name: cerebro
    ports:
      - &quot;9000:9000&quot;
    command:
      - -Dhosts.0.host=http://es01:9200
    networks:
      - elastic
volumes:
  mnt:
    driver: local

networks:
  elastic:
    driver: bridge
</code></pre><blockquote>权限问题执行 chmod -R 777 mnt/*</blockquote><h4>2、生成证书文件创建密码</h4><ul><li>进入容器  docker exec -it 5144d3b1dd56 /bin/bash</li><li>生成证书  bin/elasticsearch-certutil cert</li><li>复制证书并cp到其他节点   docker cp 09f57b6067e0:/usr/share/elasticsearch/elastic-certificates.p12 .</li></ul><h4>3、修改配置&&动态添加测试</h4><pre><code>version: '2.2'
services:
  es01:
    image: elasticsearch:7.6.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=192.168.43.128:9300
      - cluster.initial_master_nodes=es01,192.168.43.128:9300
      - cluster.fault_detection.leader_check.interval=15s
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - network.host=0.0.0.0
      - network.publish_host=192.168.43.129
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.license.self_generated.type=basic
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=elastic-certificates.p12
      - &quot;ES_JAVA_OPTS=-Xms256m -Xmx256m&quot;
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./mnt/data:/usr/share/elasticsearch/data
      - ./mnt/logs:/usr/share/elasticsearch/logs
      - ./mnt/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
    ports:
      - 9200:9200
      - 9300:9300
    networks:
      - elastic
  cerebro:
    image: lmenezes/cerebro:0.8.3
    container_name: cerebro
    ports:
      - &quot;9000:9000&quot;
    command:
      - -Dhosts.0.host=http://es01:9200
    networks:
      - elastic
volumes:
  mnt:
    driver: local

networks:
  elastic:
    driver: bridge
</code></pre><blockquote>注意证书的位置，给权限 chmod -R 777 mnt/*</blockquote><ul><li>设置密码(建议进入主节点容器中)  bin/elasticsearch-setup-passwords interactive -u '<span class="external-link"><a class="no-external-link" href="http://es01:9200" target="_blank"><i data-feather="external-link"></i>http://es01:9200</a></span>'</li><li>通用配置与window类似</li></ul><h2>springboot使用测试</h2><h3>1、引入pom</h3><pre><code>  &lt;dependency&gt;
            &lt;groupId&gt;org.elasticsearch&lt;/groupId&gt;
            &lt;artifactId&gt;elasticsearch&lt;/artifactId&gt;
            &lt;version&gt;7.6.0&lt;/version&gt;
        &lt;/dependency&gt;

&lt;dependency&gt;
            &lt;groupId&gt;org.elasticsearch.client&lt;/groupId&gt;
            &lt;artifactId&gt;elasticsearch-rest-client&lt;/artifactId&gt;
            &lt;version&gt;7.6.0&lt;/version&gt;
        &lt;/dependency&gt;

&lt;dependency&gt;
            &lt;groupId&gt;org.elasticsearch.client&lt;/groupId&gt;
            &lt;artifactId&gt;elasticsearch-rest-high-level-client&lt;/artifactId&gt;
            &lt;version&gt;7.6.0&lt;/version&gt;
        &lt;/dependency&gt;</code></pre><h3>2、 代码</h3><ol><li>EsConfiguration.class</li></ol><pre><code class="lang-java">/**
 * @author hdy
 */
@Configuration
public class EsConfiguration {
    /**
     * 集群地址
     */
    private static String hosts = &quot;192.168.43.128&quot;;
    private static String hosts1 = &quot;192.168.43.129&quot;;
    private static String hosts2 = &quot;192.168.43.130&quot;;
    /**
     * 使用的端口号
     */
    private static int port = 9200;
    /**
     * // 使用的协议
     */
    private static String schema = &quot;http&quot;;
    private static ArrayList&lt;HttpHost&gt; hostList = null;
    /**
     * 连接超时时间
     */
    private static int connectTimeOut = 1000;
    /**
     * 连接超时时间
     */
    private static int socketTimeOut = 30000;
    /**
     * 获取连接的超时时间
     */
    private static int connectionRequestTimeOut = 500;
    /**
     * 最大连接数
     */
    private static int maxConnectNum = 100;
    /**
     * 最大路由连接数
     */
    private static int maxConnectPerRoute = 100;

private RestClientBuilder builder;

private final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();

static {
        hostList = new ArrayList&lt;&gt;();
        hostList.add(new HttpHost(hosts, port, schema));
        hostList.add(new HttpHost(hosts1, port, schema));
        hostList.add(new HttpHost(hosts2, port, schema));
    }

@Bean(&quot;restHighLevelClient&quot;)
    public RestHighLevelClient client() {
        builder = RestClient.builder(hostList.toArray(new HttpHost[0]));
        setConnectTimeOutConfig();
        setMutiConnectConfig();
        return new RestHighLevelClient(builder);
    }

/**
     * 异步httpclient的连接延时配置
     */
    private void setConnectTimeOutConfig() {
        builder.setRequestConfigCallback(requestConfigBuilder -&gt; {
            requestConfigBuilder.setConnectTimeout(connectTimeOut);
            requestConfigBuilder.setSocketTimeout(socketTimeOut);
            requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeOut);
            return requestConfigBuilder;
        });
    }

/**
     * 异步httpclient的连接数配置
     */
    private void setMutiConnectConfig() {
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(&quot;elastic&quot;, &quot;123456&quot;));
        builder.setHttpClientConfigCallback(httpClientBuilder -&gt; {
            httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            httpClientBuilder.setMaxConnTotal(maxConnectNum);
            httpClientBuilder.setMaxConnPerRoute(maxConnectPerRoute);
            return httpClientBuilder;
        });
    }

}
</code></pre><ol start="2"><li>ElasticsearchApplicationTests.class</li></ol><pre><code class="lang-java">@Log4j2
@RunWith(SpringRunner.class)
@SpringBootTest
public class ElasticsearchApplicationTests {

@Autowired
    RestHighLevelClient restHighLevelClient;

@Test
    public void contextLoads() {

for (int i = 1000000; i &gt;= 0; i--) {
            Map&lt;String, Object&gt; jsonMap = new HashMap&lt;&gt;();
            jsonMap.put(&quot;name&quot;, &quot;测试&quot; + i);
            jsonMap.put(&quot;age&quot;, &quot;&quot; + i);
            jsonMap.put(&quot;des&quot;, &quot;啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦啦拉了拉&quot;);
            jsonMap.put(&quot;des1&quot;, &quot;des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1des1&quot;);
            jsonMap.put(&quot;des2&quot;, &quot;des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2des2&quot;);
            jsonMap.put(&quot;des3&quot;, &quot;des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3des3&quot;);
            jsonMap.put(&quot;des4&quot;, &quot;des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4des4&quot;);
            jsonMap.put(&quot;postDate&quot;, new Date());
            jsonMap.put(&quot;message&quot;, &quot;trying out Elasticsearch&quot;);
            IndexRequest indexRequest = new IndexRequest(&quot;test&quot;).id(&quot;&quot; + i).source(jsonMap);
            try {
                IndexResponse response = null;
                try {
                    response = restHighLevelClient.index(indexRequest, RequestOptions.DEFAULT);
                } catch (IOException e) {
                    e.printStackTrace();
                }
                log.info(response.toString());
            } catch (ElasticsearchException e) {
                if (e.status() == RestStatus.CONFLICT) {
                    System.out.println(&quot;e = &quot; + e);
                }
            }
        }

GetRequest getRequest = new GetRequest(&quot;posts&quot;, &quot;2&quot;);
        GetResponse response = null;
        try {
            response = restHighLevelClient.get(getRequest, RequestOptions.DEFAULT);
            log.info(response.toString());
        } catch (IOException e) {
            e.printStackTrace();
        }

}

}</code></pre><ul><li>错误信息,少引入elasticsearch-rest-client pom包</li></ul><pre><code>org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'restHighLevelClient' defined in class path resource [com/dy/client/EsConfiguration.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [org.elasticsearch.client.RestHighLevelClient] from ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:570) ~[spring-beans-5.1.13.RELEASE.jar:5.1.13.RELEASE]
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) [junit-rt.jar:na]
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) [junit-rt.jar:na]
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.elasticsearch.client.RestHighLevelClient] from ClassLoader [sun.misc.Launcher$AppClassLoader@18b4aac2]
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:507) ~[spring-core-5.1.13.RELEASE.jar:5.1.13.RELEASE]
    ... 38 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/client/Cancellable
    at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_191]
    at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:489) ~[spring-core-5.1.13.RELEASE.jar:5.1.13.RELEASE]
    ... 45 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.client.Cancellable
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_191]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_191]
    ... 49 common frames omitted</code></pre>

Elasticsearch7.6 集群部署、集群认证及使用

Elasticsearch7.6 集群部署、集群认证及使用

window 环境部署集群

1、elasticsearch.yml配置：

2、依次运行生成集群

3、生成证书

4、给认证的集群创建用户密码

5、配置kibana认证

centos(docker-compose) 环境部署集群

1、docker-compose.yml配置

2、生成证书文件创建密码

3、修改配置&&动态添加测试

springboot使用测试

1、引入pom

2、代码

docker 搭建typecho

dockercompose搭建nacos集群

docker-compose + canal + mysql + redis

linux 使用某个角色运行程序

CentOS 7.0防火墙设置和关闭

Nexus 批量导入jar

elasticsearch-rest-client 7.1.1 入手

Linux Centos 7 - Nginx离线安装

org.elasticsearch.action.index.IndexRequest.ifSeqNo

分布式FastDFS集群部署

Elasticsearch7.6 集群部署、集群认证及使用

Elasticsearch7.6 集群部署、集群认证及使用

window 环境部署集群

1、elasticsearch.yml配置：

2、依次运行生成集群

3、生成证书

4、给认证的集群创建用户密码

5、配置kibana认证

centos(docker-compose) 环境部署集群

1、docker-compose.yml配置

2、生成证书文件创建密码

3、修改配置&&动态添加测试

springboot使用测试

1、引入pom

2、 代码

Elasticsearch7.6 集群部署、集群认证及使用

2、代码