ElasticSearch2.4.0单机版及集群安装

ShihLei

浏览: 333716 次
性别:
来自: 北京

最近访客更多访客>>

webinweb

xiaomabobo

我怀念的不过是镜花水月

meixianping

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

elasticsearch elasiicsarch v2.4 cluster

编写不易，转载请注明（ http://shihlei.iteye.com/blog/2327971）！

概述

（一）简介

基于lucene 实现的近实时搜索服务，Restful 接口操作

（二）名词解释

Indics：索引库，相当于RDBMS的数据库，整体控制分片（shard）和副本（Replica），一旦创建，分片不能改变。

Document type：索引类型

Document：索引记录，由唯一ID区分，ID决定Shard位置

Filed：一条索引记录中的组成字段，有类型概念，通过Mapping控制类型，是否分词，索引等

一单机版安装

（一）环境

环境要求：

java 7 以上

环境：

CentOS Linux release 7.2.1511 (Core)

jre：1.8.0_101

版本：

elasticsearch-2.4.0.tar.gz

核心配置文件：

elasticsearch-2.4.0/config/elasticsearch.yml : yml协议的默认配置文件

elasticsearch-2.4.0/config/logging.xml ：yml协议格式的日志配置文件

（二）安装

1）创建用户

groupadd es

useradd -r -g es -s /bin/bash es

chown -hR es:es /opt/elasticsearch-2.4.0

注：由于不能用Root用户启动，否则报java.lang.RuntimeException，所以创建用户

Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.

at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:160)

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.

2）解压缩

tar -xvf elasticsearch-2.4.0.tar.gz

3）启动

cd /opt/elasticsearch-2.4.0

bin/elasticsearch

日志：

[2016-09-22 08:45:21,503][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: your kernel is buggy and you should upgrade

[2016-09-22 08:45:21,608][INFO ][node ] [Deacon Frost] version[2.4.0], pid[205], build[ce9f0c7/2016-08-29T09:14:17Z]

[2016-09-22 08:45:21,608][INFO ][node ] [Deacon Frost] initializing ...

[2016-09-22 08:45:21,977][INFO ][plugins ] [Deacon Frost] modules [reindex, lang-expression, lang-groovy], plugins [], sites []

[2016-09-22 08:45:21,994][INFO ][env ] [Deacon Frost] using [1] data paths, mounts [[/ (none)]], net usable_space [45gb], net total_space [59gb], spins? [possibly], types [aufs]

[2016-09-22 08:45:21,994][INFO ][env ] [Deacon Frost] heap size [998.4mb], compressed ordinary object pointers [unknown]

[2016-09-22 08:45:23,147][INFO ][node ] [Deacon Frost] initialized

[2016-09-22 08:45:23,148][INFO ][node ] [Deacon Frost] starting ...

[2016-09-22 08:45:23,251][INFO ][transport ] [Deacon Frost] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}

[2016-09-22 08:45:23,255][INFO ][discovery ] [Deacon Frost] elasticsearch/6TSRp8FFQ3aXBPtkS4sROQ

[2016-09-22 08:45:26,316][INFO ][cluster.service ] [Deacon Frost] new_master {Deacon Frost}{6TSRp8FFQ3aXBPtkS4sROQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)

[2016-09-22 08:45:26,341][INFO ][http ] [Deacon Frost] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}

[2016-09-22 08:45:26,343][INFO ][node ] [Deacon Frost] started

[2016-09-22 08:45:26,359][INFO ][gateway ] [Deacon Frost] recovered [0] indices into cluster_state

注1 ：后台启动方法：bin/elasticsearch -d

注2 : 这里绑定了两个端口

9200：默认的Http端口，用于Restful API的调用

9300：默认TCP端口，用户ES Node 之间的通讯，以及TCP 基本调用 Restful API ， Java API TransportClient 需要连接该端口才能访问

注3：默认绑定127.0.0.1 本机访问，

远程访问需修改 ${ES_HOME}/config/elasticsearch.yml 的 network.host绑定指定的IP

# ---------------------------------- Network -----------------------------------

# Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 0.0.0.0

（三）验证

浏览器：http://127.0.0.1:9200 返回

{

"name" : "Abomination",

"cluster_name" : "elasticsearch",

"version" : {

"number" : "2.4.0",

"build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",

"build_timestamp" : "2016-08-29T09:14:17Z",

"build_snapshot" : false,

"lucene_version" : "5.5.2"

"tagline" : "You Know, for Search"

}

由节点名称等，说明安装成功

（四）简单操作

（1）插入：

$ curl -X PUT 'http://localhost:9200/indexdb/type/1' -d '

> {

> "name":"test",

> "desc":"test"

> }

> '

{"_index":"indexdb","_type":"type","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":

（2）查询：

$ curl -XGET 'http://localhost:9200/indexdb/type/1'

{"_index":"indexdb","_type":"type","_id":"1","_version":1,"found":true,"_source":

{

"name":"test",

"desc":"test"

}

（五）关闭

for pid in `jps | grep Elasticsearch | awk '{print $1}'`

kill -9 $pid

done;

（六）常用插件安装

（1）Head：

方便对es进行各种操作的客户端（https://github.com/mobz/elasticsearch-head）

bin/plugin install mobz/elasticsearch-head

访问：

http://127.0.0.1:9200/_plugin/head

（2）bigdesk：

监控es状态的插件（https://github.com/lukas-vlcek/bigdesk）

bigdesk 的源码开发在 2015 年夏天就已经停止，所以默认是无法支持 Elasticsearch 2.x 的监控的

二集群搭建

（一）概述

（1）ES Node 的角色：

Master Node : 通过选举出来，控制集群元数据和状态，压力较小，CRUD不需要通过（ node.master=true）

Data Node : 拥有数据，并控制数据的CRUD，有查询压力（node.data=true）

Client Node：只能路由，转发响应的请求给Master Node 或者Data Node ，做聚合操作，可以分担部分Data Node 的压力，成本合适，可以创建。

Trib Node : 联合节点，控制到多个集群的访问

（2）架构

ES 的所有Node 节点是对等的，可以服务于客户端的查询，索引请求。结点通过选举成为Master，维持Cluster 状态，Master 所有的Node 周知。

（二）集群搭建

（1）架构规划

说明：客户端通过 ClientNode 访问集群，ClientNode开放tcp，http访问，

所有DataNode置于后端，只开发TCP进行通讯

（2）目录规划

ES_HOME : /opt/elasticsearch-2.4.0

结点名称	配置文件目录	数据目录	日志目录	pid目录
ClientNode	cluster/config/ClientNode	cluster/data/ClientNode	cluster/logs/ClientNode	cluster/ClientNode.pid
DataNode1	cluster/config/DataNode1	cluster/data/DataNode1	cluster/logs/DataNode1	cluster/DataNode1.pid
DataNode2	cluster/config/DataNode2	cluster/data/DataNode2	cluster/logs/DataNode2	cluster/DataNode2.pid
DataNode3	cluster/config/DataNode3	cluster/data/DataNode3	cluster/logs/DataNode3	cluster/DataNode3.pid

（3）端口规划

结点名称	Http端口	TCP端口
ClientNode	9200	9300
DataNode1	9201	9301
DataNode2	9202	9302
DataNode2	9203	9303

（4）安装

步骤1：拷贝配置文件到约定目录：

cd /opt/elasticsearch-2.4.0

mkdir -p cluster/config/ClientNode && cp config/logging.yml cluster/config/ClientNode

mkdir -p cluster/config/DataNode1 && cp config/logging.yml cluster/config/DataNode1

mkdir -p cluster/config/DataNode2 && cp config/logging.yml cluster/config/DataNode2

mkdir -p cluster/config/DataNode3 && cp config/logging.yml cluster/config/DataNode3

步骤2：编辑个结点配置文件：

特别注意：ES的集群是通过自发现机制，需要cluster.name配置相同才会自动加入相同的集群，这里集群统一命名为ES_CLUSTER

下面的配置文件已经过删减，可以直接替换elasticsearch.yml

ClientNode：

ClientNode ：node.master , node.data标志都为false，并开启http访问

vi cluster/config/ClientNode/elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

cluster.name: ES_CLUSTER

# ------------------------------------ Node ------------------------------------

node.name: ClientNode

http.enabled: true

node.master: false

node.data: false

# ----------------------------------- Paths ------------------------------------

# Path to directory where to store the data (separate multiple locations by comma):

path.data: /opt/elasticsearch-2.4.0/cluster/data/ClientNode

# Path to log files:

path.logs: /opt/elasticsearch-2.4.0/cluster/logs/ClientNode

# ---------------------------------- Network -----------------------------------

# Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 0.0.0.0

# Set a custom port for HTTP:

http.port: 9200

# Set a custom port for TCP:

transport.tcp.port: 9300

# --------------------------------- Discovery ----------------------------------

# Pass an initial list of hosts to perform discovery when new node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

DataNode1：

可选举成为Master 的DataNode ： node.master,node.data都为true，由于客户端方位从ClientNode口出，http.enabled禁掉了下面同

vi cluster/config/DataNode1/elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

cluster.name: ES_CLUSTER

# ------------------------------------ Node ------------------------------------

node.name: DataNode1

http.enabled: false

node.master: true

node.data: true

# ----------------------------------- Paths ------------------------------------

# Path to directory where to store the data (separate multiple locations by comma):

path.data: /opt/elasticsearch-2.4.0/cluster/data/DataNode1

# Path to log files:

path.logs: /opt/elasticsearch-2.4.0/cluster/logs/DataNode1

# ---------------------------------- Network -----------------------------------

# Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 0.0.0.0

# Set a custom port for HTTP:

http.port: 9201

# Set a custom port for TCP:

transport.tcp.port: 9301

# --------------------------------- Discovery ----------------------------------

# Pass an initial list of hosts to perform discovery when new node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

DataNode2

vi cluster/config/DataNode2/elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

cluster.name: ES_CLUSTER

# ------------------------------------ Node ------------------------------------

node.name: DataNode2

http.enabled: false

node.master: true

node.data: true

# ----------------------------------- Paths ------------------------------------

# Path to directory where to store the data (separate multiple locations by comma):

path.data: /opt/elasticsearch-2.4.0/cluster/data/DataNode2

# Path to log files:

path.logs: /opt/elasticsearch-2.4.0/cluster/logs/DataNode2

# ---------------------------------- Network -----------------------------------

# Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 0.0.0.0

# Set a custom port for HTTP:

http.port: 9202

# Set a custom port for TCP:

transport.tcp.port: 9302

# --------------------------------- Discovery ----------------------------------

# Pass an initial list of hosts to perform discovery when new node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

DataNode3

vi cluster/config/DataNode3/elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

cluster.name: ES_CLUSTER

# ------------------------------------ Node ------------------------------------

node.name: DataNode3

http.enabled: false

node.master: true

node.data: true

# ----------------------------------- Paths ------------------------------------

# Path to directory where to store the data (separate multiple locations by comma):

path.data: /opt/elasticsearch-2.4.0/cluster/data/DataNode3

# Path to log files:

path.logs: /opt/elasticsearch-2.4.0/cluster/logs/DataNode3

# ---------------------------------- Network -----------------------------------

# Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 0.0.0.0

# Set a custom port for HTTP:

http.port: 9203

# Set a custom port for TCP:

transport.tcp.port: 9303

# --------------------------------- Discovery ----------------------------------

# Pass an initial list of hosts to perform discovery when new node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

特别注意：——坑：

transport.tcp.port

参考：http://knktc.com/2016/06/10/elasticsearch-multiple-instances/?utm_source=tuicool&utm_medium=referral

由于到了2.x版本之后，ES取消了默认的广播模式来发现master节点，需要使用该配置来指定发现master节点。这个配置在单机双实例的配置中需要特别注意下，因为习惯上我们配置时并未指定master节点的tcp端口，如果实例的transport.tcp.port配置为9301，那么实例启动后会认为discovery.zen.ping.unicast.hosts中指定的主机tcp端口也是9301，可能导致这些节点无法找到master节点。因此在该配置中需要指定master节点提供服务的tcp端口。

所以配置了

discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

（5）启动

启动命令为：

bin/elasticsearch -Des.path.conf=cluster/config/ClientNode -d -p cluster/ClientNode.pid

bin/elasticsearch -Des.path.conf=cluster/config/DataNode1 -d -p cluster/DataNode1.pid

bin/elasticsearch -Des.path.conf=cluster/config/DataNode2 -d -p cluster/DataNode2.pid

bin/elasticsearch -Des.path.conf=cluster/config/DataNode3 -d -p cluster/DataNode3.pid

（6）状态：

bash-4.2$ jps -lm

3815 org.elasticsearch.bootstrap.Elasticsearch start -Des.path.conf=cluster/config/DataNode1 -d -p cluster/DataNode1.pid

4152 sun.tools.jps.Jps -lm

4089 org.elasticsearch.bootstrap.Elasticsearch start -Des.path.conf=cluster/config/ClientNode -d -p cluster/ClientNode.pid

3836 org.elasticsearch.bootstrap.Elasticsearch start -Des.path.conf=cluster/config/DataNode2 -d -p cluster/DataNode2.pid

3885 org.elasticsearch.bootstrap.Elasticsearch start -Des.path.conf=cluster/config/DataNode3 -d -p cluster/DataNode3.pid

ClientNode

{
"name" : "ClientNode",
"cluster_name" : "ES_CLUSTER",
"version" : {
"number" : "2.4.0",
"build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",
"build_timestamp" : "2016-08-29T09:14:17Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}