ElasticSearch 2.1.1 (2) - Setup

Installation

  • Elasticsearch can be started using:

      $ bin/elasticsearch
  • Running as a daemon

      $ bin/elasticsearch -d
  • PID

    The PID is written to a file called pid.

      $ bin/elasticsearch -d -p pid

    The kill command sends a TERM signal to the PID stored in the pid file.

      $ kill `cat pid`

    Another feature is the ability to pass -D or getopt long style configuration parameters directly to the script. When set, all override anything set using either JAVA_OPTS or ES_JAVA_OPTS. For example:

      $ bin/elasticsearch -Des.index.refresh_interval=5s --node.name=my-node

Configuration

Environment Variables

  • JAVA_OPTS

  • ES_JAVA_OPTS > JAVA_OPTS

  • ES_HEAP_SIZE

      ES_MIN_MEM (defaults to 256m)
    ES_MAX_MEM (defaults to 1g)

    It is recommended to set the min and max memory to the SAME value, and enable mlockall.

System Configuration

  • File Descriptors

    Setting it to 32k or even 64k is recommended.

      -Des.max-open-files=true

    max_file_descriptors

      curl localhost:9200/_nodes/stats/process?pretty
  • Virtual memory

    Elasticsearch uses a hybrid mmapfs / niofs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. On Linux, you can increase the limits by running the following command as root:

      sysctl -w vm.max_map_count=262144

    To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf.

  • Memory Settings

    Most operating systems try to use as much memory as possible for file system caches and eagerly swap out unused application memory, possibly resulting in the elasticsearch process being swapped. Swapping is very bad for performance and for node stability, so it should be avoided at all costs.

    • Disable swap

      • Box -

        ES_HEAP_SIZE

      • Linux -

        Temporarily

          sudo swapoff -a

        Permanently (comment out swap)

          /etc/fstab
      • Windows -

        System Properties → Advanced → Performance → Advanced → Virtual memory

    • Configure swappiness

      This reduces the kernel’s tendency to swap and should not lead to swapping under normal circumstances, while still allowing the whole system to swap in emergency conditions.

        vm.swappiness

      From kernel version 3.5-rc1 and above, a swappiness of 0 will cause the OOM killer to kill the process instead of allowing swapping. You will need to set swappiness to 1 to still allow swapping in emergencies.

    • mlockall

      • Linux - mlockall

          bootstrap.mlockall: true
      • Windows - VirtualLock

      Check:

        curl http://localhost:9200/_nodes/process?pretty

      Grant(as root):

        $ ulimit -l unlimited

      Directory(/tmp):

        ./bin/elasticsearch -Djna.tmpdir=/path/to/new/dir

      mlockall might cause the JVM or shell session to exit if it tries to allocate more memory than is available!

Elasticsearch Settings

  • Directory

      ES_HOME/config
    • module - elasticsearch.yml

    • logging - logging.yml

    • network

      The address all network based modules will use to bind and publish to:

        network :
      host : 10.0.0.4
  • Path

      path:
    logs: /var/log/elasticsearch
    data: /var/data/elasticsearch
  • Cluster name

    Don’t forget to give your production cluster a name, which is used to discover and auto-join other nodes:

      cluster:
    name: <NAME OF YOUR CLUSTER>

    Don’t reuse the same cluster names in different environment

    • development: logging-dev
    • staging: logging-stage
    • production: logging-prod
  • Node name

    • Default (Marvel Character Name)

    • Provided

        node:
      name: <NAME OF YOUR NODE>
    • Single Node on Machine

        node:
      name: ${HOSTNAME}
  • Configuration style

    • JSON

      elasticsearch.json:

        {
      "network" : {
      "host" : "10.0.0.4"
      }
      }
    • Command

        $ elasticsearch -Des.network.host=10.0.0.4
    • Default

        es.default.
    • Environment

        {
      "network" : {
      "host" : "${ES_NET_HOST}"
      }
      }
    • Non-store

      ${prompt.text} or ${prompt.secret}

        node:
      name: ${prompt.text}

      On execution:

        Enter value for [node.name]:

      Elasticsearch will not start if ${prompt.text} or ${prompt.secret} is used in the settings and the process is run as a service or in the background.

Index Settings

  • Index Level (YAML or JSON)

      $ curl -XPUT http://localhost:9200/kimchy/ -d \
    '
    index:
    refresh_interval: 5s
    '
  • Node Level (elasticsearch.yml)

      index :
    refresh_interval: 5s
  • Collapsed

      $ elasticsearch -Des.index.refresh_interval=5s

Logging

  • Log4j (log4j-extras)

  • Format

    • .yml
    • .yaml
    • .json
    • .properties
  • Deprecation

    config/logging.yml

      deprecation: DEBUG, deprecation_log_file

Running as a Service on Linux

Linux

ES_USER

The user to run as, defaults to elasticsearch

ES_GROUP

The group to run as, defaults to elasticsearch

ES_HEAP_SIZE

The heap size to start with

ES_HEAP_NEWSIZE

The size of the new generation heap

ES_DIRECT_SIZE

The maximum size of the direct memory

MAX_OPEN_FILES

Maximum number of open files, defaults to 65535

MAX_LOCKED_MEMORY

Maximum locked memory size. Set to "unlimited" if you use the bootstrap.mlockall option in elasticsearch.yml. You must also set ES_HEAP_SIZE.

MAX_MAP_COUNT

Maximum number of memory map areas a process may have. If you use mmapfs as index store type, make sure this is set to a high value. For more information, check the linux kernel documentation about max_map_count. This is set via sysctl before starting elasticsearch. Defaults to 65535

LOG_DIR

Log directory, defaults to /var/log/elasticsearch

DATA_DIR

Data directory, defaults to /var/lib/elasticsearch

CONF_DIR

Configuration file directory (which needs to include elasticsearch.yml and logging.yml files), defaults to /etc/elasticsearch

ES_JAVA_OPTS

Any additional java options you may want to apply. This may be useful, if you need to set the node.name property, but do not want to change the elasticsearch.yml configuration file, because it is distributed via a provisioning system like puppet or chef. Example: ES_JAVA_OPTS="-Des.node.name=search-01"

RESTART_ON_UPGRADE

Configure restart on package upgrade, defaults to false. This means you will have to restart your elasticsearch instance after installing a package manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continuous shard reallocation resulting in high network traffic and reducing the response times of your cluster.

ES_GC_LOG_FILE

The absolute log file path for creating a garbage collection logfile, which is done by the JVM. Note that this logfile can grow pretty quick and thus is disabled by default.

Debian/Ubuntu

  • runlevels

    update-rc.d

  • init script

    /etc/init.d/elasticsearch

  • configuration

    /etc/default/elasticsearch

  • after install

      dpkg -i
  • start on boot and start up

      sudo update-rc.d elasticsearch defaults 95 10
    sudo /etc/init.d/elasticsearch start

Oracle JDK

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
java -version

RPM based distribution

  • Using chkconfig

    • init script

      /etc/init.d/elasticsearch

    • configuration

      /etc/sysconfig/elasticsearch

    • manual start

        sudo /sbin/chkconfig --add elasticsearch
      sudo service elasticsearch start
  • Using systemd

    • system rpm based

      /etc/default/elasticsearch

    • configuration

      /etc/sysconfig/elasticsearch

    • up

        sudo /bin/systemctl daemon-reload
      sudo /bin/systemctl enable elasticsearch.service
      sudo /bin/systemctl start elasticsearch.service
    • MAX_MAP_COUNT setting

      /etc/sysconfig/elasticsearch (no effect)

      /usr/lib/sysctl.d/elasticsearch.conf

Running as a Service on Windows

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service-win.html

Directory Layout

  • Type(Setting) - Default Location

    Description

  • home(path.home) - N/A

    Home of elasticsearch installation.

  • bin(N/A) - {path.home}/bin

    Binary scripts including elasticsearch to start a node.

  • conf(path.conf) - {path.home}/config

    Configuration files including elasticsearch.yml

  • data(path.data) - {path.home}/data

    The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

  • logs(path.logs) - {path.home}/logs

    Log files location.

  • plugins(path.plugins) - {path.home}/plugins

    Plugin files location. Each plugin will be contained in a subdirectory.

  • repo(path.repo) - Not configured

    Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

  • script(path.script) - {path.conf}/scripts

    Location of script files.

Multiple data paths

---------------------------------
path.data: /mnt/first,/mnt/second
---------------------------------

Or array format

----------------------------------------
path.data: ["/mnt/first", "/mnt/second"]
----------------------------------------

To stripe shards across multiple disks, please use a RAID driver instead.

Default Path

  • deb & rpm

      -----------------------------------------------------------------------------------
    Type | Location Debian/Ubuntu | Location RHEL/CentOS
    -----------------------------------------------------------------------------------
    home | /usr/share/elasticsearch | /usr/share/elasticsearch
    -----------------------------------------------------------------------------------
    bin | /usr/share/elasticsearch/bin | /usr/share/elasticsearch/bin
    -----------------------------------------------------------------------------------
    conf | /etc/elasticsearch | /etc/elasticsearch
    -----------------------------------------------------------------------------------
    conf | /etc/default/elasticsearch | /etc/sysconfig/elasticsearch
    -----------------------------------------------------------------------------------
    data | /var/lib/elasticsearch/data | /var/lib/elasticsearch
    -----------------------------------------------------------------------------------
    logs | /var/log/elasticsearch | /var/log/elasticsearch
    -----------------------------------------------------------------------------------
    plugins | /usr/share/elasticsearch/plugins | /usr/share/elasticsearch/plugins
    -----------------------------------------------------------------------------------
    repo | Not configured | Not configured
    -----------------------------------------------------------------------------------
    script | /etc/elasticsearch/scripts | /etc/elasticsearch/scripts
    -----------------------------------------------------------------------------------
  • zip

      -----------------------------------------------------------------------------------
    Type | Description | Location
    -----------------------------------------------------------------------------------
    home | Home of elasticsearch installation | {extract.path}
    -----------------------------------------------------------------------------------
    bin | scripts to start a node | {extract.path}/bin
    -----------------------------------------------------------------------------------
    conf | files elasticsearch.yml and logging.yml | {extract.path}/config
    -----------------------------------------------------------------------------------
    data | location of files of each index / shard | {extract.path}/data
    -----------------------------------------------------------------------------------
    logs | Log files location | {extract.path}/logs
    -----------------------------------------------------------------------------------
    plugins | Plugin files location. | {extract.path}/plugins
    -----------------------------------------------------------------------------------
    repo | Shared file system repository locations. | Not configured
    -----------------------------------------------------------------------------------
    script | Location of script files. | {extract.path}/config/scripts
    -----------------------------------------------------------------------------------

Repositories

PGP Key

We use the PGP key D88E42B4, Elasticsearch Signing Key, with fingerprint

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

APT

Download and install the Public Signing Key:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Save the repository definition to /etc/apt/sources.list.d/elasticsearch-2.x.list:

echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

Run apt-get update and the repository is ready for use. You can install it with:

sudo apt-get update && sudo apt-get install elasticsearch

Configure Elasticsearch to automatically start during bootup. If your distribution is using SysV init, then you will need to run:

sudo update-rc.d elasticsearch defaults 95 10

Otherwise if your distribution is using systemd:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

YUM

Download and install the public signing key:

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example elasticsearch.repo

[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

And your repository is ready for use. You can install it with:

yum install elasticsearch
  • SysV init

      chkconfig --add elasticsearch
  • Systemd

      sudo /bin/systemctl daemon-reload
    sudo /bin/systemctl enable elasticsearch.service

Reference

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html

ElasticSearch 2 (2) - Setup的更多相关文章

  1. jar hell &amp; elasticsearch ik 版本问题

    想给es 安装一个ik 的插件, 我的es 是 2.4.0, 下载了一个版本是 1.9.5, [2016-10-09 16:56:26,248][INFO ][node ] [node-2] init ...

  2. Elasticsearch Configuration 中文版

    ##################### Elasticsearch Configuration Example ##################### # This file contains ...

  3. centos6.5安装elasticsearch

    java下载地址:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmles下载地址 : ...

  4. 分布式搜索引擎Elasticsearch的简单使用

    官方网址:https://www.elastic.co/products/elasticsearch/ 一.特性 1.支持中文分词 2.支持多种数据源的全文检索引擎 3.分布式 4.基于lucene的 ...

  5. Elasticsearch配置文件说明

    一.Cluster  setting Cluster indices.ttl.interval  允许设置多久过期的文件会被自动删除.默认值是60秒. indices.cache.filter.siz ...

  6. ElasticSearch学习-centos下安装

    1.安装java运行环境(jre) //这里我安装了jdk 其实只需要安装jre就可以了 0)cd /usr;mkdir /usr/java; cd java 1)wget http://downlo ...

  7. elasticsearch单机多实例环境部署

    elasticsearch的功能,主要用在搜索领域,这里,我来研究这个,也是项目需要,为公司开发了一款CMS系统,网站上的搜索栏功能,我打算采用elasticsearch来实现. elasticsea ...

  8. ELK——安装 logstash 2.2.0、elasticsearch 2.2.0 和 Kibana 3.0

    本文内容 Elasticsearch logstash Kibana 参考资料 本文介绍安装 logstash 2.2.0 和 elasticsearch 2.2.0,操作系统环境版本是 CentOS ...

  9. ELK初学搭建(elasticsearch)

    ELK初学搭建(elasticsearch) elasticsearch logstash kibana ELK初学搭建 elasticsearch 1.环境准备 centos6.8_64 mini ...

随机推荐

  1. iOS MRC ARC 内存管理

    转自:http://www.jianshu.com/p/48665652e4e4 1. 什么是内存管理 程序在运行的过程中通常通过以下行为,来增加程序的的内存占用 创建一个OC对象 定义一个变量 调用 ...

  2. [转载]C# 判断字符是否中文还是英文

    private static bool IsHanZi(string ch) { byte[] byte_len = System.Text.Encoding.Default.GetBytes(ch) ...

  3. CentOS镜像163更新源

    首先备份/etc/yum.repos.d/CentOS-Base.repo mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-B ...

  4. FileUtils类介绍

    Java的文件操作太基础,缺乏很多实用工具,比如对目录的操作,支持就非常的差了.如果你经常用Java操作文件或文件夹,你会觉得反复编写这些代码是令人沮丧的问题,而且要大量用到递归. 下面是的一个解决方 ...

  5. Keil C减小代码编译量大小的方法(gai)

    keil-C减小代码编译大小的方法整理 方法一:(通过优化代码减小) 1.1少做乘除运算,使用左/右移位来实现乘除 Eg ,普通:a = 0x80*4: 优化:a = 0x80<<2: 1 ...

  6. git的使用,将本地项目push到github上

    Git教程(推荐): http://www.liaoxuefeng.com/wiki/0013739516305929606dd18361248578c67b8067c8c017b000git是先用a ...

  7. boost.spirit之解析C++头文件

    环境:win7_64旗舰版,VS2008 场景:C++与lua交互是比较繁琐的,当我们编写一个C++类后,如果要给lua使用,就必须写一个lua包装类,将lua与原始C++类关联起来.其实这部分代码编 ...

  8. Hibernate学习(三)一对多数据的保存

    保存习惯代码 @Test public void saveTwoTableTest(){ Transaction transaction = null; try { transaction = ses ...

  9. matlab 加根号

    text(3,0.5,'z=0.2$$\sqrt{c/h_0}$$+0.3','interpreter','latex')

  10. eclipse svn不显示提交人、提交时间的问题

    最近eclipse换到最新的mars版本,装了svn插件subversive后,拉下项目后发现都不显示提交人了,以前都不这样的呀,新版不一样了么? 后来网上搜索总算找到方法解决: Window--&g ...