Elasticsearch 7 中文文档功能介绍

Author

Reads 1K

Men typing in the Google search engine from realme 6 pro. "Google" is the number one search web.
Credit: pexels.com, Men typing in the Google search engine from realme 6 pro. "Google" is the number one search web.

Elasticsearch 7 中文文档支持多种语言的文档功能,包括中文文档。

中文文档功能使用户能够以中文阅读和使用 Elasticsearch 7 的文档。

Elasticsearch 7 中文文档功能包括支持中文的索引、映射和分析等功能。

这使得用户能够以中文索引和分析数据。

中文文档功能的优势在于提高了用户的阅读和使用体验。

基本概念

type 是 index 中的一个逻辑分类,每个 index 可以定义一个或多个 type。在 Elasticsearch 6.x 版本之前,每个索引中可以定义一个或多个 type,而在 6.X 版本之后,一个 index 中只能定义一个 type。

一个 Index 是一个包含多个 document 的容器,每个 document 都必须属于某一个具体的 type,或者说每个 document 都有一个 _type 属性。Index 的结构包括 mappings、type、settings、number_of_shards 和 number_of_replicas 等重要结构名词。

Additional reading: Elasticsearch X Pack

基本概念

In Elasticsearch, a Shard is also known as a primary shard, used to store index data.

The concept of Shard is important for understanding how Elasticsearch stores and retrieves data.

A Shard can be thought of as a container for storing documents with similar structures.

In Elasticsearch, a Shard is a critical component for storing and retrieving data efficiently.

Here's a breakdown of some key terms related to Shards:

Each Shard is responsible for storing a portion of the data in an index.

功能强大,操作简单。

Elasticsearch provides millisecond-level delay search, backed by Lucene technology, so you can get instant answers regardless of how fast your data moves.

With Elasticsearch, you can easily expand your search capabilities. The platform can automatically scale, rebalance, and replicate data, handling petabyte-level data volumes without downtime or issues.

Elasticsearch is a single platform that can handle all your data needs, from search to observability to security. You can build it once and reuse it anywhere.

Here are some key benefits of Elasticsearch's advanced search and relevance capabilities:

  • Full-text search, vector search, and hybrid search can be used to find value in your data.
  • Elasticsearch supports fuzzy queries, semantic search, and exact matches for unparalleled accuracy.

Elasticsearch offers high-performance, cost-effective data storage that's searchable. You can store data on local disks, object storage, or a combination of both, and still get the speed and cost benefits you need.

Elasticsearch provides built-in connectors, generic patterns, and open standards to connect and integrate with your data. You can use APIs, clients, and pipelines to quickly ingest data from anywhere, and Elasticsearch will seamlessly connect, transfer, analyze, and search all types of data.

On a similar theme: Elasticsearch Storage

索引和分片

Credit: youtube.com, 10 Elasticsearch集群之分片和副本

索引可以通过索引名称来执行索引、搜索、更新和删除等操作。每个索引都有一个唯一的名称。Elasticsearch会索引所有字段,经过处理后写入一个反向索引(Inverted Index)。查找数据的时候,直接查找该索引。

一个索引可以分成多个分片,每个分片都是一个Lucene实例,可以托管在集群的任一节点上。分片可以实现数据的分散式执行,提高性能和流量。一个索引可以有多个分片,每个分片都有一个唯一的名称。分片可以根据需要进行扩展和缩减。

Shard

Shard is a crucial concept in Elasticsearch that allows you to split a large index into smaller pieces, called shards, to distribute data across multiple nodes. This approach is essential for handling massive amounts of data.

Each shard is an independent Lucene instance, responsible for indexing and searching a portion of the data. This means that shards can be hosted on any node in the cluster, allowing for flexible distribution of data. A single Lucene instance can store up to 2147483519 documents, or about 2 billion documents.

Shards can be used to improve performance and scalability by allowing multiple nodes to work together to handle searches and analytics. This approach is particularly useful for large datasets that don't fit on a single node.

Here are some key benefits of using shards:

  • Improved performance: By distributing data across multiple nodes, shards can handle large amounts of data and improve search performance.
  • Scalability: Shards allow you to add or remove nodes as needed, making it easy to scale your Elasticsearch cluster.
  • Flexibility: Shards can be hosted on any node in the cluster, giving you flexibility in how you distribute data.

It's worth noting that shards can be created dynamically, allowing you to adjust the number of shards based on your data's changing needs. However, once a shard is created, it cannot be modified.

For more insights, see: Elasticsearch Shards

中文分词设置

Credit: youtube.com, 数据分片中的索引怎么处理?DDIA分布式系统必读教程精读系列

中文分词设置的选项通常包括自动分词和手动分词。自动分词会根据系统的算法来分词,而手动分词则需要人工干预。

手动分词通常用于特殊的场景,如处理含有多义词的文本。

自动分词通常会根据词频和词关联度来进行分词。

分词设置的优化通常需要根据具体的数据和应用场景来进行调整。

Index

An index in Elasticsearch is a collection of documents with similar structures. It's essentially the same as a collection in Solr. For example, you might have a product classification index and an order index. Each index needs to have a unique name, which must be in lowercase.

You can have multiple indices in a cluster, as long as their names are different. The name of the index is used to execute operations like indexing, searching, updating, and deleting.

A single index can store a large number of documents, but it's limited by the maximum value of an integer, which is 2147483519. This is because each index is stored as a Lucene instance, and Lucene has its own limitations.

To monitor the size of each index, you can use the `_cat/shards` API. This API allows you to see the size of each shard, which is a smaller unit of storage within an index.

Curious to learn more? Check out: Elasticsearch _template

Credit: youtube.com, Python 程式語言 設計課程 : 基礎篇 指令介紹及使用 002 - 005:Indexing 索引 Slicing 分片

Here's a breakdown of the roles of indices in Elasticsearch:

Indices are the top-level unit of data management in Elasticsearch. They are equivalent to a single database. Each index has a unique name, which must be in lowercase.

中文文档

在 Elasticsearch 7 中,中文文档以简洁明了的方式解释了 Elasticsearch 的核心概念。

Elasticsearch 7 中的中文文档提供了一个易于理解的参考指南。

Document 2.3

A document in Elasticsearch is the smallest unit of data, and it's represented in JSON format. It's like a single piece of information that can be a customer data, product category data, or order data.

The structure of a document is flexible, but it's recommended to keep the same structure for all documents in the same index to improve search efficiency.

Here's a breakdown of the important fields in a document:

In Elasticsearch, documents can have multiple fields, and each field is a data string.

4.4 中文分词器

在中文文档中,分词器的准确率至关重要,直接影响后续的文本分析和处理结果。

中文分词器通常使用统计模型和机器学习算法来实现分词。

例如, IKAnalyzer 是一个流行的中文分词器,使用统计模型和词典来进行分词。

IKAnalyzer 的准确率可以达到 95% 以上,适用于大多数中文文档处理任务。

4.5 中文分词器

通过使用中文分词器,我们可以有效地处理中文文本,进行信息抽取、文本分析等操作。例如,使用中文分词器,我们可以将一段中文文本分成单独的词语,如“中文分词器”可以将“中文文档”的第一段文本分成“中文”和“文档”这两个词语。

中文分词器的准确度直接影响到中文文本的处理结果。准确的分词器可以帮助我们正确识别中文文本中的词语和短语,从而提高信息抽取和文本分析的效率。

查询和分析

Elasticsearch 7 中文文档中提到,Elasticsearch提供了多种查询类型,包括Simple Query、Filtered Query和Native Query等。

Elasticsearch 7 中文文档中提到,Elasticsearch的分析功能可以帮助我们对数据进行统计分析,例如计算总和、平均值和标准差等。

Term Query

Credit: youtube.com, 117、全文检索 ElasticSearch 进阶 term查询

Term Query is the most basic unit of expression in search queries, and it's used extensively in Elasticsearch.

Term queries are of various types, including Term Query and Range Query.

In Elasticsearch, term queries don't perform any text analysis or tokenization, treating the input as a whole entity.

Term queries search for exact terms in the inverted index and score them.

Using Constant Score can convert a term query into a filter, avoiding scoring and leveraging caching to boost query efficiency.

On a similar theme: Elasticsearch Term Query

Match Phrase

In the world of querying and analysis, having the right match phrase can make all the difference in getting accurate results.

A match phrase is a crucial component in querying and analysis, allowing you to search for specific words or phrases within a dataset.

Using a match phrase, you can search for exact phrases like "coffee shop" or "customer service", which can help you identify relevant data points.

This is especially useful when searching for phrases that have multiple words, as it ensures that the search results include the exact phrase you're looking for.

For example, searching for "coffee shop" will return results that include both words, whereas searching for "coffee" or "shop" separately may return irrelevant results.

By using match phrases, you can refine your search queries and get more accurate results, saving you time and effort in the long run.

Expand your knowledge: Elasticsearch Match

组合查询

Credit: youtube.com, 026 Elasticsearch 7.8.0 入门 JavaAPI 文档 高级查询 组合查询 & 范围查询

通过使用组合查询,我们可以创建更复杂的查询条件,比如查找年龄大于18且收入超过5000的用户。我们可以使用AND关键字来连接多个条件。

组合查询还可以使用括号来指定条件的顺序,比如查找年龄大于18且收入超过5000的用户,或年龄大于25且收入超过10000的用户。

聚合查询

聚合查询可以帮助我们快速找到数据之间的模式和趋势。

数据源可以是数据库、文档或其他类型的数据存储。

例如,我们可以使用聚合查询来找出销售数据中最常见的产品类别。

聚合查询还可以帮助我们分析数据的分布情况,例如平均值、最大值和最小值。

通过使用聚合函数如SUM、AVG和MAX,我们可以快速计算出数据的总和、平均值和最大值。

例如,SUM函数可以帮助我们计算出所有订单总额。

AVG函数可以帮助我们计算出所有订单的平均价格。

MAX函数可以帮助我们找出最贵的订单。

自动补全

在一些数据库系统中,自动补全可以通过预先建立关键词索引来实现。这使得系统可以快速找到匹配的关键词。

自动补全可以大大提高用户的工作效率,特别是在输入大量数据时。

Near Real-time Search (NRT)

Near Real-time Search (NRT) is a crucial feature in Elasticsearch that enables fast and efficient searching. It's designed to overcome the performance bottleneck caused by traditional disk-based storage.

In traditional systems, writing data to disk can take a long time, often exceeding 1 minute, making it difficult to retrieve data in a timely manner. This is because the system has to wait for the data to be written to disk before it can be searched.

The process of writing data to disk involves several steps, including writing data to a buffer, committing the data to a new segment, writing the segment to the operating system's cache, and finally writing the segment to disk. This process is slow because it requires waiting for the data to be written to disk.

Elasticsearch optimizes this process by prioritizing the writing of data to the operating system's cache, allowing for direct searching of the data without waiting for the commit process to complete.

Here's an interesting read: Elasticsearch Spring Data

Credit: youtube.com, SEARCH near real time search

The refresh process in Elasticsearch is responsible for writing data to new segments and making them available for searching. By default, Elasticsearch refreshes every 1 second, enabling near real-time search capabilities.

Here's a step-by-step comparison of the traditional and optimized processes:

  1. Traditional: Write data to buffer
  2. Traditional: Commit data to new segment
  3. Traditional: Write segment to operating system's cache
  4. Traditional: Write segment to disk and clear buffer
  5. Traditional: Search segment
  1. Optimized (Elasticsearch): Write data to buffer
  2. Optimized (Elasticsearch): Write buffer data to new segment
  3. Optimized (Elasticsearch): Write segment to operating system's cache
  4. Optimized (Elasticsearch): Search segment without waiting for commit

数据类型和操作

每个索引中,可以有一个或多个类型,每个类型下面的文档都有相同的字段。例如,日常商品、电器商品和生鲜商品都有不同的类型。

在实际操作中,我们可以使用这些类型来组织和管理我们的数据,使其更容易查询和分析。

Term and Terms

A term is the smallest unit of expression in Elasticsearch, and it's used in most searches. It's a fundamental concept that you need to understand to work with Elasticsearch effectively.

Term queries are a type of query in Elasticsearch that searches for exact terms in a field. There are different types of term queries, including Term Query and Range Query. Term queries are different from other queries in that they don't perform any text analysis or tokenization on the input content. Instead, they search for the exact term in the inverted index.

In Elasticsearch, term queries can be converted to filters using the Constant Score query, which can improve the query's efficiency by avoiding scoring and utilizing caching.

Here's a summary of the types of term queries:

Data Types

Credit: youtube.com, 020--python教程(数据类型的区别,differences in data types)

Data types in Elasticsearch are a way to group documents together based on their structure and content. This is done to facilitate filtering and querying of documents within an index.

In Elasticsearch, a type is a virtual logical grouping of documents. It's used to filter documents based on their structure and content. Different types should have similar structures, so you can't have a type where the id field is a string in one type and a number in another.

You can list the types in an index using a specific command. As of Elasticsearch 6.x, each index can only contain one type, and in version 7, types will be removed altogether.

A type is essentially a logical classification of data within an index. In Elasticsearch 6.x and earlier, each index could have multiple types, but in 6.x and later, each index can only have one type. A type is typically defined as a group of documents with a common set of fields.

Check this out: Elasticsearch Index Api

Credit: youtube.com, 019--python教程(数据类型,data types)

Here's a summary of the differences between types in Elasticsearch:

For example, in a blog system, you could define a type for user data, a type for blog posts, and a type for comments. Each document would then belong to one of these types, and would have a corresponding _type attribute.

Index Operations

An index in a data system is a collection of documents with similar structures, equivalent to a collection in Solr. It can be named after the type of data it contains, such as a product classification index or an order index.

Each index must have a unique name, which should be in lowercase. This name is used to perform operations like executing, searching, updating, and deleting the index.

A cluster can have multiple indices as long as their names are different.

Elastic indexes all fields and writes them to a reverse index, making it easier to find data. This is the top-level unit of data management in Elastic, equivalent to a single database.

Every index must have a unique name, which should be in lowercase.

Explore further: Elastic Search Cluster

Men typing in the Google search engine from realme 6 pro. "Google" is the number one search web.
Credit: pexels.com, Men typing in the Google search engine from realme 6 pro. "Google" is the number one search web.

You can create a new index by sending a PUT request to the Elastic server. For example, creating a weather index would look like this. The server will respond with a JSON object indicating whether the operation was successful.

To delete an index, you can send a DELETE request to the server.

On a similar theme: Elasticsearch Delete Document

安装和配置

To install Elasticsearch, you'll need a Java 8 environment. Make sure your machine has Java installed and the environment variable JAVA_HOME is set correctly.

Installing Java is the first step, and then you can follow the official documentation to install Elasticsearch. Downloading the compressed package is a straightforward way to get started.

If you encounter an error saying "max virtual memory areas vm.maxmapcount [65530] is too low", you can run a specific command to resolve the issue. If everything goes smoothly, Elasticsearch will run on the default port 9200.

On a similar theme: Install Elasticsearch Windows

一、安装

To install Elastic, you'll need Java 8 installed on your machine. Make sure the environment variable JAVA_HOME is set correctly.

Credit: youtube.com, 【新手必看】PVE安装和优化 | 新手保姆教程 | All in One 基础篇

You can install Java by following the instructions in the article, and then use the official Elastic documentation to install Elastic. Downloading the compressed package is a straightforward option.

Once you've downloaded and extracted the package, navigate to the directory and run the command to start Elastic.

If you encounter an error saying "max virtual memory areas vm.maxmapcount [65530] is too low", run the command to increase the limit.

If everything goes smoothly, Elastic will start running on the default 9200 port. Open a new command line window and request the 9200 port to get information about the current node, cluster, and version.

使用当前使用的语言编写代码

When you're working with Elasticsearch, you have the freedom to write your code in the language you're already using. This means you can use languages like Java, Python, or Go for your client-side code, and still access the raw API when needed.

Your code is in your hands, and the client is designed to be quick, simple, and flexible. This flexibility is meant to support your projects on Elasticsearch.

The client is built to be easy to use, making it perfect for a wide range of projects.

搜索和存储

Credit: youtube.com, 053 Elasticsearch 7.8.0 框架集成 SpringData 集成测试 文档搜索

Elasticsearch 的数据搜索方式会影响效能。例如,如果要查找关键字“登入”,则会搜寻整个表格,导致数据量越大,效能越慢。

Elasticsearch 的搜索方式可以分为两种:全文检索和倒排索引。

其他功能

Elasticsearch 7 支持多种索引类型,包括整数、浮点数、日期和布尔值等。

Elasticsearch 7 提供了一个叫做 watcher 的功能,可以让你在某些条件发生时触发动作。

Watcher 可以监控 Elasticsearch 的各种指标,如 CPU 使用率、内存占用率等。

Elasticsearch 7 还支持使用脚本来处理数据,比如说你可以使用脚本来计算某些字段的值。

这些脚本可以使用各种语言编写,如 Groovy、Painless 和 Python 等。

You might like: Elasticsearch Watcher

Wm Kling

Lead Writer

Wm Kling is a seasoned writer with a passion for technology and innovation. With a strong background in software development, Wm brings a unique perspective to his writing, making complex topics accessible to a wide range of readers. Wm's expertise spans the realm of Visual Studio web development, where he has written in-depth articles and guides to help developers navigate the latest tools and technologies.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.