【elasticsearch】elasticsearch查詢報(bào)錯(cuò)問(wèn)題整理

問(wèn)題一：1. http_code : 4092. 報(bào)錯(cuò)內(nèi)容：Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Result window is too large, from + size must be less than or equal to: [20000] but was [83440000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.at org.elasticsearch.search.DefaultSearchContext.preProcess(DefaultSearchContext.java:203) ~[elasticsearch-5.6.4.jar:5.6.4]at org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:95) ~[elasticsearch-5.6.4.jar:5.6.4]at org.elasticsearch.search.SearchService.createContext(SearchService.java:497) ~[elasticsearch-5.6.4.jar:5.6.4]at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.4.jar:5.6.4]at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.4.jar:5.6.4]at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.4.jar:5.6.4]at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.4.jar:5.6.4]at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178) ~[?:?]at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:107) ~[?:?]at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:92) ~[?:?]at com.floragunn.searchguard.SearchGuardPlugin$5$1.messageReceived(SearchGuardPlugin.java:493) ~[?:?]at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.4.jar:5.6.4]

3. 查詢語(yǔ)句source={"from" : 83438000,"size" : 2000,"query" : {"bool" : {"must" : [{"range" : {"createTime" : {"from" : 1631807999000,"to" : 1631894399000,"include_lower" : true,"include_upper" : false,"boost" : 1.0}}}],"disable_coord" : false,"adjust_pure_negative" : true,"boost" : 1.0}},"_source" : {"includes" : ["companyId","riskLabel"],"excludes" : [ ]}}}]

4. 報(bào)錯(cuò)原因：

Elasticsearch 查詢索引結(jié)果時(shí)，用于分頁(yè)的兩個(gè)屬性 from和size。可以使用from和size 參數(shù)對(duì)結(jié)果進(jìn)行分頁(yè)。該from參數(shù)定義了您要獲取的第一個(gè)結(jié)果的偏移量。該size參數(shù)允許您配置要返回的最大命中數(shù)。

雖然from和size可以設(shè)置為請(qǐng)求參數(shù)，但也可以在搜索正文中設(shè)置。from默認(rèn)為0，size 默認(rèn)為10。

注意，from+size不能超過(guò)index.max_result_window 默認(rèn)為 10,000的索引設(shè)置。目前ES集群返回結(jié)果最大值為20,000

而業(yè)務(wù)方卻要求83438000行。故ES拋出異常。

參考： https://www.elastic.co/guide/en/elasticsearch/reference/5.1/search-request-from-size.html

并且這種分頁(yè)也是淺分頁(yè)，可以理解為簡(jiǎn)單意義上的分頁(yè)。它的原理很簡(jiǎn)單，就是查詢前20條數(shù)據(jù)，然后截?cái)嗲?0條，只返回10-20的數(shù)據(jù)。這樣其實(shí)白白浪費(fèi)了前10條的查詢。

并且越往后的分頁(yè)，執(zhí)行的效率越低。總體上會(huì)隨著from的增加，消耗時(shí)間也會(huì)增加。而且數(shù)據(jù)量越大，效果越明顯！

scroll: 游標(biāo)查詢?cè)试S我們先做查詢初始化，然后再批量地拉取結(jié)果。這有點(diǎn)兒像傳統(tǒng)數(shù)據(jù)庫(kù)中的 cursor 。

游標(biāo)查詢會(huì)取某個(gè)時(shí)間點(diǎn)的快照數(shù)據(jù)。查詢初始化之后索引上的任何變化會(huì)被它忽略。它通過(guò)保存舊的數(shù)據(jù)文件來(lái)實(shí)現(xiàn)這個(gè)特性，結(jié)果就像保留初始化時(shí)的索引視圖一樣。

5. 解決方法：

增大返回結(jié)果。

在不改變業(yè)務(wù)原意的情況下，建議使用scroll API

問(wèn)題二：1. http_code: 5002. 報(bào)錯(cuò)內(nèi)容：Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [order_record_id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

3. 查詢語(yǔ)句：{"from" : 0,"size" : 10,"query" : {"bool" : {"filter" : [{"bool" : {"must" : [{"bool" : {"must" : [{"bool" : {"must" : [{"script" : {"script" : {"source" : "1 == 1","lang" : "painless"},"boost" : 1.0}}],"disable_coord" : false,"adjust_pure_negative" : true,"boost" : 1.0}},{"range" : {"create_time" : {"from" : 1631894400,"to" : null,"include_lower" : true,"include_upper" : true,"boost" : 1.0}}},{"range" : {"create_time" : {"from" : null,"to" : 1631980800,"include_lower" : true,"include_upper" : true,"boost" : 1.0}}},{"range" : {"begin_time" : {"from" : null,"to" : 1632002400,"include_lower" : true,"include_upper" : true,"boost" : 1.0}}},{"bool" : {"should" : [{"match_phrase" : {"mis_user_id" : {"query" : 3230,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 505150,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 505150,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 505153,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 505154,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 505155,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 505156,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 2663,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 3216,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"mis_user_id" : {"query" : 3217,"slop" : 0,"boost" : 1.0}}}],"disable_coord" : false,"adjust_pure_negative" : true,"boost" : 1.0}},{"bool" : {"should" : [{"match_phrase" : {"type" : {"query" : 1,"slop" : 0,"boost" : 1.0}}},{"match_phrase" : {"type" : {"query" : 12,"slop" : 0,"boost" : 1.0}}}],"disable_coord" : false,"adjust_pure_negative" : true,"boost" : 1.0}}],"disable_coord" : false,"adjust_pure_negative" : true,"boost" : 1.0}}],"disable_coord" : false,"adjust_pure_negative" : true,"boost" : 1.0}}],"disable_coord" : false,"adjust_pure_negative" : true,"boost" : 1.0}},"_source" : {"includes" : ["id","begin_time","end_time","mobile","order_record_id","product_apply_order_id","category_id","product_id","content","status","zifang","product_detail_id","user_type","level1_type","level2_type","level3_type","subteams","ext_type","mis_user_id","name","call_system","hotline","record_id","create_time","satisfaction_score","transfer_queue","ext_id","call_start_time","end_reason"],"excludes" : [ ]},"sort" : [{"order_record_id" : {"order" : "desc"}}]}}]

4. 問(wèn)題原因：

order_record_id字段類型為text類型，針對(duì)text類型的FIleData默認(rèn)是禁用的，ElasticSearch無(wú)法對(duì)類型為text的字段進(jìn)行聚合查詢/排序。

5. 解決方案：

避免使用text字段，使用keyword 等別的字段進(jìn)行代替。

對(duì)text字段類型開啟filedata屬性。（不推薦）

問(wèn)題三：http_codeHTTP/1.1 401 Unauthorized

1. 報(bào)錯(cuò)內(nèi)容：[2021-09-18T12:18:06,243][WARN ][o.e.t.LoggingTaskListener] 75886912359 failed with exceptionorg.elasticsearch.ElasticsearchSecurityException: no permissions for [indices:admin/create] and User [name=sentiment_rw, roles=[]]

2. 報(bào)錯(cuò)原因：

sentiment_rw ?用戶沒有寫入索引的權(quán)限功能。

3. 解決方案：

業(yè)務(wù)方寫入索引時(shí)，應(yīng)檢查自己的權(quán)限。

問(wèn)題四：http_code : 4001.問(wèn)題描述：Caused by: org.elasticsearch.index.query.QueryShardException: failed to create queryCaused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: too_many_clauses: maxClauseCount is set to 1024

2. 問(wèn)題查詢：

該查詢特別長(zhǎng)，這里只是截取了一點(diǎn)點(diǎn)。

{"bool" : {"must" : [{"range" : {"createTime" : {"from" : 1631635199000,"to" : 1631721599000,"include_lower" : true,"include_upper" : false,"boost" : 1.0}}}],"must_not" : [{"term" : {"companyId" : {"value" : "5433f1a77a775684071d5034208a9b21","boost" : 1.0}}},....]}

3. 問(wèn)題原因：

查詢sql語(yǔ)句過(guò)長(zhǎng)，超出lucene的最大子句限制（1024）。

4. 解決方法：

規(guī)范查詢，盡量在上線前，提前進(jìn)行測(cè)試。

原文：https://juejin.cn/post/7100389194740023303

ElastiSearch默認(rèn)分詞器
在Elasticsearch中的數(shù)據(jù)可以分為兩類：精確值(exact values)以及全文(full text) 。精確值：例如日期類型date，若date其有兩個(gè)值:2014-09-15與2014，那么這兩個(gè)值不相等。又例如字符串類型foo與Foo不相等。全文：通常是人類語(yǔ)言寫的文本，例如一段tweet信息、email的內(nèi)容等。精確值很容易...

阿里云Elasticsearch的X-Pack:機(jī)器學(xué)習(xí)、安全保障和可視化
阿里云Elasticsearch搭載了V5.5版本的X-Pack，提供了安全、監(jiān)控、告警、報(bào)告、圖分析和機(jī)器學(xué)習(xí)六大擴(kuò)展組件。無(wú)需額外安裝插件，就能與Elastisearch、Kibana無(wú)縫對(duì)接，為用戶提供即開即用的托管服務(wù)。安全組件提供了針對(duì)正常用戶和惡意入侵者的區(qū)分和管理，以及對(duì)Elasticsearch中存儲(chǔ)數(shù)據(jù)的可靠安全保障，確保公司...

es筆記四之中文分詞插件安裝與使用
通過(guò)查看數(shù)據(jù)的“address”字段分詞結(jié)果，我們發(fā)現(xiàn)“一個(gè)蘋果”被錯(cuò)誤地分成了四個(gè)單獨(dú)的字符。接下來(lái)，我們將介紹如何安裝中文分詞插件。在Elasticsearch安裝目錄下，執(zhí)行以下命令進(jìn)行安裝。請(qǐng)注意，這里我們使用的是7.6.0版本的ES，因此需安裝對(duì)應(yīng)的分詞插件。安裝完成后，我們需要重新建立索引，并為“addres...

部署ELK+Elastifolw進(jìn)行網(wǎng)絡(luò)流量分析
Elastiflow是一個(gè)開源的流量分析軟件，基于ELK堆棧，適用于Linux和Windows系統(tǒng)，亦可使用Docker部署。它支持IPFIX(UDP-4739)、Netflow(UDP-2055)和Sflow(UDP-6343)三種流量分析協(xié)議，能夠?qū)W(wǎng)絡(luò)設(shè)備與K8SPod的流量進(jìn)行分析。部署流程如下：1. 安裝依賴，包括JDK環(huán)境。2. 安裝ELK堆棧。3. 修改Elasticsearch配置...

架構(gòu)中的業(yè)務(wù)能力和技術(shù)能力
技術(shù)能力是從我們選擇的技術(shù)、平臺(tái)、基礎(chǔ)結(jié)構(gòu)中體現(xiàn)出來(lái)的。如AWS S3的對(duì)象存儲(chǔ)能力，Lambda的無(wú)服務(wù)器計(jì)算能力，DynamoDB的NoSQL數(shù)據(jù)存儲(chǔ)能力，ElasticSearch的搜索能力，又如ElastiCache的緩存能力和CloudFront的CDN能力。架構(gòu)師所做的架構(gòu)設(shè)計(jì)實(shí)際上就是一個(gè)函數(shù)f(x)，x是業(yè)務(wù)能力，所得f(x)是從業(yè)務(wù)能力...

店匠助力智能硬件企業(yè)出海——搭載Amazon Web Services,為酷客智能...
店匠SHOPLAZZA所使用的AWS云服務(wù)包括Amazon Elastic Kubernetes Service(Amazon EKS)、AWS Identity and Access Management (AWS IAM)、AWS WAF、AWS Shield、Amazon Route 53、Amazon ElastiCache、Amazon Elasticsearch Service (Amazon ES)、AWS Global Accelerator(AWS GA)、Amazon Simple Storage Service (...

www.tjgcgs88.cn-狠狠久久亚洲欧美专区不卡,久久精品国产99久久无毒不卡,噼里啪啦国语版在线观看,zσzσzσ女人极品另类

【elasticsearch】elasticsearch查詢報(bào)錯(cuò)問(wèn)題整理

相關(guān)評(píng)說(shuō)：