海口市建设局网站海南网站建设哪里有

海口市建设局网站,海南网站建设哪里有,成都网站建设类岗位,网站建设销售信Nginx access 日志通过 Filebeat 8.15.5 写入 Elasticsearch 8 实战指南一、环境说明二、部署步骤与深度解析步骤1#xff1a;创建Filebeat工作目录步骤 2#xff1a;下载并解压 Filebeat 安装包​步骤 3#xff1a;filebeat.yml 核心配置文件​输入配置 (filebeat.inputs)模…Nginx access 日志通过 Filebeat 8.15.5 写入 Elasticsearch 8 实战指南一、环境说明二、部署步骤与深度解析步骤1创建Filebeat工作目录步骤 2下载并解压 Filebeat 安装包​步骤 3filebeat.yml 核心配置文件​输入配置 (filebeat.inputs)模板与兼容性设置 (setup.template setup.ilm)输出配置 (output.elasticsearch)日志配置 (logging)步骤 4校验配置文件语法与 ES 连接​步骤 5迁移 Filebeat 到系统标准目录​步骤 6创建 Systemd 服务步骤 7启动服务并设置开机自启​三、常见问题与解决方案​配置校验报错 type mismatch accessing filebeat.inputs.0.file_identity​ES 写入失败 no matching index template found​首次启动 CPU 飙升​四、总结​在日志分析场景中Nginx访问日志access.log包含了丰富的用户行为、请求性能等关键数据将其高效采集并写入ElasticsearchES是实现日志检索、分析与可视化的基础。本文基于Filebeat 8.15.5和ES 8环境结合实际部署经验详细拆解从环境准备到服务上线的完整流程同时解答部署过程中常见的坑点与解决方案。一、环境说明操作系统Rocky Linux兼容CentOS、RHEL等主流Linux发行版核心组件版本Filebeat 8.15.5、Elasticsearch 8.x日志源Nginx/OpenResty访问日志双路径/mnt/data/openresty2/nginx/logs/access.log和/mnt/data/openresty/nginx/logs/access.logES配置地址http://1.2.3.4:5678索引格式nginx_%{yyyyMMdd}预处理管道pipeline-nginx-plaintext二、部署步骤与深度解析步骤1创建Filebeat工作目录cd/mnt/datamkdirfilebeat-escdfilebeat-es/解析​工作目录选择/mnt/data通常/mnt目录用于挂载数据盘存储空间充足适合存放 Filebeat 安装包、配置文件及临时数据避免因系统盘空间不足导致服务异常后续的filebeat日志也在这个地方存储以便于排查错误。​独立目录 filebeat-es便于统一管理 Filebeat 相关文件后续迁移、升级时更高效也能与其他应用目录区分提升系统整洁度。​步骤 2下载并解压 Filebeat 安装包​# 下载filbeat 到/mnt/data/filebeat-es# 下载地址https://www.elastic.co/downloads/past-releases/filebeat-8-15-5tar-zxvf filebeat-8.15.5-linux-x86_64.tar.gz -C ./​mvfilebeat-8.15.5-linux-x86_64 filebeat-es​cdfilebeat-es/​​解析​解压参数说明tar -zxvf中z 表示解压 gz 压缩包x 表示提取文件v 显示解压过程f 指定目标文件-C ./ 表示将文件解压到当前目录避免文件混乱。​重命名目录将默认的filebeat-8.15.5-linux-x86_64改为filebeat-es名称更简洁且明确用途用于对接 ES后续操作更易识别。​步骤 3filebeat.yml 核心配置文件​vifilebeat.yml​###################### Filebeat Configuration Example ########################## This file is an example configuration file highlighting only the most common# options. The filebeat.reference.yml file from the same directory contains all the# supported options with more comments. You can use it as a reference.## You can find the full configuration reference here:# https://www.elastic.co/guide/en/beats/filebeat/index.html# For more available modules and options, please see the filebeat.reference.yml sample# configuration file.# Filebeat inputs filebeat.inputs:# Each - is an input. Most options can be set at the input level, so# you can use different inputs for various configurations.# Below are the input specific configurations.# 这里定义类型是log- type: log# Change to true to enable this input configuration.enabled:true# Paths that should be crawled and fetched. Glob based paths.# 这个地方是nginx日志的实际路径可以根据实际进行修改paths: - /mnt/data/openresty/nginx/logs/access.log#- c:\programdata\elasticsearch\logs\*# 这个地方是从末尾开始读取写入ES通常nginx的日志文件很大 耗费CPU资源历史的log不再处理tail_files:trueoffset: initial: end# Exclude lines. A list of regular expressions to match. It drops the lines that are# matching any regular expression from the list.# exclude_lines: [^127.0.0.1]# Include lines. A list of regular expressions to match. It exports the lines that are# matching any regular expression from the list.#include_lines: [^ERR, ^WARN]# Exclude files. A list of regular expressions to match. Filebeat drops the files that# are matching any regular expression from the list. By default, no files are dropped.#exclude_files: [.gz$]# Optional additional fields. These fields can be freely picked# to add additional information to the crawled log files for filtering# 这个地方是给采集日志添加logtype列通过es查询的时候可以辨别 类似自定义标签可以修改# 比如某个主机上安装了两个用途不同的openresty实例或者多个实例组成的负载用于辨别。fields: logtype:mynginx01_openresty# level: debug# review: 1fields_under_root:truejson.enabled:truejson.keys_under_root:true### Multiline options# Multiline can be used for log messages spanning multiple lines. This is common# for Java Stack Traces or C-Line Continuation# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [# multiline.pattern: ^\[# multiline.pattern: ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\s-# Defines if the pattern set under pattern should be negated or not. Default is false.# multiline.negate: true# Match can be set to after or before. It is used to define if lines should be append to a pattern# that was (not) matched before or after or as long as a pattern is not matched based on negate.# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash# multiline.match: after#------------------------------------------------------------------------------------------- type: log# Change to true to enable this input configuration.enabled:true# Paths that should be crawled and fetched. Glob based paths.# 这个地方是我第二个nginx日志的实际路径可以根据实际进行修改或者删除paths: - /mnt/data/openresty2/nginx/logs/access.log#- c:\programdata\elasticsearch\logs\*# 参考第一段tail_files:trueoffset: initial: end# Exclude lines. A list of regular expressions to match. It drops the lines that are# matching any regular expression from the list.# exclude_lines: [^127.0.0.1]# Include lines. A list of regular expressions to match. It exports the lines that are# matching any regular expression from the list.#include_lines: [^ERR, ^WARN]# Exclude files. A list of regular expressions to match. Filebeat drops the files that# are matching any regular expression from the list. By default, no files are dropped.#exclude_files: [.gz$]# Optional additional fields. These fields can be freely picked# to add additional information to the crawled log files for filteringfields: logtype:mynginx01_openresty2# level: debug# review: 1fields_under_root:truejson.enabled:truejson.keys_under_root:true### Multiline options# Multiline can be used for log messages spanning multiple lines. This is common# for Java Stack Traces or C-Line Continuation# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [# multiline.pattern: ^\[# multiline.pattern: ^[0-9]{4}/[0-9]{2}/[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\s# Defines if the pattern set under pattern should be negated or not. Default is false.# multiline.negate: true# Match can be set to after or before. It is used to define if lines should be append to a pattern# that was (not) matched before or after or as long as a pattern is not matched based on negate.# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash# multiline.match: after# Filebeat modules filebeat.shutdown_timeout: 5s filebeat.config.modules:# Glob pattern for configuration loadingpath:${path.config}/modules.d/*.yml# Set to true to enable config reloadingreload.enabled:false# Period on which files under path should be checked for changes#reload.period: 10s# Elasticsearch template setting setup.ilm.enabled:falsesetup.data_streams.enabled:falsesetup.template.name:nginx_templatesetup.template.pattern:nginx*setup.template.priority:200setup.template.settings: index.number_of_shards:5#index.codec: best_compression#_source.enabled: false# General # The name of the shipper that publishes the network data. It can be used to group# all the transactions sent by a single shipper in the web interface.#name:# The tags of the shipper are included in their own field with each# transaction published.#tags: [service-X, web-tier]# Optional fields that you can specify to add additional information to the# output.#fields:# env: staging# Dashboards # These settings control loading the sample dashboards to the Kibana index. Loading# the dashboards is disabled by default and can be enabled either by setting the# options here or by using the setup command.#setup.dashboards.enabled: false# The URL from where to download the dashboards archive. By default this URL# has a value which is computed based on the Beat name and version. For released# versions, this URL points to the dashboard archive on the artifacts.elastic.co# website.#setup.dashboards.url:# Kibana # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.# This requires a Kibana endpoint configuration.setup.kibana:# Kibana Host# Scheme and port can be left out and will be set to the default (http and 5601)# In case you specify and additional path, the scheme is required: http://localhost:5601/path# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601#host: localhost:5601# Kibana Space ID# ID of the Kibana Space into which the dashboards should be loaded. By default,# the Default Space will be used.#space.id:# Elastic Cloud # These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).# The cloud.id setting overwrites the output.elasticsearch.hosts and# setup.kibana.host options.# You can find the cloud.id in the Elastic Cloud web UI.#cloud.id:# The cloud.auth setting overwrites the output.elasticsearch.username and# output.elasticsearch.password settings. The format is user:pass.#cloud.auth:# Outputs # Configure what output to use when sending the data collected by the beat.#-------------------------- Elasticsearch output ------------------------------output.elasticsearch:# Array of hosts to connect to.hosts:[1.2.3.4:5678]data_stream:falsepipeline:pipeline-nginx-plaintextindex:nginx_%{yyyyMMdd}preset: balanced# Optional protocol and basic auth credentials.#protocol: https#username: elastic#password: changeme#----------------------------- Logstash output --------------------------------#output.logstash:# The Logstash hosts#hosts: [localhost:5044]# Optional SSL. By default is off.# List of root certificates for HTTPS server verifications#ssl.certificate_authorities: [/etc/pki/root/ca.pem]# Certificate for SSL client authentication#ssl.certificate: /etc/pki/client/cert.pem# Client Certificate Key#ssl.key: /etc/pki/client/cert.key# Processors # Logging logging.level: info logging.to_files:truelogging.files: path: /mnt/data/filebeat-es name: filebeat.log keepfiles:3permissions: 0644# Sets log level. The default log level is info.# Available log levels are: error, warning, info, debug#logging.level: debug# At debug level, you can selectively enable logging only for some components.# To enable all selectors use [*]. Examples of other selectors are beat,# publish, service.#logging.selectors: [*]# Xpack Monitoring # filebeat can export internal metrics to a central Elasticsearch monitoring# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The# reporting is disabled by default.# Set to true to enable the monitoring reporter.#xpack.monitoring.enabled: false# Uncomment to send the metrics to Elasticsearch. Most settings from the# Elasticsearch output are accepted here as well. Any setting that is not set is# automatically inherited from the Elasticsearch output configuration, so if you# have the Elasticsearch output configured, you can simply uncomment the# following line.#xpack.monitoring.elasticsearch:# Migration # This allows to enable 6.7 migration aliases#migration.6_to_7.enabled: true​以下是逐段详细解释输入配置 (filebeat.inputs)这部分定义了数据源。你配置了两个独立的输入块inputs分别对应两个不同的 Nginx 实例openresty 和 openresty2。​-type:logenabled:truepaths:-/mnt/data/openresty/nginx/logs/access.log# --------------------------------------------------------# 性能优化核心配置解决 CPU 飙升问题# --------------------------------------------------------tail_files:true# 启动时忽略文件中的旧数据只看新产生的数据offset:initial:end# 配合 tail_files明确指定偏移量从文件末尾开始# --------------------------------------------------------# ️ 标签与分类区分不同业务来源# --------------------------------------------------------fields:logtype:mynginx01_openresty# 给该日志打上特定标签用于后续生成索引名fields_under_root:true# 将 logtype 字段放在根节点而不是 fields.logtype# --------------------------------------------------------# JSON 解析在 Agent 端直接解析# --------------------------------------------------------json.enabled:true# 开启 JSON 解析json.keys_under_root:true# 解析后的字段如 status, uri直接放在根节点双输入设计监控了两个路径。通过 fields.logtype 字段进行区分。tail_files offset这是你解决 “大文件导致 CPU 高” 的关键设置。它告诉 Filebeat“不要管以前那数据只盯着文件尾部有新行才发”。json.keys_under_root: true这非常重要。它意味着 Filebeat 发给 ES 的数据已经是结构化的例如 {“status”: 200, “clientip”: “…”}而不是包含一整行 JSON 字符串的 message 字段。模板与兼容性设置 (setup.template setup.ilm)这部分是为了让 Filebeat 兼容 Elasticsearch 8.x并允许你使用自定义的索引名称。# Elasticsearch template setting setup.ilm.enabled:false# 禁用索引生命周期管理 (ILM)setup.data_streams.enabled:false# 禁用数据流 (Data Streams)# 定义模板setup.template.name:nginx_template# 模板名称setup.template.pattern:nginx*# 模板匹配的索引模式setup.template.priority:200# 优先级 200 (默认是 150)确保你的模板覆盖系统默认模板setup.template.settings:index.number_of_shards:5# 强制设置每个索引有 5 个主分片什么要禁用 ILM 和 Data Streams在 ES 8 中默认强制使用 Data Streams索引名类似 .ds-logs-nginx-…。为了使用你自定义的索引名 nginx_%{yyyyMMdd}必须显式禁用这两个功能。优先级 200非常关键。这防止了 Filebeat 自身的默认模板覆盖你的设置确保分片数设置为 5 生效。输出配置 (output.elasticsearch)定义数据发送到哪里以及如何命名。output.elasticsearch:hosts:[1.2.3.4:5678]data_stream:false# 再次确认禁用数据流输出# ⚠️ 关键点Ingest Pipelinepipeline:pipeline-nginx-plaintext# 索引命名规则index:nginx_%{yyyyMMdd}# 按天生成索引例如 nginx_20251217preset:balanced# 性能预设平衡吞吐量和资源消耗index这里你设置了nginx_%{yyyyMMdd}。注意你之前的配置使用了%{[logtype]}变量如 nginx_public_…但在这个配置中去掉了 logtype。这意味着两个 Nginx 实例的日志会写入同一个索引 nginx_20251217 中。如果这是有意为之的为了减少分片数量那是没问题的如果想分开需要改回nginx_%{[logtype]}_%{yyyyMMdd}。pipeline: 这里指定了pipeline-nginx-plaintext。潜在冲突提醒你的 Input 已经开启了 JSON 解析但 Pipeline 名字叫 “plaintext”纯文本。通常 plaintext pipeline 用 Grok 解析原始文本。如果 Filebeat 发送的是已经解析好的 JSON 字段而 Pipeline 还在尝试解析 message 文本可能会报错或失效。日志配置 (logging)定义 Filebeat 自身的运行日志用于排查 Filebeat 报错。logging.level:infologging.to_files:truelogging.files:path:/mnt/data/filebeat-es#日志存放路径name:filebeat.logkeepfiles:3#只保留最近3个文件防止占满磁盘permissions:0644这份配置非常成熟适合生产环境。它解决了大文件读取、多实例区分和 ES 8 兼容性问题。步骤 4校验配置文件语法与 ES 连接​./filebeattestconfig -c filebeat.yml ./filebeattestoutput -c filebeat.yml​解析​语法校验 test config检查 filebeat.yml 是否存在缩进错误、字段拼写错误等语法问题。输出 Config OK 表示语法无错若报错需根据提示修正如 type mismatch 通常是字段格式错误。​输出连接测试 test output验证 Filebeat 能否正常连接 ES。成功会显示 ES 版本、连接状态等信息若报错 connection refused需检查 ES 地址 / 端口是否正确、防火墙是否放行 12200 端口。​​输出说明 ✅ 连接正常输出类似以下内容显示 ES 版本、连接成功 elasticsearch: http://1.2.3.4:5678... parse url... OK connection... parse host... OK dns lookup... OK addresses:1.2.3.4 dial up... OK TLS... WARN secure connection disabled talk to server... OK version:8.15.5步骤 5迁移 Filebeat 到系统标准目录​cd..mvfilebeat-es /usr/local/解析​迁移到 /usr/local/Linux 系统中/usr/local 是默认的第三方应用安装目录符合系统目录规范便于后续管理如统一升级、权限控制。​注意迁移后需确保目录权限正确后续启动服务时避免权限不足问题。​步骤 6创建 Systemd 服务vi/usr/lib/systemd/system/filebeat-es.service服务文件内容​[Unit]DescriptionFilebeat sends log data to ElasticsearchDocumentationhttps://www.elastic.co/docs/beats/filebeatWantsnetwork-online.targetAfternetwork-online.target[Service]TypesimpleUserrootGrouprootExecStart/usr/local/filebeat-es/filebeat -c /usr/local/filebeat-es/filebeat.ymlExecReload/bin/kill -HUP$MAINPIDRestartalwaysRestartSec5LimitNOFILE65535[Install]WantedBymulti-user.target​解析​Systemd 服务优势相比传统的 init.d 脚本Systemd 支持开机自启、状态管理、日志聚合等功能更适合生产环境的服务管理。​关键配置说明Afternetwork-online.target 确保服务在网络完全就绪后启动避免因 ES 未访问导致启动失败Restartalways 提高服务可用性异常退出后自动恢复LimitNOFILE65535 解决 Filebeat 处理大量日志时的文件句柄限制问题。​步骤 7启动服务并设置开机自启​systemctl daemon-reload systemctl start filebeat-es systemctl status filebeat-es systemctlenablefilebeat-es​解析​daemon-reload重新加载 Systemd 配置让新创建的 filebeat-es.service 生效修改服务文件后必须执行此命令。​start 与 status启动服务后通过 status 查看运行状态。若显示 active (running) 表示启动成功若失败可通过 journalctl -u filebeat-es -f 查看详细错误日志。​enable设置开机自启避免服务器重启后需手动启动服务确保日志采集不中断。​三、常见问题与解决方案​配置校验报错 type mismatch accessing filebeat.inputs.0.file_identity’​原因file_identity 是对象类型错误配置为字符串如 file_identity: path。​解决方案删除该错误配置默认按路径标识文件无需显式声明或改为正确对象格式​​file_identity: path: ~ES 写入失败 no matching index template found​原因Filebeat 配置的模板未在 ES 中创建。​解决方案通过 Filebeat 自动推送模板​​​/usr/local/filebeat-es/filebeat setup --index-management -c /usr/local/filebeat-es/filebeat.yml或手动通过 ES API 创建​​curl -X PUThttp://1.2.3.4:5678/_template/nginx_template-HContent-Type: application/json-d{ index_patterns: [nginx_*], settings: {index.number_of_shards:1,index.number_of_replicas:0}, mappings: {dynamic: true} }​首次启动 CPU 飙升​原因Filebeat 全量扫描大日志文件如 8GB 的 access.log。​解决方案确保配置中包含 offset.initial: end并清理注册表文件​​rm -rf /usr/local/filebeat-es/data/registry/filebeat/filestream/* systemctl restart filebeat-es​4. 日志采集成功但 ES 中无数据​原因ES 预处理管道 pipeline-nginx-plaintext 不存在或配置错误。​解决方案创建基础管道测试​​curl -X PUThttp://1.2.3.4:5678/_ingest/pipeline/pipeline-nginx-plaintext-HContent-Type: application/json-d{ description: Nginx日志解析管道, processors: [ {grok: {field: message, patterns: [%{HTTPDATE:time_local_str}]}}, {date: {field: time_local_str, target_field: time_local, formats: [dd/MMM/yyyy:HH:mm:ss Z]}} ] }四、总结​本文基于 Filebeat 8.15.5 版本详细实现了 Nginx access 日志到 ES 8 的采集流程核心亮点包括优化大日志文件采集通过 offset.initial: end 避免 CPU 飙升规范配置与服务管理采用 Systemd 实现开机自启确保服务稳定性避坑指南针对模板匹配、管道配置、权限等常见问题提供解决方案。通过这套流程可实现 Nginx 日志的高效、稳定采集为后续的日志分析、可视化如 Kibana 仪表盘奠定基础。如需扩展可进一步配置日志过滤、字段加工、ES 索引生命周期管理等功能。​​