zxf/log-fetch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

优化日志处理流程

Browse Source
This commit is contained in:
luorijun
2025-12-19 17:52:11 +08:00
parent 9b39441571
commit 03fa964669
3 changed files with 44 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
README.md
View File

@@ -18,83 +18,11 @@ ansible/
### 基本用法
```bash
# 必须指定年月日参数
ansible-playbook -i inventory/hosts.yaml playbooks/compress_logs.yaml \
-e "year=2025 month=11 start_day=21 end_day=30"
# 使用便捷脚本
./scripts/compress_logs.sh -y 2025 -m 11 -s 21 -e 30
# 指定单个服务器
ansible-playbook -i inventory/hosts.yaml playbooks/compress_logs.yaml \
--limit jhlog -e "year=2025 month=11 start_day=21 end_day=30"
```
**默认行为**:
- 压缩日志到 tar.gz
- 下载到 `/mnt/d/logs/<host_name>/<year>/<month>/`
- 删除原始文件夹和远程压缩包
- 支持断点续传
## 可配置参数
| 参数 | 默认值 | 说明 |
|------|--------|------|
| `year` | **必填** | 年份 |
| `month` | **必填** | 月份 |
| `start_day` | **必填** | 起始日期 |
| `end_day` | **必填** | 结束日期 |
| `log_base_path` | `/data/roslog/roslog` | 日志路径 |
| `local_download_path` | `/mnt/d/logs` | 本地下载路径 |
| `remove_after_compress` | `true` | 删除原文件夹 |
| `download_to_local` | `true` | 下载到本地 |
| `remove_local_archives` | `true` | 删除远程压缩包 |
## 常用场景
### 只压缩不删除
```bash
ansible-playbook -i inventory/hosts.yaml playbooks/compress_logs.yaml \
-e "year=2025 month=11 start_day=21 end_day=30 remove_after_compress=false"
```
### 只压缩不下载
```bash
ansible-playbook -i inventory/hosts.yaml playbooks/compress_logs.yaml \
-e "year=2025 month=12 start_day=1 end_day=31 download_to_local=false"
```
### 保留远程压缩包
```bash
ansible-playbook -i inventory/hosts.yaml playbooks/compress_logs.yaml \
-e "year=2025 month=11 start_day=1 end_day=20 remove_local_archives=false"
```
## 断点续传
使用 rsync 实现断点续传，网络中断后重新执行会自动从中断点继续下载。
## 注意事项
- 必须显式指定时间参数
- 确保服务器上安装了 rsync
- 确保有足够的磁盘空间
- 首次使用建议小范围测试
## 测试建议
```bash
# check 模式
ansible-playbook -i inventory/hosts.yaml playbooks/compress_logs.yaml \
--limit jhlog \
-e "year=2025 month=11 start_day=21 end_day=21 remove_after_compress=false" \
--check
# 单天测试
ansible-playbook -i inventory/hosts.yaml playbooks/compress_logs.yaml \
--limit jhlog \
-e "year=2025 month=11 start_day=21 end_day=21 remove_after_compress=false"
# 拉取日志并删除远程存档
# -y 年
# -m 月 ，必须是两位数
# -s 日，开始，必须是两位数
# -e 日，结束，必须是两位数
# 将拉取 s - e 之间日期的所有日志，会断点续传
./scripts/compress_logs.sh -y 2025 -m 12 -s 01 -e 10
```

15
inventory/hosts.yaml
View File

@@ -1,26 +1,31 @@
logs:
hosts:
43.226.58.253:
name: jhlog
label: jhlog
ansible_user: wyk
ansible_ssh_pass: adjg9815...
ansible_become_pass: adjg9815...
106.119.167.38:
name: jhlogc
label: jhlogc
ansible_user: root
ansible_ssh_pass: adjg9815...
ansible_become_pass: adjg9815...
43.226.59.253:
name: jglog
label: jglog
ansible_user: wyk
ansible_ssh_pass: adjg9815...
ansible_become_pass: adjg9815...
106.119.166.79:
name: jglogc
label: jglogc
ansible_user: root
ansible_ssh_pass: adjg9815...
ansible_become_pass: adjg9815...
125.122.23.244:
name: mglog
label: mglog
ansible_user: root
ansible_ssh_pass: adjg9815...
ansible_become_pass: adjg9815...

52
playbooks/compress_logs.yaml
View File

@@ -1,5 +1,5 @@
---
- name: 压缩和清理服务器日志
- name: 自动拉取服务器日志
hosts: logs
gather_facts: false
vars:
@@ -11,26 +11,22 @@
remove_local_archives: true
tasks:
- name: 验证必填参数
- name: 缺少必填参数
ansible.builtin.fail:
msg: "缺少必填参数！必须提供: year, month, start_day, end_day"
when: year is not defined or month is not defined or start_day is not defined or end_day is not defined
- name: 显示将要处理的日志路径
ansible.builtin.debug:
msg: "处理路径: {{ log_base_path }}/{{ year }}/{{ month }}, 日期范围: {{ start_day }}-{{ end_day }}"
- name: 确保日志目录存在
- name: 检查日志目录
ansible.builtin.stat:
path: "{{ log_base_path }}/{{ year }}/{{ month }}"
register: log_dir
- name: 检查日志目录是否存在
- name: 日志目录不存在
ansible.builtin.fail:
msg: "日志目录 {{ log_base_path }}/{{ year }}/{{ month }} 不存在"
when: not log_dir.stat.exists
- name: 压缩指定日期范围的日志文件夹
- name: 压缩日志
ansible.builtin.shell: |
cd {{ log_base_path }}/{{ year }}/{{ month }}
for day in $(seq -f "%02g" {{ start_day }} {{ end_day }}); do
@@ -46,39 +42,46 @@
become: true
register: compress_result
- name: 显示压缩结果
- name: 展示压缩结果
ansible.builtin.debug:
var: compress_result.stdout_lines
- name: 列出已生成的压缩文件
ansible.builtin.find:
paths: "{{ log_base_path }}/{{ year }}/{{ month }}"
patterns: "*.tar.gz"
register: compressed_files
- name: 删除已压缩的原始文件夹
- name: 删除原始日志
ansible.builtin.shell: |
cd {{ log_base_path }}/{{ year }}/{{ month }}
for day in $(seq -f "%02g" {{ start_day }} {{ end_day }}); do
if [ -d "$day" ] && [ -f "${day}.tar.gz" ]; then
echo "删除原始目录: $day"
rm -rf $day
else
echo "文件 ${day} 或 ${day}.tar.gz 不存在，跳过"
fi
done
args:
executable: /bin/bash
become: true
when: remove_after_compress
when: remove_after_compress | bool
register: delete_result
- name: 确保本地下载目录存在
- name: 展示删除结果
ansible.builtin.debug:
var: delete_result.stdout_lines
- name: 收集压缩文件
ansible.builtin.find:
paths: "{{ log_base_path }}/{{ year }}/{{ month }}"
patterns: "*.tar.gz"
register: compressed_files
- name: 检查下载目录
ansible.builtin.file:
path: "{{ local_download_path }}/{{ label }}/{{ year }}/{{ month }}"
state: directory
delegate_to: localhost
when: download_to_local
when: download_to_local | bool
become: false
- name: 下载压缩文件到本地
- name: 下载到本地
ansible.posix.synchronize:
src: "{{ log_base_path }}/{{ year }}/{{ month }}/{{ item.path | basename }}"
dest: "{{ local_download_path }}/{{ label }}/{{ year }}/{{ month }}/"
@@ -87,15 +90,14 @@
- "--partial"
- "--progress"
with_items: "{{ compressed_files.files }}"
when: download_to_local
become: true
when: download_to_local | bool
- name: 删除服务器上的压缩文件
- name: 删除压缩文件
ansible.builtin.file:
path: "{{ item.path }}"
state: absent
with_items: "{{ compressed_files.files }}"
when: remove_local_archives and download_to_local
when: remove_local_archives | bool and download_to_local | bool
become: true
- name: 完成信息