ZklMao-Space
large picture of the cover

Redis 持久化篇 - AOF

配置文件在哪里？怎么配置？

  • 配置文件名 redis.conf

  • 路径在/etc/redis/redis.conf

以下是来自官方配置文件模板的RDB配置内容 https://download.redis.io/redis-stable/redis.conf

############################## APPEND ONLY MODE ###############################

# By default Redis asynchronously dumps the dataset on disk. This mode is
# good enough in many applications, but an issue with the Redis process or
# a power outage may result into a few minutes of writes lost (depending on
# the configured save points).
#
# The Append Only File is an alternative persistence mode that provides
# much better durability. For instance using the default data fsync policy
# (see later in the config file) Redis can lose just one second of writes in a
# dramatic event like a server power outage, or a single write if something
# wrong with the Redis process itself happens, but the operating system is
# still running correctly.
#
# AOF and RDB persistence can be enabled at the same time without problems.
# If the AOF is enabled on startup Redis will load the AOF, that is the file
# with the better durability guarantees.
#
# Note that changing this value in a config file of an existing database and
# restarting the server can lead to data loss. A conversion needs to be done
# by setting it via CONFIG command on a live server first.
#
# Please check https://redis.io/topics/persistence for more information.

appendonly no

# The base name of the append only file.
#
# Redis 7 and newer use a set of append-only files to persist the dataset
# and changes applied to it. There are two basic types of files in use:
#
# - Base files, which are a snapshot representing the complete state of the
#   dataset at the time the file was created. Base files can be either in
#   the form of RDB (binary serialized) or AOF (textual commands).
# - Incremental files, which contain additional commands that were applied
#   to the dataset following the previous file.
#
# In addition, manifest files are used to track the files and the order in
# which they were created and should be applied.
#
# Append-only file names are created by Redis following a specific pattern.
# The file name's prefix is based on the 'appendfilename' configuration
# parameter, followed by additional information about the sequence and type.
#
# For example, if appendfilename is set to appendonly.aof, the following file
# names could be derived:
#
# - appendonly.aof.1.base.rdb as a base file.
# - appendonly.aof.1.incr.aof, appendonly.aof.2.incr.aof as incremental files.
# - appendonly.aof.manifest as a manifest file.

appendfilename "appendonly.aof"

# For convenience, Redis stores all persistent append-only files in a dedicated
# directory. The name of the directory is determined by the appenddirname
# configuration parameter.

appenddirname "appendonlydir"

# The fsync() call tells the Operating System to actually write data on disk
# instead of waiting for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
#
# Redis supports three different modes:
#
# no: don't fsync, just let the OS flush the data when it wants. Faster.
# always: fsync after every write to the append only log. Slow, Safest.
# everysec: fsync only one time every second. Compromise.
#
# The default is "everysec", as that's usually the right compromise between
# speed and data safety. It's up to you to understand if you can relax this to
# "no" that will let the operating system flush the output buffer when
# it wants, for better performances (but if you can live with the idea of
# some data loss consider the default persistence mode that's snapshotting),
# or on the contrary, use "always" that's very slow but a bit safer than
# everysec.
#
# More details please check the following article:
# http://antirez.com/post/redis-persistence-demystified.html
#
# If unsure, use "everysec".

# appendfsync always
appendfsync everysec
# appendfsync no

# When the AOF fsync policy is set to always or everysec, and a background
# saving process (a background save or AOF log background rewriting) is
# performing a lot of I/O against the disk, in some Linux configurations
# Redis may block too long on the fsync() call. Note that there is no fix for
# this currently, as even performing fsync in a different thread will block
# our synchronous write(2) call.
#
# In order to mitigate this problem it's possible to use the following option
# that will prevent fsync() from being called in the main process while a
# BGSAVE or BGREWRITEAOF is in progress.
#
# This means that while another child is saving, the durability of Redis is
# the same as "appendfsync no". In practical terms, this means that it is
# possible to lose up to 30 seconds of log in the worst scenario (with the
# default Linux settings).
#
# If you have latency problems turn this to "yes". Otherwise leave it as
# "no" that is the safest pick from the point of view of durability.

no-appendfsync-on-rewrite no

# Automatic rewrite of the append only file.
# Redis is able to automatically rewrite the log file implicitly calling
# BGREWRITEAOF when the AOF log size grows by the specified percentage.
#
# This is how it works: Redis remembers the size of the AOF file after the
# latest rewrite (if no rewrite has happened since the restart, the size of
# the AOF at startup is used).
#
# This base size is compared to the current size. If the current size is
# bigger than the specified percentage, the rewrite is triggered. Also
# you need to specify a minimal size for the AOF file to be rewritten, this
# is useful to avoid rewriting the AOF file even if the percentage increase
# is reached but it is still pretty small.
#
# Specify a percentage of zero in order to disable the automatic AOF
# rewrite feature.

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# An AOF file may be found to be truncated at the end during the Redis
# startup process, when the AOF data gets loaded back into memory.
# This may happen when the system where Redis is running
# crashes, especially when an ext4 filesystem is mounted without the
# data=ordered option (however this can't happen when Redis itself
# crashes or aborts but the operating system still works correctly).
#
# Redis can either exit with an error when this happens, or load as much
# data as possible (the default now) and start if the AOF file is found
# to be truncated at the end. The following option controls this behavior.
#
# If aof-load-truncated is set to yes, a truncated AOF file is loaded and
# the Redis server starts emitting a log to inform the user of the event.
# Otherwise if the option is set to no, the server aborts with an error
# and refuses to start. When the option is set to no, the user requires
# to fix the AOF file using the "redis-check-aof" utility before to restart
# the server.
#
# Note that if the AOF file will be found to be corrupted in the middle
# the server will still exit with an error. This option only applies when
# Redis will try to read more data from the AOF file but not enough bytes
# will be found.

什么时候触发AOF？

这个取决于配置文件redis.conf 配置项

appendonly no/yes

# The fsync() call tells the Operating System to actually write data on disk
# instead of waiting for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
#
# Redis supports three different modes:
#
# no: don't fsync, just let the OS flush the data when it wants. Faster.
# always: fsync after every write to the append only log. Slow, Safest.
# everysec: fsync only one time every second. Compromise.

appendfsync always
appendfsync everysec
appendfsync no

  • appendonly

    • yes 开启

    • no 不开

  • appendfsync

    • always: 执行一句写一句

    • everysec: 每秒一次调用 syscall write(...) 将 sds aof_buffer 写入到内核缓冲区，并调用fsync() 刷盘

    • no: 不主动写aof文件 只调syscall write(...) 将 sds aof_buffer 写入到内核缓冲区，由操作系统定期刷盘(linux 大概30s 刷一次，具体得看配置项)

综上，触发时机如下：

  1. 主eventloop中上次事件处刚处理完，空闲等待下次事件来的时候，调用的beforesleep(...) 里机会调用 redis封装的flushAppendedOnlyFile(...) 函数。 -- 总的来说就是主事件循环空闲间隙就会调用

  2. redis 周期函数servercron到期调用

  3. redis 关闭服务

  4. 使用命令关闭AOF

AOF 写入细节

redis 用sds 定义了一个 aof_buf 独享，用于作redis 用户写入缓冲区

sds aof_buf; /* AOF buffer, written before entering the event loop */

AOF写入源码

// file: aof.c
/* Perform the fsync if needed. 如果需要的话，执行fsync操作 */
// 检查`server.aof_fsync`的值是否为`AOF_FSYNC_ALWAYS`，表示每次写入AOF文件后都需要执行fsync操作。
if (server.aof_fsync == AOF_FSYNC_ALWAYS) {
    /* redis_fsync is defined as fdatasync() for Linux in order to avoid
     * flushing metadata. */
    // 在Linux系统中，`redis_fsync`被定义为`fdatasync()`，这样做是为了避免同时将文件的元数据（如修改时间）也写入磁盘。
    // 监控fsync操作的延迟
    latencyStartMonitor(latency);
    redis_fsync(server.aof_fd); /* Let's try to get this data on the disk fsync操作，将AOF文件的数据写入磁盘。 */
    // 结束监控fsync操作的延迟。
    latencyEndMonitor(latency);
    // 检查aof-fsync-always 字段配置，如果开启的话，将fsync操作的延迟添加到监控样本中。
    latencyAddSampleIfNeeded("aof-fsync-always",latency);
    // 更新最后一次fsync时间戳
    server.aof_last_fsync = server.unixtime;
}
// appendfsync 字段配置是否为 everysec 对应宏:`AOF_FSYNC_EVERYSEC`，
// 是否超时上次fsync操作的时间 
else if ((server.aof_fsync == AOF_FSYNC_EVERYSEC &&
            server.unixtime > server.aof_last_fsync)) {
    // 如果没有正在进行的同步操作，那么在后台执行fsync操作。
    if (!sync_in_progress) aof_background_fsync(server.aof_fd); // aof_background_fsync(server.aof_fd); 使用后再线程刷盘
    server.aof_last_fsync = server.unixtime;
}

AOF 写入步骤

  1. 将写入内容，追加到sds aof_buf

  2. 定时调用 syscall 将aof_buf 中的内容写入到内核缓冲区(page cache), redis 里会调用flushAppenedOnlyFile()这个函数，这个函数里会调用syscall write(...); flushAppendedOnlyFile(...)的驱动事件有四种:

    1. 主eventloop中上次事件处刚处理完，空闲等待下次事件来的时候，调用的beforesleep(...) 里机会调用 redis封装的flushAppendedOnlyFile(...) 函数。 -- 总的来说就是主事件循环空闲间隙就会调用

    2. redis 周期函数servercron到期调用

    3. redis 关闭服务

    4. 使用命令关闭AOF

  3. 将系统内核缓冲区中的数据写入磁盘(刷盘) -- 看上面源码，这里的刷盘也是在flushAppendOlnyFile(...) 函数中。

flushAppendOnlyFile(...) 源码

// file: aof.c

/* Flushes the append only file buffer on disk. */
void flushAppendOnlyFile(void) {
    /* Write the append only file buffer on disk. */
    if (server.aof_buf && sdslen(server.aof_buf) > 0) {
        /* Perform the write operation. */
        if (write(server.aof_fd, server.aof_buf, sdslen(server.aof_buf)) == -1) {
            serverLog(LL_WARNING, "Unable to write to the append only file: %s", strerror(errno));
            exit(1);
        }
        /* Update the AOF current size. */
        server.aof_current_size += sdslen(server.aof_buf);
        /* Reset the AOF buffer. */
        sdsfree(server.aof_buf);
        server.aof_buf = sdsempty();
    }

    /* Sync data on disk. */
    if (server.aof_fsync == AOF_FSYNC_ALWAYS) {
        /* Perform the fsync if needed. */
        if (fsync(server.aof_fd) == -1) {
            serverLog(LL_WARNING, "Unable to fsync the append only file: %s", strerror(errno));
            exit(1);
        }
    } else if (server.aof_fsync == AOF_FSYNC_EVERYSEC) {
        /* Check if a second has passed since the last fsync. */
        if ((server.unixtime % 1) == 0 && server.aof_last_fsync_offset != server.aof_current_size) {
            /* Perform the fsync. */
            if (fsync(server.aof_fd) == -1) {
                serverLog(LL_WARNING, "Unable to fsync the append only file: %s", strerror(errno));
                exit(1);
            }
            /* Update the AOF last fsync. */
            server.aof_last_fsync = server.unixtime;
            server.aof_last_fsync_offset = server.aof_current_size;
        }
    }
}

// 这段代码展示了 `flushAppendOnlyFile` 函数的基本工作流程：
/*******************************************************
1. 检查 AOF 缓冲区（`server.aof_buf`）是否有数据需要写入。
2. 如果有数据，使用 `write` 系统调写入数据到 AOF 文件。
3. 更新 AOF 当前文件大小（`server.aof_current_size`）。
4. 重置 AOF 缓冲区。
5. 根据配置的 `fsync` 策略，执行 `fsync` 或 `fdatasync` 操作以确保数据被同步到磁盘上。
*******************************************************/

// 请注意，这个代码是一个简化版本，实际的 Redis 源码可能包含更多的错误处理和边缘情况处理。您可以在 Redis 的官方 GitHub 仓库中找到完整的源码。

AOF 什么时候刷盘?

AOF工作流程大致分为三部，命令追加到aof_buf -> 向系统发起写请求syscall 将aof_buf内容 写入到内核缓冲区 -> 根据配置文件刷盘/或者由系统刷盘. AOF 触发时机有四个:

  • 关闭服务 shutdown

  • 通过指令关闭AOF

  • 周期函数调用

  • 空闲事件检查函数beforesleep(...)

其中调用根据配置的appendfsync刷盘策略也有不同得刷盘频率，具体见上...

AOF 重写

AOF 重写主要依赖配置文件这三项配置项

# 自动触发:根据 auto-aof-rewrite-min-size和auto-aof-rewrite-percentage 参数确定自动触发时机。
 
# auto-aof-rewrite-min-size:表示运行AOF重写时文件最小体积，默认为64M 
# auto-aof-rewrite-percentage:代表当前AOF文件空间(aof_current_size)和 上一次重写后AOF文件空间(aof_base_size)的比值。默认100%
# aof-rewrite-incremental-fsync:重写缓冲区定量刷盘(写入磁盘)数据量阈值,默认32M

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-rewrite-incremental-fsync 32mb

如何触发AOF重写

触发重写有两种方式:

  • 达到配置条件触发:

  • 手动命令bgrewriteaof

重写流程

  • 检查受否有子进程正在进行重写AOF,若有，返回下次重写，无，fork

  • 创建临时AOF文件，获取数据库中的数据，重写redis 命令，重写内容加入到重写用户缓冲区 aof_rewrite_buf 中 (这里如果有新的命令过来，会将新命令追加到aof_rewrite_buf中)

    • 重写期间如果有新命令:

1.将新命令加入到主进程中的aof_buf 并写入到旧AOF文件中

2.将新命令加入到重写进程中的aof_rewrite_buf 并写入新AOF文件中

  • 根据配置文件 字段 aof-rewrite-incremental-fsync 定量写入磁盘

  • 重写完成，替换旧AOF文件

再偷一张大佬的图

site logo

And if we

© 2025 ZklMao-Space