0%

leveldb源码阅读记录-各组件默认大小总结

至此,我们已经基本分析完整个leveldb,本文给出leveldb中的各组件的默认大小。

  • log 文件,由一系列的32kb物理块组成。

  • manifest 文件,目前来看未限制其大小,但是在系统Open时,有一次机会重新(文件大小超过2M)开辟一个新的manifest,并移除旧文件。这部分在ReuseManifest函数中有体现:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
bool VersionSet::ReuseManifest(const std::string& dscname,
const std::string& dscbase) {
...
if (!ParseFileName(dscbase, &manifest_number, &manifest_type) ||
manifest_type != kDescriptorFile ||
!env_->GetFileSize(dscname, &manifest_size).ok() ||
// Make new compacted MANIFEST if old one is too big
manifest_size >= TargetFileSize(options_)) {
return false;
}
...
}

// TargetFileSize函数
static size_t TargetFileSize(const Options* options) {
return options->max_file_size;
}

// Leveldb will write up to this amount of bytes to a file before
// switching to a new one.
// Most clients should leave this parameter alone. However if your
// filesystem is more efficient with larger files, you could
// consider increasing the value. The downside will be longer
// compactions and hence longer latency/performance hiccups.
// Another reason to increase this parameter might be when you are
// initially populating a large database.
size_t max_file_size = 2 * 1024 * 1024;
  • memtable: 4MB,源自:leveldb/options.h
1
2
3
4
5
6
7
8
9
// Amount of data to build up in memory (backed by an unsorted log
// on disk) before converting to a sorted on-disk file.
//
// Larger values increase performance, especially during bulk loads.
// Up to two write buffers may be held in memory at the same time,
// so you may wish to adjust this parameter to control memory usage.
// Also, a larger write buffer will result in a longer recovery time
// the next time the database is opened.
size_t write_buffer_size = 4 * 1024 * 1024;
  • sstable:2MB,来源:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
static uint64_t MaxFileSizeForLevel(const Options* options, int level) {
// We could vary per level to reduce number of files?
return TargetFileSize(options);
}

// Leveldb will write up to this amount of bytes to a file before
// switching to a new one.
// Most clients should leave this parameter alone. However if your
// filesystem is more efficient with larger files, you could
// consider increasing the value. The downside will be longer
// compactions and hence longer latency/performance hiccups.
// Another reason to increase this parameter might be when you are
// initially populating a large database.
size_t max_file_size = 2 * 1024 * 1024;

具体应在,在DoCompactionWork函数内部,有一段代码:

1
2
3
4
5
6
7
8
// Close output file if it is big enough
if (compact->builder->FileSize() >=
compact->compaction->MaxOutputFileSize()) { // 此时的MaxOutputFileSize()返回的就是2MB
status = FinishCompactionOutputFile(compact, input);
if (!status.ok()) {
break;
}
}
  • SStable的datablock, 变长,但最小为4kb

  • 各level的大小

    • level0,不按大小算,而是按照文件个数算,level0默认为4个文件触发一次compaction。其他还有两个配置,1个8个文件,在MakeRoomForWrite函数中使用,一个12,达到12个文件,系统将暂停写入。

      1
      2
      3
      4
      5
      6
      7
      8
      9
      // Level-0 compaction is started when we hit this many files.
      static const int kL0_CompactionTrigger = 4;


      // Soft limit on number of level-0 files. We slow down writes at this point.
      static const int kL0_SlowdownWritesTrigger = 8;

      // Maximum number of level-0 files. We stop writes at this point.
      static const int kL0_StopWritesTrigger = 12;
    • 其他level, level1 10M, level 2 100M, 逐层递乘10.

文章对你有帮助?打赏一下作者吧