Cache size and estimated number of entries

[joanbalaguero]

Hi Arthur,

One question about cache size and entries.

I have a cache of 12,288Mb size (12GB). Applying your formula (mem ≈ 49 * next_power_of_two( N * (1 + ghost) ) with N number of estimated entries and ghost allocation = 0), and if I'm not wrong, the estimated number of entries would be 134,217,728

So I apply these 2 settings when I build the cache and let it run for hours. Then, I see this in my log:

cache = [ entries: 211227840 / 134217728, weight: 12884898240 / 12884901888 ]

As you can see, the weight is respected, but the number of entries is much higher (211227840) than the estimated (134217728).

Maybe am I wrong computing the number of entries? Or quick cache is just limiting the cache growing by size, not by number of entries?

If number of entries is just an estimation for you, and I'm giving a wrong estimation (134217728 is much fewer than 211227840), will this provoke any kind of performance issues?

Thanks,
Joan.

[arthurprs]

That's right, the cache will only limit growth by "weight".

If number of entries is just an estimation for you, and I'm giving a wrong estimation (134217728 is much fewer than 211227840), will this provoke any kind of performance issues?

The estimation is only used to limit the ghost entries, and in very small caches, it'll also be used to limit the number of internal shards so that each shard is not too small.

Grossly misestimating that may have some, but often mild, effect on the cache eviction choices. In your case, with ghost entries set to 0, that's not applicable.

[joanbalaguero]

Hi Arthur,

One more question about the estimated items capactiy. I see there are several post around this matter, but I had an issue with it and I would like to comment it to you.

I'm using 2 quick caches, the big one that can store up to 50GB of data, easy to setup because each entry occuppies 20 bytes exactly.
The small one, that stores one 'u32' and one 'Bytes' (that can store documents from very differents lengths) and uses just 1 GB

My issue is with the small qc (L1 cache), and it happened because of an error setting it up:

const CACHE_INTERNAL_OVERHEAD: usize = 21;               // 'quick_cache' overhead
const CACHE_GHOST_ALLOCATION_PERCENT: f64 = 0.0;
const CACHE_L1_KEY_SIZE: usize = 8;                      // cache_key size (i64)
const CACHE_L1_VALUE_SIZE: usize = 4;                    // 'expiration time' size (u32)
const CACHE_L1_TOTAL_ENTRY_SIZE: usize = ((CACHE_L1_KEY_SIZE + CACHE_L1_VALUE_SIZE + CACHE_INTERNAL_OVERHEAD) as f64 * (1.0 + CACHE_GHOST_ALLOCATION)) as usize;

pub struct CacheMap {
    l1: Cache<i64, CacheL1Value, CacheL1ValueWeighter, BuildNoHashHasher<i64>, DefaultLifecycle<i64, CacheL1Value>>
}
pub struct CacheL1Value {
    expiration_time: u32,
    document: Bytes
}
impl CacheL1Value {
    pub fn new(expiration_time: u32, document: Bytes) -> CacheL1Value {
        CacheL1Value { expiration_time, document }
    }

    pub fn entry_size(&self) -> usize {
        CACHE_L1_TOTAL_ENTRY_SIZE + self.document.len()
    }
}
pub struct CacheL1ValueWeighter;
impl Weighter<i64, CacheL1Value> for CacheL1ValueWeighter {
    fn weight(&self, _key: &i64, val: &CacheL1Value) -> u64 {
        val.entry_size() as u64
    }
}

Now, to create the quick cache:

let l1_cache_size_bytes = CONFIG.cache.io_uring.l1_cache_size_mb as usize * 1024 * 1024;
let l1_cache_items_capacity = l1_cache_size_bytes / (CACHE_L1_TOTAL_ENTRY_SIZE + 3000 as usize);

Where:
- The size is 'l1_cache_size_bytes', 1024 * 1024 * 1024 = 1GB.
- The items capacity is 1GB / (CACHE_L1_TOTAL_ENTRY_SIZE + 3000), where this 3000 is the average response size of the documents to be inserted (in bytes).

This works perfectly, after 5 days executing the applications, the occuppied RAM keeps stable around 52GB, 50 GB big cache + 1 GB L1 cache + 1GB of other things.

But before this, I made a mistake and set up the "l1_cache_items_capacity" in this way:
let l1_cache_items_capacity = l1_cache_size_bytes / CACHE_L1_TOTAL_ENTRY_SIZE;

This crashed the app after some hours (5-6) executing due to the linux OOM killer. This is what I saw:

1. My log showed, regarding the big cache, entries: 112243886, weight: 5499950414. So 5GB on the big cache.
2. I don't have info about the L1.
3. But This is the RAM used:

Rss: 11686796 kB
Pss: 11684531 kB
Pss_Dirty: 11679888 kB
Pss_Anon: 11679136 kB
Pss_File: 5395 kB
Pss_Shmem: 0 kB
Shared_Clean: 2456 kB
Shared_Dirty: 0 kB
Private_Clean: 4452 kB
Private_Dirty: 11679888 kB
Referenced: 11686796 kB
Anonymous: 11679136 kB

Around 11GB, so more than the double.

And this is reproducible, every time I overestimate the number of entries in a big factor, there is a lot of RAM consumed until the OOM killer kills it.

So, my question: reading the other posts, it seems that failing on estimating the items capacity should not be a problem. But in my case this crashes the app, and taking into account that this parameter, the average response size of the documents to be inserted, fixed now to 3000, will be configurable by the end user (so he could set up a very wrong value), how really dangerous is to fail estimating it?

Thanks!

Joan.

[joanbalaguero]

Well, let me investigate a bit more about this ...

[arthurprs]

At first, I suspected that severely underestimating the count may lead the cache to accumulate a lot of "ghost entries" (entries w/o a key/value but a hash to the original key).

But since you are setting "ghost allocation percentage" as 0, then that's probably not it.