fix flaky tests in NvmCacheTests

Summary:
This change fixes following flaky tests in NvmCacheTests.
* NvmCacheTest.Delete
* NvmCacheTest.NvmEvicted
* NvmCacheTest.EvictToNvmGetCheckCtime

The root cause of the failures are essentially the same as D42443647 (facebook@5e7ff9a) which fixed the problem for
NvmCacheTest.EvictToNvmGet; we are inserting enough items that could be spilled to NVM cache, where
the NvmCache::put() can be dropped and the item is evicted completely when the delete operations
(and tombstones) issued as part of the insertion are still outstanding. In order to fix the problem,
this change flushes the NVM cache periodically during the insertions.

Also, since this could cause more regions are used, the size of NVM cache needs to be increased.
This change bumps the default size of NVM cache to 200MB (previous 100MB). Also, the size of persist
storage used in the test PersistenceCache has been bumped by 100MB accordingly, i.e., from 400MB to
500MB.

This change addresses the github issue facebook#169

Reviewed By: therealgymmy

Differential Revision: D43592888

fbshipit-source-id: f0968884eb39fb5728b59129e98345df3240f01e

Author: Jaesoo Lee

cachelib/allocator/nvmcache/tests/NvmCacheTests.cpp

@@ -245,7 +245,13 @@ TEST_F(NvmCacheTest, EvictToNvmGetCheckCtime) {
     ASSERT_NE(nullptr, it);
     cache_->insertOrReplace(it);
     keyToCtime.insert({key, it->getCreationTime()});
+    // Avoid any nvm eviction being dropped due to the race with still
+    // outstanding remove operation for insertion
+    if (i % 100 == 0) {
+      nvm.flushNvmCache();
+    }
   }
+  nvm.flushNvmCache();
 
   const auto nEvictions = this->evictionCount() - evictBefore;
   ASSERT_LT(0, nEvictions);
@@ -331,6 +337,11 @@ TEST_F(NvmCacheTest, Delete) {
     auto it = nvm.allocate(pid, key, 15 * 1024);
     ASSERT_NE(nullptr, it);
     nvm.insertOrReplace(it);
+    // Avoid any nvm eviction being dropped due to the race with still
+    // outstanding remove operation for insertion
+    if (i % 100 == 0) {
+      nvm.flushNvmCache();
+    }
   }
   nvm.flushNvmCache();
 
@@ -533,6 +544,11 @@ TEST_F(NvmCacheTest, NvmEvicted) {
     auto it = nvm.allocate(pid, key, allocSize);
     ASSERT_NE(nullptr, it);
     nvm.insertOrReplace(it);
+    // Avoid any nvm eviction being dropped due to the race with still
+    // outstanding remove operation for insertion
+    if (i % 100 == 0) {
+      nvm.flushNvmCache();
+    }
   }
   nvm.flushNvmCache();
 

cachelib/allocator/tests/NvmTestUtils.h

@@ -27,7 +27,7 @@ namespace utils {
 using NavyConfig = navy::NavyConfig;
 inline NavyConfig getNvmTestConfig(const std::string& cacheDir) {
   NavyConfig config{};
-  config.setSimpleFile(cacheDir + "/navy", 100 * 1024ULL * 1024ULL);
+  config.setSimpleFile(cacheDir + "/navy", 200 * 1024ULL * 1024ULL);
   config.setDeviceMetadataSize(4 * 1024 * 1024);
   config.setBlockSize(1024);
   config.setNavyReqOrderingShards(10);

cachelib/persistence/tests/PersistenceCache.h

@@ -213,7 +213,7 @@ class PersistenceCache {
  public:
   const uint32_t kNumKeys = 1024 * 1024;    // 1 million
   const size_t kCacheSize = 100 * kNumKeys; // 100MB
-  const size_t kCapacity = 4 * kCacheSize;  // 400MB
+  const size_t kCapacity = 5 * kCacheSize;  // 500MB
 
   std::unique_ptr<folly::IOBuf> buffer_;
   std::string cacheDir_;