Of how much of a file is in RAM
Memory my friend !
Nowadays RAM is so cheap, you might be tempted to just rely on his database being in RAM to get the wanted performance. Disk is just there for persistence.
Many people talk on the web about their production setup bein in TmpFs, or using the RAMDirectory.
But isn’t your OS supposed to make sure that the stuff your accessing is page cache? Let’s see how we can measure how much of your db/index/data is in page cache.
What’s page cache anyway?
It takes from 5 to 10ms to read something from a random part of your hard disk. Accessing data in RAM on the other hand, takes between 50 ns and 100 ns. It is only natural for the OS to make sure that the same data is not loaded twice if we can afford caching it in RAM. That’s precisely the role of the page cache.
If you are on Linux or MacOS, here is a very simple experiment to see the page cache in action. Go find a fat and useless file sleeping on your hard disk. That DivX of Beethoven 2 will do. Do not open it, just run the following command twice
time cat ./free-willy-2.mpg > /dev/null
The command reads your whole file and print out the duratio of the operation. The second time, you should get a pretty nice performance improvement. By reading the file the first time, we made sure that the file was sitting in RAM for the second turn.
This trick is actually pretty legit. You can actually warmup files by cat’ing them to your good old /dev/null.
pmap to the rescue
Assuming your database is using memory mapping (mmap), pmap will actually give a nice picture of what’s in your virtual memory and help you a bit about how much of your database file are in RAM.
The default parameters however won’t be helpful to know how much of your files are in RAM. To know that, you need to stick it the -x param.
pmap -x <pid>
You can find the pid of your process by running
ps -aux
Let’s take a look at a very cold Solr in which I just pushed 1M+ documents.
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 4 4 0 r-x-- java
0000000000600000 4 4 4 rw--- java
000000000234e000 132 12 12 rw--- [ anon ]
00000006fae00000 56704 27564 27564 rw--- [ anon ]
00000006fe560000 4800 0 0 ----- [ anon ]
00000006fea10000 22464 0 0 rw--- [ anon ]
0000000700000000 146304 144384 144384 rw--- [ anon ]
0000000708ee0000 23744 0 0 ----- [ anon ]
000000070a610000 2626176 0 0 rw--- [ anon ]
00000007aaab0000 1398080 1387668 1387668 rw--- [ anon ]
00007f6c071fe000 280 4 0 r--s- _1.fdx
00007f6c07244000 64492 4 0 r--s- _1.fdt
00007f6c0b13f000 36 4 0 r--s- _1_nrm.cfs
00007f6c0b148000 1460 540 0 r--s- _1_Lucene40_0.tim
00007f6c0b2b5000 3472 4 0 r--s- _1_Lucene40_0.prx
00007f6c0b619000 4732 184 0 r--s- _1_Lucene40_0.frq
00007f6c0bab8000 284 4 0 r--s- _2.fdx
00007f6c0baff000 66200 4 0 r--s- _2.fdt
00007f6c0fba5000 36 4 0 r--s- _2_nrm.cfs
00007f6c0fbae000 1392 488 0 r--s- _2_Lucene40_0.tim
00007f6c0fd0a000 3532 4 0 r--s- _2_Lucene40_0.prx
00007f6c1007d000 4892 164 0 r--s- _2_Lucene40_0.frq
00007f6c3f21f000 284 4 0 r--s- _d.fdx
00007f6c3f266000 69544 4 0 r--s- _d.fdt
00007f6c43650000 69224 4 0 r--s- _e.fdt
00007f6c479ea000 280 4 0 r--s- _f.fdx
00007f6c47a30000 68916 4 0 r--s- _f.fdt
00007f6c4bd7d000 68552 4 0 r--s- _g.fdt
00007f6c54f25000 705388 4 0 r--s- _i.fdt
00007f6c80000000 132 8 8 rw--- [ anon ]
00007f6c80021000 65404 0 0 ----- [ anon ]
00007f6d9789d000 1016 120 120 rw--- [ anon ]
00007f6d9799b000 32 28 0 r-x-- libmanagement.so
00007f6d979a3000 2044 0 0 ----- libmanagement.so
00007f6d9c296000 1016 92 92 rw--- [ anon ]
00007f6d9c394000 12 12 0 r--s- lucene-highlighter-4.0.0.jar
Anonymous is all the stuff that is not associated with a file, in this case your Java heap. You should see shared native libraries and jar. They indeed are mapped in your process virtual memory. At this point you need to locate which files are the actual data of your database. They may not appear here if you are using a database working mainly in anonymous space, or if your database does not rely on mmap to access the data.
| In my case, we see that the file of our index are mapped into memory. The so-called posting lists are the file matching the _*_Lucene.(frq | tim | prx | tip). |
Let’s check how much of these are in RAM.
RSS stands for resident memory. It’s the part of your virtual memory that is actually sitting on your actual physical memory rather than on your file in your filesystem (for mmapped files) or your swap for anonymous memory.
Wait a minute… pmap showing its limits.
Ok, let’s check whether the RSS column is working out as expected.
If we cat _2_Lucene40_0.prx to /dev/null we saw that it was loaded into RAM. Right now only 476 / 688 KBytes are in RAM, we should observe this figure to go 100%.
cat _2_Lucene40_0.prx > /dev/null
pmap -x 10988 | grep _2_Lucene40_0.prx
gives me back :
00007f6c0fd0a000 3532 4 0 r--s- _2_Lucene40_0.prx
This does not work as expected. Why the hell did this happen?
Minor and major page faults
MMap mapped a segment of the virtual memory of our program to a segment of the disk. All this operation is lazy and at this point nothing was read from disk or anything.
On the first attempt to access data from this virtual memory range, the OS will do whatever necessary to map the virtual memory page to a physical memory page that holds the same information as the disk.
If at this moment, the file is actually in page cache, the OS just have to create the mapping between the virtual memory and the page cache (yes most of the time mmap are actually direcly mapped to the page cache!). This is usually called minor page fault.
If however the page is not in page cache, we need to wait for the system to read the info from the disk and put it in page cache. This is the dreaded major page fault.
If our process tried to access a segment not marked as in resident in our Lucene file right now, this would result in a minor page fault… but not a major page fault. The OS would just have to map the virtual memory to the already filled page cache.
You can check for the number of page fault (minor and major) by using ps.
ps -o min_flt,maj_flt
# What can we do? mincore to the rescue.
A database may mmap and munmap files or you may restart your process, or a process may mmap a file that have been just created by another process. Since what we really want to avoid is major page fault, pmap’s figures are not exactly reliable.
I don’t know any linux command that answer this question directly, but [mincore](http://man7.org/linux/man-pages/man2/mincore.2.html) is a system call that makes it possible to know whether accessing a page virtual memory page will require an IO or not.
We can therefore mmap a file, and ask mincore whether accessing each or each byte would trigger a major page fault or not.
I wrote a little utility doing that, and you can find it on github.
Let’s use it to take a look at our _2_Lucene40_0.prx file again.
$ isresident _2_Lucene40_0.prx
FILE RSS SIZE PERCT _2_Lucene40_0.prx 3530 3530 100 %
Hurray ! We indeed observe that the file is indeed completely in RAM.
You can run it use wildcard to use it on a directory as well.
$ ./isresident /usr/lib/*