I believe I had a bit of a breakthrough with regards to solving the server lag problem. I just posted this in the mod forums in reply to a discussion we've been having, but figured it wouldn't hurt to get more input from other people on the matter.
- MLC SSD - it degrades too quickly. A 60GB drive lasts about 3 months with the current level of usage. Double the size of the drive, and you'll double the lifespan, but also double or more the cost. Also, there is no TRIM support on the Server 2008 I am running.
- SLC SSD - Too expensive. At $375 for 32GB, it could certainly run the minecraft servers, but without TRIM support, it would slow down just as the current SSD has. Reports say that even if I did have an OS capable of sending TRIM commands, the Intel SLC SSD only supports a few Intel chipsets for sending such commands. Other SLC SSDs are largely unproven and even more expensive.
- Velociraptor HDD - Too slow. Random access time is roughly 15x that of an SSD.
- Any 7200 RPM HDD - Even slower random access times. Not going to help us.
- RAID of classic HDD - Unreliable, and wouldn't help random access times, would only help sustained r/w speeds, which is the opposite of what we need with 4KB files.
- Shortstroking VR HDD - Might work. It's certainly been proven that random access times can be significantly improved by shortstroking a drive. If I had a 300GB VR, and short-stroked it to 40GB or so, it would still give plenty of space for storing the minecraft files we are using, and would only have to cross about 1/8th of the platter to get to any file. I'd guess a 60% improvement over the normal access times.
Anyone have any thoughts/suggestions? We definitely need to get both servers on separate hard drives, which I will likely do tonight (put the PvP on the SSD), but that's not going to fix the problem, only semi-bandaid it for the time being. Whatever solution I end up going with will have to be doubled to accomodate having two servers.
There's no right answer to this problem at the moment. Here's the various solutions, and why they don't work:Here's another interesting tidbit...
During an automated save, the hard drive still only utilizes 1-2MB/s (3MB/s peak). Since everything else pretty much stops while this is happening, I have to assume that all the processes are waiting on the hard drive to complete the save. I also have to assume that the hard drive is the bottleneck (since CPU/RAM/Network are not fully utilized during this time), and that the hard drive is saving as quickly as it possibly can (why would it not be?).
Based on this information, I believe the hard drive is (and has been) the bottleneck on the server, even with the SSD. Benchmarks show much higher read/write capabilities, but real world, I have never seen the VR drive exceed 3MB/s while running the minecraft servers, and have never seen the SSD exceed 5-6 MB/s while running the minecraft servers. Come to think of it, during world saves the SSD didn't exceed that number either.
Essentially, both hard drives have been running at max capacity, and still cannot keep up with the demands from Minecraft.
Notch said he is working on storing the world in a new format, and that he is liking how it works so far. I can only hope this new format isn't still stored in 4KB files...
In the meantime, any suggestions for improvement? Like I said, I do not believe that RAID will help, as that only really helps sustained r/w speeds, not random access times, which is what I believe we are up against here.
Since there's still some life left in the SSD, I was thinking about moving the PvP server back to it, since it is the server that lags the most. I think even with the lack of TRIM on the SSD, it'll still be faster than trying to run both server off a single slower drive.
- MLC SSD - it degrades too quickly. A 60GB drive lasts about 3 months with the current level of usage. Double the size of the drive, and you'll double the lifespan, but also double or more the cost. Also, there is no TRIM support on the Server 2008 I am running.
- SLC SSD - Too expensive. At $375 for 32GB, it could certainly run the minecraft servers, but without TRIM support, it would slow down just as the current SSD has. Reports say that even if I did have an OS capable of sending TRIM commands, the Intel SLC SSD only supports a few Intel chipsets for sending such commands. Other SLC SSDs are largely unproven and even more expensive.
- Velociraptor HDD - Too slow. Random access time is roughly 15x that of an SSD.
- Any 7200 RPM HDD - Even slower random access times. Not going to help us.
- RAID of classic HDD - Unreliable, and wouldn't help random access times, would only help sustained r/w speeds, which is the opposite of what we need with 4KB files.
- Shortstroking VR HDD - Might work. It's certainly been proven that random access times can be significantly improved by shortstroking a drive. If I had a 300GB VR, and short-stroked it to 40GB or so, it would still give plenty of space for storing the minecraft files we are using, and would only have to cross about 1/8th of the platter to get to any file. I'd guess a 60% improvement over the normal access times.
Anyone have any thoughts/suggestions? We definitely need to get both servers on separate hard drives, which I will likely do tonight (put the PvP on the SSD), but that's not going to fix the problem, only semi-bandaid it for the time being. Whatever solution I end up going with will have to be doubled to accomodate having two servers.