Solving Server Lag

[SgtSpike]

I believe I had a bit of a breakthrough with regards to solving the server lag problem. I just posted this in the mod forums in reply to a discussion we've been having, but figured it wouldn't hurt to get more input from other people on the matter.

Here's another interesting tidbit...

During an automated save, the hard drive still only utilizes 1-2MB/s (3MB/s peak). Since everything else pretty much stops while this is happening, I have to assume that all the processes are waiting on the hard drive to complete the save. I also have to assume that the hard drive is the bottleneck (since CPU/RAM/Network are not fully utilized during this time), and that the hard drive is saving as quickly as it possibly can (why would it not be?).

Based on this information, I believe the hard drive is (and has been) the bottleneck on the server, even with the SSD. Benchmarks show much higher read/write capabilities, but real world, I have never seen the VR drive exceed 3MB/s while running the minecraft servers, and have never seen the SSD exceed 5-6 MB/s while running the minecraft servers. Come to think of it, during world saves the SSD didn't exceed that number either.

Essentially, both hard drives have been running at max capacity, and still cannot keep up with the demands from Minecraft.

Notch said he is working on storing the world in a new format, and that he is liking how it works so far. I can only hope this new format isn't still stored in 4KB files...

In the meantime, any suggestions for improvement? Like I said, I do not believe that RAID will help, as that only really helps sustained r/w speeds, not random access times, which is what I believe we are up against here.

Since there's still some life left in the SSD, I was thinking about moving the PvP server back to it, since it is the server that lags the most. I think even with the lack of TRIM on the SSD, it'll still be faster than trying to run both server off a single slower drive.

There's no right answer to this problem at the moment. Here's the various solutions, and why they don't work:
- MLC SSD - it degrades too quickly. A 60GB drive lasts about 3 months with the current level of usage. Double the size of the drive, and you'll double the lifespan, but also double or more the cost. Also, there is no TRIM support on the Server 2008 I am running.
- SLC SSD - Too expensive. At $375 for 32GB, it could certainly run the minecraft servers, but without TRIM support, it would slow down just as the current SSD has. Reports say that even if I did have an OS capable of sending TRIM commands, the Intel SLC SSD only supports a few Intel chipsets for sending such commands. Other SLC SSDs are largely unproven and even more expensive.
- Velociraptor HDD - Too slow. Random access time is roughly 15x that of an SSD.
- Any 7200 RPM HDD - Even slower random access times. Not going to help us.
- RAID of classic HDD - Unreliable, and wouldn't help random access times, would only help sustained r/w speeds, which is the opposite of what we need with 4KB files.
- Shortstroking VR HDD - Might work. It's certainly been proven that random access times can be significantly improved by shortstroking a drive. If I had a 300GB VR, and short-stroked it to 40GB or so, it would still give plenty of space for storing the minecraft files we are using, and would only have to cross about 1/8th of the platter to get to any file. I'd guess a 60% improvement over the normal access times.


Anyone have any thoughts/suggestions? We definitely need to get both servers on separate hard drives, which I will likely do tonight (put the PvP on the SSD), but that's not going to fix the problem, only semi-bandaid it for the time being. Whatever solution I end up going with will have to be doubled to accomodate having two servers.

 

[FrothingLuck]

Well from what you are describing it seems that if we can't upgrade the server's hardware we need to downgrade the software. I had mentioned that perhaps we should lower the player limit and remove certain features within the mods that we are running. This is only a theory I don't know if that would have any significant difference.

 

[SgtSpike]

I'd certainly like to leave that option as a last resort, but we may have to do that Frothing. Either that, or start a new map so we're not dealing with huge map files... even though I don't really see how that would help (it should only be saving/reading the portions of the map that a player is actually on).

If a single server cannot be handled by a single hard drive, then we'll probably have to downgrade software, as you are saying. Whether that means a lower playerlimit or a limited map will be up for discussion.

 

[oliverw92]

I don't see how we could 'downgrade software' beyond limiting the map size? Give it two weeks and we would be back to large maps. Also disabling plugins will do nothing since plugins don't really utilize resources that are actually the issue.

 

[the_actual_God]

Solution:

SgtSpike, your ShoutBox text "Aiming for a ramdrive next though, I think that's the only way to get the performance I am looking for" is the answer:

You need to employ a ramdrive solution that regularly saves its content to disk as one image instead of 1000 files! Maybe TrueCrypt or something like that can help to turn the many files into one. I am convinced that this demand is not a new revolutionary idea in the computer world - there must be appropriate solutions for a problem like this in existence.

Also, I can't resist to say this: There are servers out there which do not lag and carry many more players. How do they do this? The knowledge exists, and I don't think that the answer of all those people would be "throw hardware at it". Those admins should be asked. Or it should be considered that building a server versus using an established specialized service was not the best use of the money.

 

[FrothingLuck]

Solution:
Or it should be considered that building a server versus using an established specialized service was not the best use of the money.

regardless of whether or not it was "best use of the money" most established servers cost monthly fees which means it would be a finite time until spike would run out of donation money. Also, whats been done is done and complaining about it certainly won't help the situation.

 

[SgtSpike]

It'd be $105/month to rent servers to run the number of players I would currently like to run (20+20). I have considered the option though. I don't generally like renting servers, because there always seems to be options I want to use that are turned off or not allowed.

taG, I couldn't agree more than a ramdrive is the best solution at this point. The more I look in to it, the more I like the idea. Just need to find good software for virtualizing a ramdisk. I have already ordered the ram, overnighted for tomorrow (because it was a cheap option, and I didn't want to wait until the end of next week for it).

I am more than welcome to hear ideas from other server owners. I have talked to a couple in the past, but they did not know the details of their server's configuration. I wouldn't mind if someone else wanted to try and tackle some research on that front as well. *hint hint*

 

[oliverw92]

If we rented a server we wouldn't even have enough donations to run for a year (probably). Ramdisk is the way forward for us

 

[Valkorian]

we wouldn't even have enough donations to run for a year (probably)

As soon as I get a job (which most likely will be this year) there'll be steady donations coming your way (and as soon as I find my 25 euro, some of that aswell). Wonder if we could get others to do the same?

 

[SgtSpike]

Valk, if we did go that direction, I'd probably ask for monthly donations for VIP to keep it up. I bet we could do it, but I'd first like to see if this ramdisk solution will solve the lag problem until a better map format is developed.

 

[FrothingLuck]

That would be ridiculous Sgt. Do you realize how much work it would be to check payments on a 100+ donations a month? I really hope that doesn't happen.

 

[SgtSpike]

Which is why I hope we don't have to go there, Frothing.

I'd recruit someone else to manage the tracking though. And I'd probably request that people donate in chunks of 6 or 12 months... heck if we had 100+ people donating, it'd only be $1/month for each of them. Or I could program an interface for paypal, to where paypal would automatically populate a database of donaters and how much was donated when.

 

[Domino]

I'm not too educated in this field, but is there a way to eliminate as much compression on the file as possible? Could help in shortening saving times....?

 

[SgtSpike]

I'm pretty sure the files are uncompressed.

The reason for the lengthy saves is that it is saving 10's of thousands of individual files at the same time. The new update should fix that though, as it'll bring that 10's of thousands down to, oh, 10's of 10's.