Importance of cleaning log files

[Denacke]

Hey fellow server owners!

I just wanted to give you a heads up about an issue i had been having. Some of my soldatserver processes started to go 100% cpu for no apparent reason from time to time, creating some lags and making me restart the server often. It usually showed up a few days after restarting the server.

I did a full server reboot at times too to see if that would fix it, I also minimised the amount of admin connections to the servers to see if that would resolve it. Neither had any effect on the issue.

I then had a look in the log folders of the affected servers. They were considerably large with logs going back to 2011. I figured the soldat process might somehow be looking into those logs to figure out which it has to append to / has to write a new one? Either way, I moved the old logs to a different subfolder and I can now say that it has been running for a full week without any 100% cpu issues anymore. So if you seem to have a similar issue, try cleaning out the log folder!

[Akinaro]

Good point, I did backup of logs every month.
Its really good to read them after 1-2 years xD

PS Did you encounter any lags when few admins tried to login with bad password? I found out that that also create lags and problems with Arsse.

[Szaman]

Lagging servers are also often caused by:
a) broken script that falls into infinite loop
b) attackers who are starting many connections to the admin port (using some "soldat-crasher")

[jrgp]

It would be better if the files were saved in a subfolder hierarchy with the year month and day. Having a huge number of files in the same folder is extremely inefficient and it can make directory traversing extremely disk and cpu intensive.

The cpu is probably it trying to find a file from the current date to increment the last number at the end of the filename, afaik

When it happens next time try strace'ing and lsof'ing the process to see what it's doing.

[Akinaro]

Maybe it would be good if soldat would have automatic sorting system(with subfolders) every 30 days or something...