JRG Stats v1.6b (fast php killog parser / stats program)

[jrgp]

When I tested it with a lot of log files I got this error:

Code: [Select]

Parsing logs to get kills......
<br />
<b>Fatal error</b>:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 549 bytes) in <b>/var/www/web0/html/files/update.php</b> on line <b>84</b><br />

Yeah, happened to me when I used a lot of logfiles too.

I've got several ideas of how to get around this.

Expect a solution in the next release.

[bja888]

There where so many inner joins in the code I had to maintain for my last job. It drove me crazy. The whole app was so poorly put together and they would join 3 tables for 3 sql query's on one page.

f***ing hated it. Not using joins is cleaner code in my opinion.

I'm sorry about your poor experience. Sounds like your business was using joins incorrectly.

If used correctly, joins can increase performance. They also make it very easy to spot table relationships.

Nothing that company was doing was right.

I'll take your word for it because I don't have actual data to back it up.

[jrgp]

New release!

v1.4b (9/19/2007) -
 - Much better update script
 - Optimised SQL queries
 - Signature Image Support
 - Random bug fixes
 - Various Optimisations
 - Pagination added to the Players List page

When I tested it with a lot of log files I got this error:

Code: [Select]

Parsing logs to get kills......
<br />
<b>Fatal error</b>:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 549 bytes) in <b>/var/www/web0/html/files/update.php</b> on line <b>84</b><br />

This should be fixed.

(I do not know for sure that it is fixed and would really appreciate it if someone were to test the new update script with a lot of logs.)


Edit:

Another new release!

v1.5 (9/23/2007) -
 - Update script extremely optimised
 - Ability to get logs via ftp
 - First non beta release

[urraka]

Would it make sense to check the file modification time to skip old files in the update script?

By the way, I may take your script and modify it to satisfy my needs, hope you don't mind

edit: I've just noticed the table definitions and i have a doubt... why do you use int(100) for kills, deaths and suicides? (I don't really remember what the number '100' does). And also, maybe making it unsigned would be better.

[jrgp]

@BlueDog:

Yes, the next release will check the file modification time so it runs faster by skipping untouched files.

Go ahead, modify it in whatever way you want for your needs, just make sure I am credited for making it somewhere.

Yeah, unsigned might be better. The INT(100) allows for a 100 digit number. I suppose someone killing a hell of a lot might require this, but maybe not.

[ElephantHunter]

No player will ever reach 10^7 kills, let alone 10^100 kills. With that in mind, you might consider UNSIGNED MEDIUMINT(10) to speed up your statistics. This data type will store any number up to 16,777,214. Remember, the number between the parentheses doesn't change how much can stored, just how the number is displayed in the admin. See more information in the official MySQL Numeric Types documentation.

[chrisgbk]

No player will ever reach 10^7 kills, let alone 10^100 kills. With that in mind, you might consider UNSIGNED MEDIUMINT(10) to speed up your statistics. This data type will store any number up to 16,777,214. Remember, the number between the parentheses doesn't change how much can stored, just how the number is displayed in the admin. See more information in the official MySQL Numeric Types documentation.

As a 3 byte integer, from a CPU standpoint, will have problems, first and foremost being allignment stalls, due to a 3 byte number not being supported natively. So code to access it will always run slower. Even if it is in actuality stored as a 4 byte integer to prevent allignment stalls, it will still have the overhead of checking for overflows in software, whereas a 4 byte integer could be checked efficiently in hardware. So a 3 byte integer will always be slower than a 4 byte integer. Unless memory bandwidth holds you back.

These are the same problems as 24 bit color vs 32 bit color, disregarding the alpha channel. 24 bit used to be popular when bandwidth from main memory to the GPU memory was the limiting factor, but as that is now in many cases not an issue, 32 bit color is used exclusively; even in cases when alpha is not used.

Also why programming languages such as C++ pad structures to 32 bits by default.

Note that when 64 bit processors become the norm instead of the exception, most of these limits will also increase to 64 bits.

@Yeah, unsigned might be better. The INT(100) allows for a 100 digit number. I suppose someone killing a hell of a lot might require this, but maybe not.

As EH mentioned, and the documentation says, the (100) only means that it will be padded to 100 characters. With commas, the maximum length of an UNSIGNED INT is 13 characters (4,294,967,295), so you will have at least 87 characters of padding, which WILL slow you down. (Plus problems with temporary tables in complicated joins, as the page also mentions, due to exceeding the default maximum width.)

[urraka]

@BlueDog:

Yes, the next release will check the file modification time so it runs faster by skipping untouched files.

Go ahead, modify it in whatever way you want for your needs, just make sure I am credited for making it somewhere.

Well, I ended up rewriting the whole thing. I added it some stuff that zitrostats had, like points, player opponents and player weapons. Also, your script doesn't handle this kind of bug (a line is missing):

--- 07-10-02 23:53:30
[MAsAR] PerroAZUL
Grenade

I made it ignore things like that.
I also had to change line endings to \r\n (Windows fault?).

I'll post what I did if anyone wants to take a look (the code is not complete).

[ElephantHunter]

As a 3 byte integer, from a CPU standpoint, will have problems, first and foremost being allignment stalls, due to a 3 byte number not being supported natively. So code to access it will always run slower. Even if it is in actuality stored as a 4 byte integer to prevent allignment stalls, it will still have the overhead of checking for overflows in software, whereas a 4 byte integer could be checked efficiently in hardware. So a 3 byte integer will always be slower than a 4 byte integer. Unless memory bandwidth holds you back.

The records for MySQL are stored in files. The smaller the records are the faster these files can be read. In this case, differences in CPU speed are negligible compared with HD access.

[Spasm]

I've got some interesting errors.  http://elitectf.com/zombiestats

I've got gun names switched with player profiles and when I run the update script every so often it's saying that no new entries are found but there's always people in the server (including myself I checked to make sure) registering kills.  No updates though


Maybe because of the layout switch and the removal of top 10 ranks?

[jrgp]

I've got some interesting errors.  http://elitectf.com/zombiestats

I've got gun names switched with player profiles and when I run the update script every so often it's saying that no new entries are found but there's always people in the server (including myself I checked to make sure) registering kills.  No updates though


Maybe because of the layout switch and the removal of top 10 ranks?

No. This has nothing to do with your layout customizations. (which I really like btw )

This is entirely my fault by not supporting Soldat's logging bugs and using a not so great log parser. Anyways, I will work on it a lot this weekend and hopefully have a better release sometime within the next few days.

Thanks for your patience and customizations. btw, do you like the way the program has you modify the layout?

[Spasm]

Yes, it's rather simple.  I basically just added a few more tables and switched things around.  Took about a half hour.  I appreciate you putting some time in to help resolve this.  I am going to leave the page to openly display so at least people from the server get a general idea of what it'll look like.

(PS:  Did you like the dedicated signature image I made for the server? )

[jrgp]

I appreciate you putting some time in to help resolve this. 

Your welcome.

(PS:  Did you like the dedicated signature image I made for the server? )

Yes, very much. One thing though, why do the weapon images not appear anywhere on the site?

[Spasm]

One thing though, why do the weapon images not appear anywhere on the site?

My best guess is that it's searching for your default weapon's path.  I'm trying to find which file has the weapons img directory setting.  It's probably looking for "xxx/img" directory and it's not recognizing "zombiestats/img"