Optimizing log parsing (ideas?)

[MR X]

Hello. im creating a log parser and it parses a bit to slow for my taughts.
it parses this....

Code: [Select]

08-04-20 16:08:00    Kill Log Started
--- 08-04-20 16:13:04
Stevie
-=Teqskater=-
Spas-12
--- 08-04-20 16:13:06
Sniper
Stevie
HK MP5
--- 08-04-20 16:13:13
Terminator
Boogie Man
Spas-12
--- 08-04-20 16:13:15
Stevie
Terminator
Steyr AUG
--- 08-04-20 16:13:15
-=Teqskater=-
Stevie
Grenade
--- 08-04-20 16:13:21
Sniper
Boogie Man
HK MP5
--- 08-04-20 16:13:23
Terminator
Stevie
Ruger 77
--- 08-04-20 16:13:29
-=Teqskater=-
Boogie Man
M79
--- 08-04-20 16:13:31
Terminator
-=Teqskater=-
Ruger 77
--- 08-04-20 16:13:36
Stevie
Terminator
HK MP5
--- 08-04-20 16:13:41
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:13:45
-=Teqskater=-
Terminator
FN Minimi
--- 08-04-20 16:13:48
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:13:52
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:14:03
Sniper
Stevie
Selfkill
--- 08-04-20 16:14:06
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:14:10
Sniper
Stevie
Selfkill
--- 08-04-20 16:14:14
Terminator
-=Teqskater=-
Barrett M82A1
--- 08-04-20 16:14:24
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:14:31
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:14:33
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:14:37
Stevie
-=Teqskater=-
Spas-12
--- 08-04-20 16:14:41
Sniper
Stevie
Selfkill
--- 08-04-20 16:14:46
-=Teqskater=-
Terminator
FN Minimi
--- 08-04-20 16:14:54
Stevie
Boogie Man
HK MP5
--- 08-04-20 16:14:59
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:15:03
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:15:09
Sniper
Stevie
Selfkill
--- 08-04-20 16:15:19
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:15:22
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:15:35
Stevie
Sniper
Steyr AUG
--- 08-04-20 16:15:40
Terminator
Boogie Man
HK MP5
--- 08-04-20 16:15:48
Sniper
Terminator
HK MP5
--- 08-04-20 16:15:49
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:16:11
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:16:16
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:16:22
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:16:24
-=Teqskater=-
Terminator
FN Minimi
--- 08-04-20 16:16:25
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:16:35
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:16:40
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:16:53
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:16:54
Sniper
Stevie
Desert Eagles
--- 08-04-20 16:16:56
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:16:57
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:17:02
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:17:03
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:17:07
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:17:13
Terminator
Stevie
Desert Eagles
--- 08-04-20 16:17:15
-=Teqskater=-
Terminator
FN Minimi
--- 08-04-20 16:17:18
-=Teqskater=-
Sniper
Hands
--- 08-04-20 16:17:33
Stevie
Terminator
Ak-74
--- 08-04-20 16:17:34
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:17:44
Terminator
Boogie Man
FN Minimi
--- 08-04-20 16:17:48
Terminator
Stevie
FN Minimi
--- 08-04-20 16:17:52
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:17:57
Terminator
Stevie
FN Minimi
--- 08-04-20 16:17:58
Terminator
Sniper
FN Minimi
--- 08-04-20 16:18:03
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:18:10
Sniper
Boogie Man
Steyr AUG
--- 08-04-20 16:18:15
-=Teqskater=-
Terminator
FN Minimi
--- 08-04-20 16:18:28
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:18:35
-=Teqskater=-
Sniper
FN Minimi
--- 08-04-20 16:18:42
-=Teqskater=-
Terminator
FN Minimi
--- 08-04-20 16:18:50
Terminator
Sniper
Spas-12
--- 08-04-20 16:18:51
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:18:58
Terminator
Sniper
Spas-12
--- 08-04-20 16:19:00
Stevie
-=Teqskater=-
HK MP5
--- 08-04-20 16:19:02
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:19:10
Terminator
Boogie Man
Spas-12
--- 08-04-20 16:19:16
Sniper
Terminator
HK MP5
--- 08-04-20 16:19:18
Stevie
Sniper
HK MP5
--- 08-04-20 16:19:26
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:19:26
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:19:29
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:19:37
Terminator
Stevie
Ak-74
--- 08-04-20 16:19:40
-=Teqskater=-
Sniper
FN Minimi
--- 08-04-20 16:19:47
Stevie
Terminator
FN Minimi
--- 08-04-20 16:19:50
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:20:11
Stevie
Boogie Man
Steyr AUG
--- 08-04-20 16:20:14
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:20:15
Terminator
Sniper
Ak-74
--- 08-04-20 16:20:22
Stevie
Terminator
Steyr AUG
--- 08-04-20 16:20:25
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:20:31
Stevie
Boogie Man
Steyr AUG
--- 08-04-20 16:20:35
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:20:37
Sniper
-=Teqskater=-
Selfkill
--- 08-04-20 16:20:41
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:20:44
Terminator
Stevie
M79
--- 08-04-20 16:20:44
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:20:58
-=Teqskater=-
Stevie
FN Minimi
--- 08-04-20 16:21:04
Sniper
Boogie Man
Spas-12
--- 08-04-20 16:21:04
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:21:14
Sniper
Stevie
HK MP5
--- 08-04-20 16:21:17
-=Teqskater=-
Sniper
Grenade
--- 08-04-20 16:21:25
Sniper
Boogie Man
Selfkill
--- 08-04-20 16:21:30
-=Teqskater=-
Stevie
Ak-74
--- 08-04-20 16:21:33
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:21:42
-=Teqskater=-
Boogie Man
Ak-74
--- 08-04-20 16:21:47
Sniper
-=Teqskater=-
Selfkill
--- 08-04-20 16:21:57
-=Teqskater=-
Boogie Man
Grenade
--- 08-04-20 16:22:04
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:22:07
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:22:10
-=Teqskater=-
Boogie Man
Ak-74
--- 08-04-20 16:22:19
Sniper
Stevie
Selfkill
--- 08-04-20 16:22:20
Terminator
-=Teqskater=-
Ruger 77
--- 08-04-20 16:22:28
Terminator
-=Teqskater=-
Ruger 77
--- 08-04-20 16:22:35
Sniper
Terminator
Selfkill
--- 08-04-20 16:22:39
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:22:43
Sniper
Boogie Man
Selfkill
--- 08-04-20 16:22:46
-=Teqskater=-
Sniper
Ruger 77
--- 08-04-20 16:22:48
Terminator
Stevie
Selfkill
--- 08-04-20 16:22:56
Stevie
-=Teqskater=-
Hands
--- 08-04-20 16:22:57
Stevie
Terminator
Steyr AUG
--- 08-04-20 16:22:57
Sniper
Boogie Man
Ak-74
--- 08-04-20 16:23:05
Sniper
-=Teqskater=-
Ak-74
--- 08-04-20 16:23:12
Boogie Man
Sniper
Chainsaw
--- 08-04-20 16:23:18
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:23:18
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:23:19
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:23:28
-=Teqskater=-
Sniper
Spas-12
--- 08-04-20 16:23:35
-=Teqskater=-
Terminator
Spas-12
--- 08-04-20 16:23:37
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:23:42
-=Teqskater=-
Boogie Man
Spas-12
--- 08-04-20 16:23:44
Sniper
-=Teqskater=-
Selfkill
--- 08-04-20 16:23:54
Sniper
Boogie Man
Selfkill
--- 08-04-20 16:23:55
-=Teqskater=-
Sniper
Chainsaw
--- 08-04-20 16:23:55
Terminator
Stevie
M79
--- 08-04-20 16:23:58
Terminator
-=Teqskater=-
M79
--- 08-04-20 16:24:05
-=Teqskater=-
Stevie
Chainsaw
--- 08-04-20 16:24:09
Terminator
Boogie Man
M79
--- 08-04-20 16:24:13
-=Teqskater=-
Terminator
Chainsaw
--- 08-04-20 16:24:15
-=Teqskater=-
Sniper
Chainsaw
--- 08-04-20 16:24:16
-=Teqskater=-
Stevie
Chainsaw
--- 08-04-20 16:24:22
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:24:24
Sniper
Stevie
Desert Eagles
--- 08-04-20 16:24:27
Terminator
Boogie Man
Selfkill
--- 08-04-20 16:24:33
Terminator
Stevie
Selfkill
--- 08-04-20 16:24:34
Stevie
Terminator
Grenade
--- 08-04-20 16:24:42
-=Teqskater=-
Sniper
Chainsaw
--- 08-04-20 16:24:42
Terminator
Boogie Man
Spas-12
--- 08-04-20 16:24:43
-=Teqskater=-
Stevie
Chainsaw
--- 08-04-20 16:24:45
Terminator
-=Teqskater=-
Spas-12
--- 08-04-20 16:24:52
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:24:55
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:24:57
Stevie
Boogie Man
Spas-12
--- 08-04-20 16:25:00
Sniper
Stevie
HK MP5
--- 08-04-20 16:25:11
Sniper
Stevie
HK MP5
--- 08-04-20 16:25:12
Sniper
Boogie Man
HK MP5
--- 08-04-20 16:25:13
Sniper
-=Teqskater=-
HK MP5
--- 08-04-20 16:25:23
Stevie
Terminator
Steyr AUG
--- 08-04-20 16:25:26
Sniper
-=Teqskater=-
HK MP5
--- 08-04-20 16:25:30
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:25:38
-=Teqskater=-
Terminator
Chainsaw
--- 08-04-20 16:25:41
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:25:44
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:25:52
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:25:56
Terminator
-=Teqskater=-
FN Minimi
--- 08-04-20 16:26:11
Sniper
Sniper
Selfkill
--- 08-04-20 16:26:14
Terminator
Stevie
FN Minimi
--- 08-04-20 16:26:15
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:26:23
Boogie Man
Stevie
Chainsaw
--- 08-04-20 16:26:23
-=Teqskater=-
Sniper
Chainsaw
--- 08-04-20 16:26:29
Terminator
Boogie Man
FN Minimi
--- 08-04-20 16:26:38
-=Teqskater=-
Stevie
Chainsaw
--- 08-04-20 16:26:44
-=Teqskater=-
Stevie
Chainsaw
--- 08-04-20 16:26:56
-=Teqskater=-
Sniper
Chainsaw
--- 08-04-20 16:26:58
-=Teqskater=-
Boogie Man
Chainsaw
--- 08-04-20 16:27:00
Stevie
-=Teqskater=-
Steyr AUG
--- 08-04-20 16:27:08
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:27:15
-=Teqskater=-
Stevie
Chainsaw
--- 08-04-20 16:27:19
Sniper
-=Teqskater=-
Selfkill
--- 08-04-20 16:27:26
Stevie
Terminator
Steyr AUG
--- 08-04-20 16:27:28
-=Teqskater=-
Sniper
Chainsaw
--- 08-04-20 16:27:33
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:27:36
Sniper
-=Teqskater=-
Selfkill
--- 08-04-20 16:27:43
Terminator
Stevie
Selfkill
--- 08-04-20 16:27:44
Boogie Man
Terminator
Chainsaw
--- 08-04-20 16:27:51
-=Teqskater=-
Stevie
Chainsaw
--- 08-04-20 16:27:53
Boogie Man
-=Teqskater=-
Chainsaw
--- 08-04-20 16:28:00
Sniper
Boogie Man
Selfkill
--- 08-04-20 16:28:04
Terminator
-=Teqskater=-
FN Minimi
--- 08-04-20 16:28:13
-=Teqskater=-
Sniper
Chainsaw
--- 08-04-20 16:28:14
-=Teqskater=-
Terminator
Chainsaw
--- 08-04-20 16:28:18
-=Teqskater=-
Boogie Man
FN Minimi
--- 08-04-20 16:28:21
-=Teqskater=-
Stevie
Chainsaw
...in about 17-25 seconds to a SQLite database(only counts kills deaths and selfkills of player and saves the last time of their last action. and before that it fixes the log if it has that 'missing weapon bug').  zitrostats does it faster but forgets the last kill/death

This is pretty slow isnt it? I code the log parser in Autoplay media studio with programming language LUA.and i've coded it "perfectly" but i don't know if those 17-25 seconds is acceptable enough? Or maybe it's just my pc. When i can make it faster i will, but currently i just don't see how. tell me what you think.

[FliesLikeABrick]

Yes, that is extremely slow, considering that most people use log parses on tens/hundreds of thousands of lines of code at once.  The speed issue is likely a combination of you using LUA and SQLite, though I'm not terribly familiar with each and can't say for sure.

Can you show us the pieces of your code that do the actual parsing so that we can see how you're doing it and possibly make suggestions?

[MR X]

hope this is clear enough: 

Parse code:

Code: [Select]

Code removedI appreciate your help. and you may make suggestions.

and please don't use this code without my permission because this is 3 days of work.

FliesLikeABrick you got my permission to use this code

#EDIT:
I attached the compiled project and a fake soldat folder in a rar file so that you can see it it working. just point Teqlog (the parse program) to the included fake soldat folder (to ensure that your real logs are safe),
and click on update database after you have created a database.

#EDIT2:
I also discovered a bug in the logs:
08-04-20 18:34:33    Kill Log Started
--- 08-04-19 18:35:30

look at the dates:
killog started at 08-04-20
--- 08-04-19 18:35:30