Event Logs: Some Tests

1

Hey there,

I ran some garage tests and patches my end and this was some of the data I concluded…

I am building my own cmd mimification algorithm that can mimify everything.
But some of my finds for example are:

The word: “Position” for example occurs 3 Million Times.
Merely Changing the occurrence of “Position” to “P” reduces my filesize by 50 MB.
And that was a singular change.

What I am going to do is write a json log mimifier that replaces recurring words with shorter ones that I note in shorthand.
But also something that strips out text that I dont need for the visualiser
Like: “RegenerationRate”:null that has no use but takes 21MB and occurs 900 000 times.

Im not going to go too crazy,
Im basically just using:

Find and Replace Text, Hooked to a bat file like below:

fart.exe -c *.json ““Position”” ““P”” &
fart.exe -c --remove *.json ““RegenerationRate”:null,” &
cmd /k

And then I can run my match, copy it in my replay folder, Run my own “Mimifier” and then just make sure that my visualiser runs the mimified code.

The bad side of this is:
Its not very global friendly.

I can update my visualiser to run a mimified version of the logs.
And I can post my exact mimification settings here.

But I cannot upload the mimification program to github for example.

If anyone would be interested in my visualiser tweaked for a mimified log,
As well as the exact bat metrics if you have the following file:

Feel free to let me know in this thread.

This might not do anything for the actual competition itself.
Or restrictions where the actual game-engine cannot handle logs.

But it will at least let me run my visualiser and build while the logs are relooked at.

For now I might not even remove anything,
Just Shorthand all “Object keys” from things like: “Position” → P and “RegenerationRate → RR”

It suits my AI programming style. (I really just do this cause its hyper fun)
But if I shorthand the Object Keys I can keep the visualiser running without stripping content.

This is my own thoughts, Experiences and what i am doing from a “lets reduce the size” perspective.
As I mentioned, Position occurs 3 million times in my match. That means that theres 21 million characters that exist merely for readability in this single key.

Im building complete key replacing script now.
Replacing all lengthily keys (For now)

If my estimations are correct, I should be able to half the size of my logs.
This will fix my visualiser.

Entelect might want to look at a similar approach.
As mentioned in another thread,

Im not sure what happened with the condensed log settings from the first event.
If anyone has acces to them still let me know.
I can also try and adapt the Seuntjie port of my visualiser for those logs for Event 3.

Till then.
Running a text mimifier to fix the logs so they are small enough to simulate and load into memory is the only way I can go.

**Also need to impliment fast so i can work on my actual entry.

2

Hey Willie, I actually built something similar quite a while back since the visualizer stopped working for me in phase 2 already when I played 4v4 logs (anything over 1GB just doesn’t work). Maybe you can use this (remember to install the tqdm package before using the script):

It works by recursively going through the log and removing any paths that are listed in the delete set. It should be fairly easy to add replacement logic for e.g. “position” → “p”. As it currently stands it reduced logs from around 1.1G to ~800Mb IIRC.

To use it run ./cleaner.py path/to/logfile.json - it will then create a faile named path/to/logfile_clean.json

1 Like
3

Just finished my own as well:

Results:
Initial File Size: 1040 MB
End File Size: 709 MB

Effect of each replacement:

fart.exe -c *.json ““Position”” ““P””
Replaced 3010794 occurence(s) in 1 file(s).

fart.exe -c *.json ““MaxUnits”” ““MU””
Replaced 2584481 occurence(s) in 1 file(s).

fart.exe -c *.json ““CurrentUnits”” ““CU””
Replaced 2583212 occurence(s) in 1 file(s).

fart.exe -c *.json ““MaxResourceAmount”” ““MR””
Replaced 1355039 occurence(s) in 1 file(s).

fart.exe -c *.json ““CurrentRegenTick”” ““CT””
Replaced 1355142 occurence(s) in 1 file(s).

fart.exe -c *.json ““GameObjectType”” ““GT””
Replaced 3008712 occurence(s) in 1 file(s).

fart.exe -c *.json ““TerritorySquare”” ““TS””
Replaced 386103 occurence(s) in 1 file(s).

fart.exe -c *.json ““ScoreMultiplier”” ““S””
Replaced 386114 occurence(s) in 1 file(s).

fart.exe -c *.json ““Amount”” ““A””
Replaced 1809205 occurence(s) in 1 file(s).

fart.exe -c *.json ““Reward”” ““R””
Replaced 1356861 occurence(s) in 1 file(s).

fart.exe -c *.json ““WorkTime”” ““W””
Replaced 1356472 occurence(s) in 1 file(s).

fart.exe -c --remove *.json ““RegenerationRate”:null,”
Replaced 902762 occurence(s) in 1 file(s).

fart.exe -c --remove *.json ““TerritoryBase”:“00000000-0000-0000-0000-000000000000”,”
Replaced 2570028 occurence(s) in 1 file(s).

fart.exe -c *.json ““RegenerationRate”” ““RR””
Replaced 454349 occurence(s) in 1 file(s).

fart.exe -c *.json ““PositionsInTerritory”” ““PT””
Replaced 59043 occurence(s) in 1 file(s).

fart.exe -c *.json ““TerritoryNodes”” ““TN””
Replaced 39943 occurence(s) in 1 file(s).

fart.exe -c *.json ““Nodes”” ““N””
Replaced 52461 occurence(s) in 1 file(s).

This also runs in under a minute, so speed is pretty cool.
I will tweak it if the logs push beyond the 1GB mark again, But for now i have a lot of space to dabble

4

One advantage of this is i just copy my log into the file and run and it will process ALL Json files in the directory.

Weakness is I have to run it 3 times to make sure i find everything, I can probably try and search from the back. But for now its fine. Just want to test my Bot under heavy territorial pressures

5

This solution is a LOT cleaner than mine, as it does not change the actual Variables.

My Round 2 bot was terrible, I never even scratched the surface of building potential

6

I had to change 5 lines to get the new logs to work.
IF I run into size issues, I will consult your solution.
You can mimify as well if needed.

The replay does not rely on most of the functionality.
I think Nodes, Position and Current units and Amount was all I relied on from ALL my replacements.
Anyways, Was a great goosechase,

Now I can build and test under heavier presure, Was doing matches where 3 was builders and my refining bot was stuck between builders. See how far I can optimize my build strat.
But I need the larger logs for 4v4 for sure

7

This method dropped my 4v4 file from 1.88gb to 1.77gb so doesn’t really make a difference

8

I just deleted every 2nd entry, for now so should still have sufficient data to see choices in the visualizer

1 Like
9

Thank you very much for this.

After upgrading to Version 3.4 my logs exploded (Also noticed a large buff in my pending actions).
Possibly caused by the resource sharing

One strange question,

When I run it it outputs:
cleaner.py GameStateLog_2022-08-16_07-17-36.json
cleaning GameStateLog_2022-08-16_07-17-36.json
100%|█████████████████████████████████████████████████████████████████████████████| 2501/2501 [00:22<00:00, 111.20it/s]

But no file is written. Im debugging this now and changing the way the data is saved at the end. Does not seem to be file permissions either. Going to try and hardcode the end path. See if its maybe just saving thats an issue.

10

Passed that, But experienced something similar to @Kortgat

In its current form it reduces my log from 2030 MB to 1930 MB
But I am using this to process and run some more tests.

Perhaps I can expand it AND parse my own mimifier.
Im looking at my output logs now to see where my own overhead is.

Then Im back in business.
Perhaps doing what @Kortgat described where I remove every second round and just draw even rounds could be best as it will immediately remove half of the overhead.

Im modifying your script to do that for me as well.
As everything is already a bat file.
So theres that.
Thank you for sharing though.
Would be hard without the resource

11

Thanks @Kortgat for the suggestion, I updated the script (check out the original link) to add a step parameter, if specified as ./cleaner.py <log file> <step> it will only write out every step'th row. I was able to reduce a 2.1GB log file to ~700MB using a step of 2

1 Like
12

Pretty great,

I had the same results, Now I can continue for a while. And have enough room for when my actions increase the size.

Thank you, any timesaver can be great.