Python + TF tournament submission issue

My submission is failing in the player portal and I think I may have found the issue but I can’t be sure. TensorFlow takes ~18 seconds to import the first time it is used by a script. Only after clearing my environment and starting fresh (as the build server would have) did this become an issue. I’m still working on trying to fix this on my side, is there anything that can be done for my tournament submission? Increasing the bot warm up time to 30 seconds in the engine should resolve the issue. I feel like the spirit of the tournament isn’t really ‘how fast can your bot can start up’ so it really sucks if I lose out because of this.

you can do all of your training in python or whatever, and then write an inference bot in Go using the C tensorflow bindings. This launches ~100x faster than in python and isn’t very difficult.
Alternatively use pytorch, it loads way faster on a clean install than tensorflow (at least from what I’ve seen). but if you’re insistent on writing the inference bot in python, there isn’t much you can do in terms of importing a library faster.

@marcin_k thanks for this, the only issue is when the script runs for the very first time on a machine. Subsequent runs (and imports of tensorflow) happen within two seconds. From then on there is absolutely no issue, seems weird that I have to make such a fundamental change just because of this anomaly. In the real world I would have just added an ‘import tensorflow’ at the end of the setup phase and there would be no problem. I don’t know if anyone at entelect can give me closure on this?

@floreggian_entelect, @Renette, @KyleMc

Hi Rossalot,

I will have a look at your submissions. What email address did you sign up with?

You can DM me this information.

@floreggian_entelect I’ve dm’ed you. I’ve submitted a bot now that has fixed the issue that I had with tensorflow taking long to import for the first time. It however still says ‘Failed’ under the match status. I suspect this is because I changed my bot to use python’s setuputils and a ‘setup.py’ file (originally it was a requirements.txt). If this is the case it just needs a `python setup.py install’ to be installed correctly. If that’s not the cause of the issue, it’s really hard to tell from the match logs, I’m stumped. Could you perhaps take a look and let me know what’s going on there?

I will have a look :+1:

1 Like

Hi @Rossalot can you please reupload your bot again.

@floreggian_entelect I’ve uploaded again. It’s been stuck ‘In Progress’ on the Match Status step for 30min now.

Hi, yes it will only be scheduled after the tournament is complete

@floreggian_entelect Failed again, not sure why. Anything I can do to help make this easier?

Downloading game engine
Downloading 2020.1.8.zip
Extracting zip: engine-d2cf02bc-1e73-4b0d-a390-4fa35baeff35.zip
Retrieving bot path from directory
Downloading bots
Downloading entries/main/694bc332-6e41-48aa-90f8-fd059f53498a/4ab4b134-3622-424c-ad80-5059bee949e3/binaries/4ab4b134-3622-424c-ad80-5059bee949e3.zip
Downloading reference-bots/2020-1-2/reference-bot.zip
Extracting zip: player-dfc5866e-a5d3-4fa6-9d0a-3ab190a3caa2.zip
Extracting zip: player-7f9526e8-4878-49a3-9007-f447a970f7d3.zip
Config for player A : tournament-tmp/extracted/player-dfc5866e-a5d3-4fa6-9d0a-3ab190a3caa2/4ab4b134-3622-424c-ad80-5059bee949e3
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Failed call: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Retrying request: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
Exception during match execution
java.io.IOException: Request failed: Request{method=POST, url=http://localhost:55555/instantiate_bot, tags={class retrofit2.Invocation=za.co.entelect.challenge.network.BotServices.instantiateBot() [okhttp3.MultipartBody$Part@be35cd9]}}
at za.co.entelect.challenge.network.RetryableCall.execute(RetryableCall.java:48) ~[game-runner.jar:?]
at za.co.entelect.challenge.player.TournamentPlayer.instantiateBot(TournamentPlayer.java:60) ~[game-runner.jar:?]
at za.co.entelect.challenge.player.TournamentPlayer.(TournamentPlayer.java:47) ~[game-runner.jar:?]
at za.co.entelect.challenge.player.bootstrapper.PlayerBootstrapper.parsePlayer(PlayerBootstrapper.java:75) ~[game-runner.jar:?]
at za.co.entelect.challenge.player.bootstrapper.PlayerBootstrapper.loadPlayers(PlayerBootstrapper.java:49) ~[game-runner.jar:?]
at za.co.entelect.challenge.engine.bootstrapper.GameBootstrapper.run(GameBootstrapper.java:67) [game-runner.jar:?]
at za.co.entelect.challenge.engine.bootstrapper.GameBootstrapper.main(GameBootstrapper.java:47) [game-runner.jar:?]
Notifying of match failure
Creating zip: 495fec1c-fb3d-421e-830f-1759defd7528 for directory: ./match-logs

It seems like your bot struggles to startup in time for the game runner (that is why the instantiateBot call is failing)

Do me a favour, re submit again. Your reference match is unfortunately lost in the tournament matches.

And it will be easier for me to debug it now

@floreggian_entelect I’ve submitted again. Let me know what else I can do to help? I’ve tested this build on my side and it seems to be working without issue.

Hi, it seems that you have removed the requirements.txt file that comes default with the bot.

Please do not remove this file.

As the runtime environment needs it.

@floreggian_entelect I can put it back in, it was just to ensure that the build didn’t use it. requirements.txt will not work for my case and I need the build to use the setup.py. If it just needs it because of some sanity check I can add it back though…let me try resubmitting it

@floreggian_entelect I’ve submitted again with the ‘requirements.txt’ in the folder, it failed again on the ‘Match Status’. I just want to reiterate, I had a problem with Tensorflow taking a long time to import on the very first run. The only way I could get around it was to implement a sort of post install hook to pre-emptively import Tensorflow once before it is needed by the game engine (this 100% works on my machine everytime). I just need whatever process your build engine is using to NOT ‘pip install -r requirements.txt’ but instead ‘python setup.py install’ . Once again, this is so difficult to debug without logs. If your build pipeline is doing this and there are no error logs then I’m stumped :see_no_evil:

I mentioned it earlier in this thread in case you missed it:

@floreggian_entelect I’ve dm’ed you. I’ve submitted a bot now that has fixed the issue that I had with tensorflow taking long to import for the first time. It however still says ‘Failed’ under the match status. I suspect this is because I changed my bot to use python’s setuputils and a ‘setup.py’ file (originally it was a requirements.txt). If this is the case it just needs a `python setup.py install’ to be installed correctly. If that’s not the cause of the issue, it’s really hard to tell from the match logs, I’m stumped. Could you perhaps take a look and let me know what’s going on there?

Thanks for the help so far.

1 Like

@floreggian_entelect if you could get back to me on this, I want to keep working this weekend and I can’t do that if I’m unsure whether I need to abandon what I’ve done and try other frameworks :see_no_evil: thanks for the help so far.

1 Like

Hi everyone, I apologize for the lack in communication.

We all have day jobs right XD.

But I would like to say that the team is looking at this specific problem and we will try and come up with the best solution.