How do we make our bots better?

So I know we’re still only around halfway through this thing but I think it’s evident at this point that my bot is only “pretty good”, looks like I’m going to be sitting in the bottom half of top 10 or somewhere around there… Now my question is, how do we go and improve on our bots if we can’t see where they’re going wrong?

I don’t know how to make my bots strategy any better without seeing what’s beating it and I’m sure almost all of us are in that same position. So any ideas where we go from here?

1 Like

Yeah I agree with you Travis, you can’t improve if you don’t know what is going “wrong”. It would be nice to at least get the console output of some of the matches you lost, each round’s board state would go a long way

1 Like

I would suggest that you ask someone to “release” their bot code to the public.

If they are willing to throw away their strategy, then everyone has the information and a baseline bot to attempt to win. However, it could discourage those that do not win against that bot.

In addition, it could result in lots of effective clones of a really good bot, which does not really make a competition.

Releasing state files for the match is very close to releasing it’s source.

I disagree. consider a chess bot for example, being able to see which moves the bot chooses to make does not give you any information on how the bot gets to that move. It might open some rule based strategies in this game, but I think that’s okay.

I guess the question is what is this competition trying to achieve.

3 Likes

Gotta agree with linjoehan. If becoming a master of anything was as simple as analyzing a limited sample of a masters actions then our world would make no sense, that’s just not how things work.

Opening up the state files would encourage further participation, adaptation and learning anyway, and I don’t see what’s wrong with that other than upsetting the very top guy who gets to sit up top all pretty and do nothing further because no one is allowed to adapt to what’s happening.

1 Like

I also have to agree with @linjoehan on this one. Simply having one set of state file from your opponent during a match does not give away any strategy, unless that strategy is extremely straightforward.
And sure enough, you could probably come up with a bot that would effectively counter that bot using the information, but that does not mean you will have any decent kind of bot, just because it can counter one player’s strategy.
Add on top of that for the next tournament there will be new rules and buildings, and simply being able to view how somebody managed to defeat your current strategy reveals nothing of importance going forward.

1 Like

One way to handle this, that I think would work for both sides, is to have an opt-in system, where everyone who has opted in can see the full logs of their matches against the others who have opted in, but those who have not opted in can’t see anything, or have any of their matches seen.

4 Likes

That has to many holes, I think this is very much an all or nothing kind of thing without limits. Either make all games public or none. Even something like only being able to see your own games could allow sharing of games between friends to get more information about some players, and there will be no way to stop people from cheating in this way.

Submit a dummy bot with a random profile and opt in to see some stuff for example?

I also think it will be cool to watch some top players games against each other

3 Likes

I agree with Malman’s method…

I would very much like to see how Ralf handles his mathes, along with the other top 10 players.

But one match would give me everything I need to copy their strategy…

In the top bots its like that.

  • I see your energy tactic, your offensive tactic and your defensive tactic and I can pretty much
    pick up on your “score” logic…
  • If I can analyze it I can build it.

Regardless, I feel that part of the bot building experience is seeing the strategy. And learning how to build it. To scratch your head and search for those game changing ideas.

If I for example ran 1000 tests and came to the conclusion that defensive buildings are useless (Stupid example). And I ignore them, and I win the event. And everyone else sees that I never built defensive buildings, so they do not either, Sure, I might be better with attack then them, but them copying my homework would not be cool. There is a little bit of human in every bot and I do not like the idea that that part gets shared.

I personally would opt out for logs. As much as I can gain, I never liked the idea of someone seeing my product before its time…

I personally loved the challenge system we had in Bomber man where you had ±5 tokens where you can challenge another player and see the replay from that game. If the other person thinks you will copy his strategy you can just reject the challenge. It gives you a limited platform to improve your bot without giving your source to anybody but gets valuable feedback in your bot logs and replays.

Hi guys,

We are hearing a lot of frustration around this issue and so we are in the planning phases of getting an Opt-In challenge system built so that you guys can challenge anyone else who has opted in.

As I said, we are still planning this and trying to find ways of “detecting” or otherwise mitigating what @linjoehan said, where someone might upload a ‘do-nothing’ bot and facing off against people, gathering info as they go along.

That being said, we will not be releasing the Tournament logs, for various reasons.

We will keep you guys appraised around the progress of this feature and if you guys have any ideas or suggestions, let us know and we will consider them.

5 Likes

Great to hear you are working on something for this!

What about, rather than a challenge system, running an opt-in, ongoing tournament, with the replay files available? So players who wanted to could join it, and it would play a round robin tournament with all the bots that were submitted to it, and they could see the replay files of all their bots’ matches.

By ongoing, I mean that whenever a player submits an updated bot to this tournament, all their games are replayed and the leaderboard recalculated (though the replay files of the games their previous bots played would still have to be available for other players to view). With a limit of only one bot update per player per day, or three days or something, to save your server. Plus whatever non-exploitation ideas you come up with.

I think doing well in this kind of tournament would be a fun challenge by itself, as well as a way for people to get more insight into how to improve their bots.

2 Likes

Also, since we’re doing ideas: I think it would be interesting if all the replay files from the final round robin tournament were made available for all bots. At that point, there are no more tournaments to enter, so giving away strategy doesn’t matter, and I think a lot of people would like to be able to check out the showdowns between the best bots. I know it’s a lot of data, but I did a quick test and compressing a match directory tree with bzip2 gives around a 10x reduction in file size over ZIP! So 2360 matches could fit in around 150MB!

1 Like

That’s to much information.

I think the only thing that might be released should be the command.txt files, or gamestate. Logs also contain outputs players might be using to debug which should not be public. commands are small and would be very small compared to full states, it will be a little work for players to go through it though.

By log files I meant the game state for each round, wasn’t thinking about the bot outputs but I don’t think people should have access to their opponents’ bot outputs.

I suppose the ultimate compression for a deterministic game like this would be to store each game history as a list of the moves in each round, then people can regenerate the game state using the game engine. This could even be a mode of the game engine (play a game and generate all the normal files from two lists of moves). But then players wouldn’t be able to see their bots’ own logs, and maybe it isn’t worth all that fuss to save a few hundred megs of storage (S3 is cheap!)

1 Like

I disaggree here, This could be discouraging if one bot finds a strategy that will win 90% of the time. I would say making all matches available after finals would be better. Otherwise Just now I crack a unbeatable strategy :stuck_out_tongue: No but serious. Some mystery should also be cool so that nobody knows. I mean, if your like me it does not matter anyways, Because I will not check logs anyways, to just keep things a surprize, so honnestly I dont feel this point matters too much. But not after the first 2 events.

Last year I pulled my bot from the clan games to not give away strategy…

I don’t know about this. This might seem negative, but seeing as the issue was raised, I might as well raise my thoughts too.

Last year you didn’t have much of an effect on the opponent directly. The only direct way you could influence your opponent was the placement of your ships, and destroying specific ships to disable abilities. But because you couldn’t really know where they are, it was mostly an efficiency AI.
This year it it is a much more real-time influence. You have to directly act and react to what your opponent does.

My bot could fairly efficiently defeat the reference bot. I could eventually run any number of matches and have a 100% winrate.

Now, I didn’t place very high, which, although disappointing, isn’t a big problem. But there is no way for me to know why.
I can’t analyse my bot’s behavior vs its purpose, because I can’t see the direct results of its choices, only the final result of its performance, which effectively helps me nothing in determining its faults or weaknesses.

What’s the point of building an AI bot if you can’t collect results of outcome except against a single reference bot and itself? Isn’t the point of AI to help it evolve and improve? To do that you have to be able to collect real data on its purpose (defeating another bot), to analyse and determine its mistakes, wrong choices and weaknesses, because you have a direct influence on your opponent’s performance.

How do we make our bots better?

3 Likes

Build a bot that beats your current AI

A fresh one perhaps?

You beat me to it. Now I know not everyone has that kind of time, But I currently have two different active “Bots”. and I build them up against each other.

Now it does not mean that I entered both, But one is slightly better though they both follow a different strategy. So I try and build different bots with different strategies if one bot completely thrashes the older one I will archive it. And currently I have 2 builds left thats close to each other. But the one reached the point of “I have no idea how I will better this” The other is a work in progress. But there’s potential.

The problem with the top tier bots is, The moment your bot slips up (Skips a move. Makes a move that causes them to trade say 2 Attack buildings for 1 attack building.) Your at a disadvantage. I am 90% sure that if you had to skip 2 turns after round 20 against Ralf, Justin or possibly Andre, You will lose. Against me not so much because my existing bot does not go for the game ending kill. sometimes it plays with the opponent. so one of my problems is I give my opponent chances to “get back”. cool against the lower bots for score, Not as cool against the top bots which might use that “wasted turn” of mine to turn the match.

Now on to finding possible code improvements: Here is what I do. I will play my bot against itself and view my matches using the replay viewer. Then I will analyze move for move and look if there wouldn’t have been a better move and I will check why it did not decide to make the move. This has helped me a bit in the past. For example using chess: “The perfect move is Knight to A5”. Oh look, it just moved a pawn. Why did it do that, Was its reasoning better than mine or did it have a flaw in its logic.

This so called observation is what lead to most of my “ideas”. I am already working on the bot that will break my existing entry. if I had my way I will finish it tonight. But then again, my “reference” bot ended third. So theres a rather high bar. But if I do not improve I can lose to any other player in the top 8 with a 50% 50% chance. And to me that is not good enough. And nothing is stopping them from improving. I mean, Every single bot can still develop.

My one and only strategy in developing my bot is to keep playing it against itself and trying to “see” a better strategy in replays. This is the trick I feel. You cannot implement a tactic that you cannot see. And you cannot see a tactic if you are not actually looking for one. By devising strategies against your “own” bot you are often able to build one that’s better, and if you repeat this process enough. Eventually your bot will get better. And often the one who does this most often will also be the one with the better bot.

Dont know if this will help, But trying to not deviate from the How do I build a better bot question.

3 Likes

I personally find that the theme here tends to be that leading players want to keep everything hidden, which is fair as it gives them a better chance of winning. However that does mean that it’s not beginner friendly.

The first time I played was pacman as a total beginner. It was totally fun and a decent learning experience, however for me a lot was lost as nothing was released afterwards. I haven’t really bothered since, as I always felt that I wasn’t good enough. I entered this year because I feel like I’m a bit better and I might have a chance now.

Contrast that to my experience on other platforms like hackerearth or codingame where I able to view replays, ask questions about my approach and get some advise even from the leaders. At the end the top players would post a postmortem about how they approached it. When doing those as a beginner I felt engaged all the time, no discouragement. I always had something to look at to get ideas from, and when out of ideas just ask better players what terms to search etc.

That’s my rant for today.

Also for those that have time codingame.com has a competition starting next week, try it I’ll be there :wink:

3 Likes