I have a question for the course designers. I feel like with the replay I did using AI golfers I was able to locate every flaw in every course made. Obviously that wasn't my goal, but the AI players just seem to find the bugs. I was also using 8 AI golfers on 3 different PCs at once so I had 24 play testers at a time. I then played 45 years so you can imagine the play that I "witnessed" on each course.
What I began to notice that on certain courses, certain holes always generated crazy over par (like +8 or +10) scores for all or a percentage of the golfers. I worked around this in my replay by capping the over par to +5 on any hole, but it allowed me to find the occasional issues. Keep in mind that most courses played beautifully without issues.
I bring this all up because my AI golfers just found an issue with Ballyhack and I'm currently running test rounds on all the newly created courses so I can determine if I can use them for my next round of AI seasons/careers. I absolutely loved Ballyhack when I took for a practice round so it was the first course I tested with my AI golfers.
I guess what I'm proposing is for designers to grab a group of AI golfers (8 is max) and have them play a full round (it should take about 3 hours). After the round check the scorecard for holes that have crazy scores on them. That prevents you from having to actually watch the round. Then you can isolate in the holes that create anomalies from a scoring standpoint.
In the case of Ballyhack, I was using a group of golfers in the 74 to 76 score range. I think that's a perfect rated AI golfer for testing because they are good enough to not just generally suck, but not so good that you will never see weird stuff.
Just my .02, so I guess I've given .04 tonight. My wife says I have no sense and I don't make any sense so I guess I'll quit while I'm ahead.
Course Testing
Re: Course Testing
As a player (not a designer) my question would be is the problem the inflated score, or is there an issue with the course that led to that score? is it a pin that may be questionable for the conditions. If I were a designer, this is just one of the types of questions I'd ask. And, please, don't think that I'm questioning what you've found! I'm not! Seems like you go to a lot of effort doing what you do.
No matter where you go, there you are.
-
Big Sexy JC
- Posts: 221
- Joined: November 15th, 2022, 8:10 am
- Location: Indiana
Re: Course Testing
GoBucks,
I think it could be a combo of things, but all of them are signs of an issue. A bad pin placement could ruin a hole/course as much as a bug could. I think anytime AI golfers that average par to 5 under par are getting +8 on a single hole it's a sign that something needs looked at. I can tell you every time I see these scores I pay attention to that hole on that course to try to find the root cause. I'd say 70% of the time it's some bug in the design and 30% it's a bad pin placement.
The point of my post is that, in my opinion, the AI golfers make better testers than humans. You could get a pretty thorough test in 3 hours.
I think it could be a combo of things, but all of them are signs of an issue. A bad pin placement could ruin a hole/course as much as a bug could. I think anytime AI golfers that average par to 5 under par are getting +8 on a single hole it's a sign that something needs looked at. I can tell you every time I see these scores I pay attention to that hole on that course to try to find the root cause. I'd say 70% of the time it's some bug in the design and 30% it's a bad pin placement.
The point of my post is that, in my opinion, the AI golfers make better testers than humans. You could get a pretty thorough test in 3 hours.
Re: Course Testing
Point noted.Big Sexy JC wrote: ↑January 19th, 2024, 12:22 pm GoBucks,
I think it could be a combo of things, but all of them are signs of an issue. A bad pin placement could ruin a hole/course as much as a bug could. I think anytime AI golfers that average par to 5 under par are getting +8 on a single hole it's a sign that something needs looked at. I can tell you every time I see these scores I pay attention to that hole on that course to try to find the root cause. I'd say 70% of the time it's some bug in the design and 30% it's a bad pin placement.
The point of my post is that, in my opinion, the AI golfers make better testers than humans. You could get a pretty thorough test in 3 hours.
No matter where you go, there you are.
-
Colin Jones
- Posts: 71
- Joined: January 29th, 2021, 5:59 pm
- Location: Gold Coast, Australia
Re: Course Testing
I run hundreds of rounds with AI golfers throughout the life of a design. Always have. Usually 8, at all levels from all tees. It's incredibly time-consuming (and frustrating) to do it regularly - and then a real blitz on betas and finals, but well worth it.
I certainly don't agree, tho, that a +8 score by an AI golfer must be an indication of a design flaw. I think I am well-qualified as a designer and a prolific AI user to say that is just not the case. Might be, but might not be too.
fwiw, a problem I do find with using AI golfers during design is that no matter how low you set their levels, they are never completely shite. So AI golfers will nearly always hit the fairway and the green - or get close. A focus on testing on the results of AI players only, risks that you get focused on those areas and not outer areas of each hole where some real human players might actually find themselves.
I certainly don't agree, tho, that a +8 score by an AI golfer must be an indication of a design flaw. I think I am well-qualified as a designer and a prolific AI user to say that is just not the case. Might be, but might not be too.
fwiw, a problem I do find with using AI golfers during design is that no matter how low you set their levels, they are never completely shite. So AI golfers will nearly always hit the fairway and the green - or get close. A focus on testing on the results of AI players only, risks that you get focused on those areas and not outer areas of each hole where some real human players might actually find themselves.
Completed: Golf Club of Houston (Redstone); Banff Springs; Mauna Kea 2024
Working on: Royal Sydney (2024); NSW Golf Club (2024)
Working on: Royal Sydney (2024); NSW Golf Club (2024)
-
Big Sexy JC
- Posts: 221
- Joined: November 15th, 2022, 8:10 am
- Location: Indiana
Re: Course Testing
Colin,
I mean a +5--+8 on a single hole, not a round. A score that high on a single hole, in my experience, is almost always a sign of an issue. An issue could be defined a number of ways. I remember on one of the Bay Hill holes that was a long par 5 around a lake where the players that hit in the water would drop and then hit right back in the water.
They would literally do this until they dropped out of the hole because they were at 12 shots. On that same hole, some AI players would realize they couldn't reach the green from the drop and instead of just playing out to the fairway on the right they would hit the ball back towards the tee box and get to the hole from the complete opposite direction. In other words they would go the other way around the lake to the hole. I don't necessarily think that's a design flaw unless it's some kind of pathing issue. In some cases it's just the AI from a game made in the late 90s rearing it's head.
I've been in IT for 30 years so I just look for efficiency in troubleshooting. I'm not saying AI players are the only answer and I'm one that doesn't think there is ever a single answer for anything. I'm saying they should definitely be a big part of the play testing on a new course and it sounds like you do a pretty thorough job of that. A 3 hour run with 8 AI players is a pretty efficient way to get a heads up of potential issues with a course.
I"ve tested about 8 courses over the last couple days and just print the scorecard to a text file and start the next round. I dump the text scorecards into a spreadsheet and identify holes that could be issues. 3 of the 8 courses I've tested have had at least one problem hole. I don't think that means the course should be put out to pasture and is a bad course. Hell, the only reason I do it is to help determine what courses I use in my rotation of tournaments so it's completely useless info to anyone else but me.
I'm just an idea guy and I was throwing one out there. If designers choose to or choose not to do it won't hurt my feelings any.
I mean a +5--+8 on a single hole, not a round. A score that high on a single hole, in my experience, is almost always a sign of an issue. An issue could be defined a number of ways. I remember on one of the Bay Hill holes that was a long par 5 around a lake where the players that hit in the water would drop and then hit right back in the water.
They would literally do this until they dropped out of the hole because they were at 12 shots. On that same hole, some AI players would realize they couldn't reach the green from the drop and instead of just playing out to the fairway on the right they would hit the ball back towards the tee box and get to the hole from the complete opposite direction. In other words they would go the other way around the lake to the hole. I don't necessarily think that's a design flaw unless it's some kind of pathing issue. In some cases it's just the AI from a game made in the late 90s rearing it's head.
I've been in IT for 30 years so I just look for efficiency in troubleshooting. I'm not saying AI players are the only answer and I'm one that doesn't think there is ever a single answer for anything. I'm saying they should definitely be a big part of the play testing on a new course and it sounds like you do a pretty thorough job of that. A 3 hour run with 8 AI players is a pretty efficient way to get a heads up of potential issues with a course.
I"ve tested about 8 courses over the last couple days and just print the scorecard to a text file and start the next round. I dump the text scorecards into a spreadsheet and identify holes that could be issues. 3 of the 8 courses I've tested have had at least one problem hole. I don't think that means the course should be put out to pasture and is a bad course. Hell, the only reason I do it is to help determine what courses I use in my rotation of tournaments so it's completely useless info to anyone else but me.
I'm just an idea guy and I was throwing one out there. If designers choose to or choose not to do it won't hurt my feelings any.
Re: Course Testing
JC,
I have never thought about using anis for testing, and I see a lot of sense in what you say, though I am unsure if I have the patience to do it. (3 hours for each pin position
)
Thanks,
Ian
I have never thought about using anis for testing, and I see a lot of sense in what you say, though I am unsure if I have the patience to do it. (3 hours for each pin position
) What do you mean by this? Does this mean that there is a pin or a number of pins or all the pins are a problem?3 of the 8 courses I've tested have had at least one problem hole.
Thanks,
Ian
Re: Course Testing
What conditions (wind, firmness, speed) do you use to test these courses? How do you select the pins for each round?
How does the AI player select where to drop? Real testers have options. If I continue to rinse and repeat from the same drop, I'm thinking that that's not an issue with the course.
How does the AI player select where to drop? Real testers have options. If I continue to rinse and repeat from the same drop, I'm thinking that that's not an issue with the course.
No matter where you go, there you are.
-
Colin Jones
- Posts: 71
- Joined: January 29th, 2021, 5:59 pm
- Location: Gold Coast, Australia
Re: Course Testing
I'm a constant tinkerer - textures, alphas, objects, everything! So I am always designing then testing in game play. When I test with the anis, I have a notepad where I'll jot down everything I am not happy with and need to change.
My 8 ani golfers (a mix of great and totally useless golfers playing off varying tees - 4 off the back, 1 each off the other tees) give me 8 different views of the course for just about every shot - that's the main reason I use them. In the case of Mauna Kea, the design was finished where my golfers had all played a round and there was nothing to write in my notebook. During all this testing, I vary the pins and the golfer settings (close-up, panorama, etc) as much as I think I need to (or much as I remember to do so). I might select all 1 pins for a test then all 2s the next time I test. I write down most things that I have done. Random weather conditions are perfect.
I know from all this that even the best AI golfers do some wierd shit when faced with challenging (but not necessarily faulty) conditions and shots! Hence my comments about +8 not being a sign of a course design fault necessarily.
Ian, I don't always have 18 holes for every test, so it's not a full round every time. So, if I have been working on holes 12-15 for example, then I select to play the back nine of my course, use a single ani golfer for holes 10 & 11 then add in all the other anis for holes 12-15.
As far as "drops" and dropzones go? That can be tricky, but you should be able to work it out as a designer (I fear I am about to be sprung with a drop issue with MK am I?)
I don't play Links myself.
My 8 ani golfers (a mix of great and totally useless golfers playing off varying tees - 4 off the back, 1 each off the other tees) give me 8 different views of the course for just about every shot - that's the main reason I use them. In the case of Mauna Kea, the design was finished where my golfers had all played a round and there was nothing to write in my notebook. During all this testing, I vary the pins and the golfer settings (close-up, panorama, etc) as much as I think I need to (or much as I remember to do so). I might select all 1 pins for a test then all 2s the next time I test. I write down most things that I have done. Random weather conditions are perfect.
I know from all this that even the best AI golfers do some wierd shit when faced with challenging (but not necessarily faulty) conditions and shots! Hence my comments about +8 not being a sign of a course design fault necessarily.
Ian, I don't always have 18 holes for every test, so it's not a full round every time. So, if I have been working on holes 12-15 for example, then I select to play the back nine of my course, use a single ani golfer for holes 10 & 11 then add in all the other anis for holes 12-15.
As far as "drops" and dropzones go? That can be tricky, but you should be able to work it out as a designer (I fear I am about to be sprung with a drop issue with MK am I?)
I don't play Links myself.
Completed: Golf Club of Houston (Redstone); Banff Springs; Mauna Kea 2024
Working on: Royal Sydney (2024); NSW Golf Club (2024)
Working on: Royal Sydney (2024); NSW Golf Club (2024)
Re: Course Testing
Colin,
(How do I get a mix of "great and totally useless golfers"?)
Ian
That sounds a very good way of testing which on future courses I will use. Thank you.My 8 ani golfers (a mix of great and totally useless golfers playing off varying tees - 4 off the back, 1 each off the other tees) give me 8 different views of the course for just about every shot - that's the main reason I use them.
(How do I get a mix of "great and totally useless golfers"?)
Ian