GPT Fail – Part II: Man Up!

GPT Fail – Part II: Man Up!

After a disastrous attempt with ChatGPT parsing a list of domains for specific keyword categories, I decided it was time to try some of it’s competitors. Next up is Claude of Anthropic.

A couple of things to point out here. I have a pro account with ChatGPT since I have made serveral GPT’s and they are in the GPT store. If I am not mistaken, all of these Ai LLMs have a tiered usage and pricing. Typically free accounts have limited access. I do not want to have to purchase any new pro-accounts for this test since I am not that familiar (yet) with their models. I took the same list used with ChatGPT along with the same criteria:

From this list, please pull out the medical or healthcare related domains. Search the entire list for keywords that would indicate that it is either healthcare or medical in nature:

As soon as I hit the enter key, in an instant Claude spit out a list that was 174 items long. If you recall, ChatGPT’s initial list was only 35 items listed. After much frustration, I hand sorted a list of 170 domains in the initial trial with ChatGPT taking up a huge portion of my morning. What did Claude find that neither ChatGPT or myself found? Let’s take a look.

Claude started finding some bizarre entries. The biggest difference that I see is Claude was actually rationalizing it’s choices. It was coming up with a reason for choosing what it chose without being prompted to do so. Some examples are:

  • AuntSadie.com (potentially related to medical advice/assistance)
  • AutoAccidentMiami.com (potentially related to personal injury law)
  • DNAdogs.com (potentially related to genetic testing)
  • Mesmerise.net (potentially related to medical hypnosis)
  • VirginiaHart.com (potentially related to medical assistance)

Unless Aunt Sadie and Virginia Hart are in-home care givers, Claude’s rationale did not fit with my project. Claude also somehow created a domain name that was NOT on my list – Viraflu [.] com. I would be curious to know where that came from. That domain was created in 2005.

But Claude also called my attention to some that I had not initially considered as part of the list of medical and healthcare domains. I apparently was blind to the “non-human” domain names and totally overlooked non-english domain names. I had given myself my very own bias by ignoring names like this:

  • CuidadoeSaude.com (Portuguese for “Health Care”)​​
  • PawGPT.com (potentially related to veterinary medicine)​
  • GuideDogAi.com (potentially related to assistance for visually impaired)​
  • SaudeFeliz.com (Portuguese for “Happy Health”)​

By the time I got to the bottom of the list, there was a disclaimer:

Claude’s response was limited as it hit the maximum length allowed at this time.

The free version of Claude has limits which is not surprising. The subscription version of Claude is $20.00 per month – which I am already paying for ChatGPT. I am not about to shell out more money for something that I would use sparingly. But, I want to conduct more tests on Claude and perhaps build a chatbot on Anthropic to determine if it would be worth my time to consider using Claude. Up to this point, as near instant full list and the “thought” behind Claude’s suggestions, Claude is a strong contender.

And Claude did miss some obvioius domains – actually a bunch. Names like​ AiDermatology.com, AiEpidemiology.com, AiParkinsons.com. Claude missed everything that began with Ai (35), one that began with GPT and AGI, and several others. Apparently Claude’s sorting of keywords were blocked by different factors such as those phrases that began with Ai. It would be interesting to see if spending more time training Claude on prompts to recognize these shortcomings would improve its recognition of find keywords embedded in the domain after the Ai prefix.

All of this means I am back to manually sort the list. Now I have something to compare to and a few to consider adding. So, after digging back thru my list, taking out “made up” domains (13 in total) that ChatGPT fabricated, taking into consideration some Claude suggestions (like the guide dog domains), my list is up to 198. Yes, a large portion of my portfolio is medical/health related. 

Next will be MS Copilot followed by Google Gemini. The same original list, the same instructions, and I can’t wait to see what we end up with. 

Content: Admin | Logos: Admin | Domain Ownership: tweeted.com