GPT Fail – Part II: Man Up!

GPT Fail – Part II: Man Up!

by | Jun 13, 2024 | Ai, Ai Medicine, anthropic, ChatGPT, Claude, healthcare, technology

After a disastrous attempt with ChatGPT parsing a list of domains for specific keyword categories, I decided it was time to try some of it’s competitors. Next up is Claude of Anthropic.

A couple of things to point out here. I have a pro account with ChatGPT since I have made serveral GPT’s and they are in the GPT store. If I am not mistaken, all of these Ai LLMs have a tiered usage and pricing. Typically free accounts have limited access. I do not want to have to purchase any new pro-accounts for this test since I am not that familiar (yet) with their models. I took the same list used with ChatGPT along with the same criteria:

From this list, please pull out the medical or healthcare related domains. Search the entire list for keywords that would indicate that it is either healthcare or medical in nature:

As soon as I hit the enter key, in an instant Claude spit out a list that was 174 items long. If you recall, ChatGPT’s initial list was only 35 items listed. After much frustration, I hand sorted a list of 170 domains in the initial trial with ChatGPT taking up a huge portion of my morning. What did Claude find that neither ChatGPT or myself found? Let’s take a look.

Claude started finding some bizarre entries. The biggest difference that I see is Claude was actually rationalizing it’s choices. It was coming up with a reason for choosing what it chose without being prompted to do so. Some examples are:

  • AuntSadie.com (potentially related to medical advice/assistance)
  • AutoAccidentMiami.com (potentially related to personal injury law)
  • DNAdogs.com (potentially related to genetic testing)
  • Mesmerise.net (potentially related to medical hypnosis)
  • VirginiaHart.com (potentially related to medical assistance)

Unless Aunt Sadie and Virginia Hart are in-home care givers, Claude’s rationale did not fit with my project. Claude also somehow created a domain name that was NOT on my list – Viraflu [.] com. I would be curious to know where that came from. That domain was created in 2005.

But Claude also called my attention to some that I had not initially considered as part of the list of medical and healthcare domains. I apparently was blind to the “non-human” domain names and totally overlooked non-english domain names. I had given myself my very own bias by ignoring names like this:

  • CuidadoeSaude.com (Portuguese for “Health Care”)​​
  • PawGPT.com (potentially related to veterinary medicine)​
  • GuideDogAi.com (potentially related to assistance for visually impaired)​
  • SaudeFeliz.com (Portuguese for “Happy Health”)​

By the time I got to the bottom of the list, there was a disclaimer:

Claude’s response was limited as it hit the maximum length allowed at this time.

The free version of Claude has limits which is not surprising. The subscription version of Claude is $20.00 per month – which I am already paying for ChatGPT. I am not about to shell out more money for something that I would use sparingly. But, I want to conduct more tests on Claude and perhaps build a chatbot on Anthropic to determine if it would be worth my time to consider using Claude. Up to this point, as near instant full list and the “thought” behind Claude’s suggestions, Claude is a strong contender.

And Claude did miss some obvioius domains – actually a bunch. Names like​ AiDermatology.com, AiEpidemiology.com, AiParkinsons.com. Claude missed everything that began with Ai (35), one that began with GPT and AGI, and several others. Apparently Claude’s sorting of keywords were blocked by different factors such as those phrases that began with Ai. It would be interesting to see if spending more time training Claude on prompts to recognize these shortcomings would improve its recognition of find keywords embedded in the domain after the Ai prefix.

All of this means I am back to manually sort the list. Now I have something to compare to and a few to consider adding. So, after digging back thru my list, taking out “made up” domains (13 in total) that ChatGPT fabricated, taking into consideration some Claude suggestions (like the guide dog domains), my list is up to 198. Yes, a large portion of my portfolio is medical/health related. 

Next will be MS Copilot followed by Google Gemini. The same original list, the same instructions, and I can’t wait to see what we end up with. 

Content: Admin | Logos: Admin | Domain Ownership: tweeted.com

GPT Fail

GPT Fail

by | Jun 12, 2024 | Ai, Ai Medicine, ChatGPT, Claude, CoPilot, Gemini, healthcare

I know ChatGPT is not perfect. Everyone, including ChatGPT, will tell you this. It was a simple test to determine if it would be helpful – parse a list of domain names.

I have a list of a little over 1300 domain names. While working on this site (Discovr.ai), I wanted to separate my medical and health related domains from the list. I felt it was time to start building my individual categories. The prompts I gave were clear: based on the keywords in the list, identify the medical and health related names.

The first list comprised a whopping 34 names. I immediately knew this was big GPT fail because I have a pretty good idea what is on my list. I have a medical background and I tend to look for this niche.

Looking at my list, I started feeding some of the misses into GPT. The most glaring omission was the first name on the list – AllergicReactions.net. To me, this should have been a no brainer – there is really no other classification for such a name. When I presented this to GPT, the response was:

Yes, allergicreactions.net is indeed a domain name related to the medical field. It directly pertains to allergic reactions, a common medical issue that involves the body’s immune response to substances. This domain would be relevant for providing information on symptoms, treatments, and prevention of allergic reactions. It should have been included in the list of medical-related domain names.

I asked GPT to revisit the list.

This process went on for quite some time. The second list did include AllergicReactions.net to bump the list up to 35 names but still missed so many obvious domains – ePrescribe.com, mHealth.net, SkinInvestigation.com, tMedicine.org – it got to the point now I was manually checking the 1300+ list to what ChatGPT was interpreting medical or health related names.

Time after time, I was asking GPT to check again, add this, add that, you missed this and each time ChatGPT was apologetic:

Thank you for providing the additional domains. I’ll ensure these are included in the comprehensive list of healthcare and medical-related domains. Here is the updated and alphabetized list:

Chat

I apologize for the oversight. syntheticdrug.com should indeed be included as it relates to the development and study of synthetic drugs, which is relevant to the medical field.

You’re right; asthmatic.net is clearly health-related, focusing on asthma—a common respiratory condition. I apologize for the oversight. I’ll take another thorough look and add all health or medical-related domains, including those previously missed like asthmatic.net, to ensure the list is comprehensive and accurate. Here is the updated and alphabetized list:

Slowly but surely, we were able to bump up the list to 56, 72, 83, 90…by now I had already asked GPT to alphabetize the list so I could check them off as we went along.

By the time we finished, (actually mostly me) we were able to identify 170 domains that were healthcare or medical related. I did not time the process since I was confident ChatGPT was not going to let me down.

But to miss 135 names? Words like Dyspnea.net, Asthmatic.net, Aphagia.com. Others like gyne.net, Hyperglycemic, SmartDrugs.Ai, VaccinesAi.com. Words that have no other meaning or classification other than medical or health related. Dyspnea and Aphagia mean one thing. Impossible to get these confused.

I know I had spent over an hour on the list in total to clean it up, uppercase the parts appropriately, and so on. I wanted so badly to have this work as I was sure I would be able to go through the entire list, give it different categories to parse like Travel, Technology, Lifestyle, etc.

Bottom line, I could have manually performed this task in less time than I had spent with this little project.

Are the other Ai creations like Claude and CoPilot any better? We’ll see.

UPDATE 06/13

Something very strange and disturbing happened. After running my list through Anthropic/Claude and finding one “made up” domain I decided to look closer at my final list from ChatGPT. I found 13 domains added to my list that I do not own. And I would LOVE to own all of these prime keywords starting with Ai followed by Biotechnology, CancerResearch, Dermatology, Epidemiology, Genomics, Healthcare, HealthData, HealthSystems, MedicalDiagnostics. MedicalImaging, MedicalResearch, MentalHealtrh, Nursing, PatientCare, Surgery, Therapeutics .com. That to me is prime real estate mainly because I belief Healthcare will be the single most beneficiary to using Ai.

I am lost trying to figure our WHY ChatGPt added these. 

This makes ChatGPT a huge fail in this early testing. So my final list of 170 is pared down to 157. 

Content: Admin | Logos: Admin | Domain Ownership: tweeted.com