For those interested in health coaching and the role of AI, once in a while we query AIs on their understanding of the skill. It is useful for us to understand their capabilities, differences and how to apply the technology when practising our own daily health activities.
Previously we have 'interviewed' ChatGPT, following a typically question and answer format. We thought to do the same with Google's Bard and the conversation flowed somewhat differently. Importantly, for those who use AI for health research or content creation, we discovered a major problem with its content - the creation of imaginary citations, so called, hallucinations.
Our premise was to ask Bard the same dozen questions that we had asked ChatGPT, to provide a direct comparison between the 2 AIs. The answers came in a very different format. Here is how we started with Bard, the exact initial question as posed to ChatGPT, response in italics.
"Hello Bard. I would like to interview you about health coaching and the role of AI."
Yes, health coaching has been shown to be effective for a variety of health conditions. A recent review of 65 studies found that health coaching led to significant improvements in blood pressure, cholesterol, blood sugar, weight, and physical activity. Health coaching has also been shown to improve quality of life and reduce hospital admissions in patients with chronic diseases such as diabetes and heart disease.
One of the reasons why health coaching is so effective is that it is a personalized approach to wellness. Health coaches work with clients to identify their individual goals, challenges, and motivations. They then provide guidance and support to help clients develop and implement a personalized plan to achieve their goals.
Health coaches also play an important role in helping clients to stay motivated and on track. They provide regular support and accountability, which can be essential for making lasting changes to one's health and lifestyle.
Overall, the evidence shows that health coaching is an effective way to improve health and well-being. If you are interested in improving your health, I encourage you to consider working with a health coach.
Here are some specific examples of how health coaching has been shown to be effective:
A study of people with diabetes found that those who received health coaching were more likely to have lower blood sugar levels and fewer complications from diabetes.
A study of people with heart disease found that those who received health coaching were more likely to lose weight, lower their cholesterol levels, and increase their physical activity levels.
A study of people with chronic pain found that those who received health coaching were able to reduce their pain levels and improve their quality of life."
Bard responded with a broad ranging answer covering various aspects of health coaching, it didn't specifically answer the question itself. Is this response time saving, or, pushy? Bard certainly addressed several of the other questions that I was going to ask and potentially answered questions that I might not have asked under other circumstances.
For reference, ChatGPT answered the question. "Hello! I am happy to help with your interview questions about health coaching and the role of AI. Please go ahead and ask your questions." We then moved on to Question #2.
Unlike ChatGPT, Bard provided mentions of studies, without citations. When asked to "provide references to any scientific studies that demonstrate this?" Bard responded:
"Sure. Here are some references to scientific studies that demonstrate the effectiveness of health coaching:
A randomized controlled trial of health coaching to improve cardiovascular risk factors in adults with overweight or obesity. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5405056/
The effect of health coaching on HbA1c and quality of life in adults with type 2 diabetes: a randomized controlled trial. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5387353/
The effectiveness of health coaching on weight loss and maintenance in adults with overweight or obesity: a systematic review and meta-analysis. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5470473/
The effectiveness of health coaching on pain management and quality of life in adults with chronic pain: a systematic review and meta-analysis. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5488033/
The effectiveness of health coaching on medication adherence and health outcomes in adults with chronic diseases: a systematic review and meta-analysis. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5720371/
These are just a few examples of the many studies that have been conducted on the effectiveness of health coaching. The evidence is clear that health coaching is an effective way to improve health and well-being for people with a wide range of health conditions."
Great to be provided with some studies, an opportunity to learn, as well as add to our own database of meta-analyses and systemic reviews. (please feel free to access them all here)
We checked the references. BIG PROBLEM. None of the articles, using Google Search, seem to exist; certainly not when cut and paste into the browser. The URLs Bard provided exist, but link to other unrelated articles within PubMed.
For your reference, we have included the URLs into the quoted text, Bard AI did not add the links.
We followed up with the question "We have checked the citations that you provide and clicked on the links. The named studies do not exist and the links provide access to research papers that relate to other studies. You appear to have invented this data. Is this correct?"
Bard's response: "I apologize for the mistake in my previous response. I am still under development and I am always learning. I am able to access and process information from the real world through Google Search and keep my response consistent with search results. However, I am not perfect and I sometimes make mistakes.
In this case, I was able to find some research papers on the effectiveness of health coaching, but I was not able to find the specific studies that I cited. I am not sure why this is, but it is possible that the studies have been retracted or that they are not available online.
I am working on improving my ability to access and process information accurately. I am also working on improving my ability to generate citations that are accurate and complete.
Thank you for bringing this mistake to my attention. I appreciate your help in making me a better language model."
We dug a little further and found an interview with a Google executive dealing with this very problem. The explanation? That Bard is a tool for creative work, imagination, to brainstorm ideas and generate content. for factual information best to use Google Search itself.
There is a technical word for these fabrications: "hallucinations". An August 2023 study on tis phenomenon stated that "Our main finding was that both Chat GPT 3.5 and Bard 2.0 are inaccurate at citation generation. ChatGPT, when queried in March 2023 for references during a literature search, provided a list of 35 citations of which only 6% matched actual manuscripts. Of the 33 fabricated citations provided, 12 could be matched by title to real articles and 21 had no clear origin."
ChatGPT, in the unpaid version that we have access to, is not yet able to provide references. Here is what the ChatGPT had to say on the matter:
"I don't have direct access to my training data or sources, so I can't provide specific references to scientific studies. However, I can mention that there is a growing body of research supporting the effectiveness of health coaching. Many studies have been conducted in recent years, and you can find them by searching reputable medical and psychological databases such as PubMed, Google Scholar, or the Cochrane Library.
When looking for scientific studies on health coaching, you might consider using keywords such as "health coaching effectiveness," "health coaching outcomes," or "patient coaching research." Additionally, you can specify the particular health condition or behavior you are interested in, like "diabetes management health coaching" or "weight loss health coaching," to narrow down the search results.
It's important to critically evaluate the quality of the studies you find, considering factors like the study design, sample size, duration, and the expertise of the researchers involved. Meta-analyses and systematic reviews can provide a comprehensive overview of multiple studies on a specific topic, offering valuable insights into the overall effectiveness of health coaching interventions."
Ok, some useful references to databases and even advice how to query a database. If you have ever tried to find papers from within PubMed, knowing how to use language correctly - the correct combinations of word and using the correct vocabulary - is vital.
Interestingly Bard also had this to say, as its final words to the original, single question...
"If you are considering working with a health coach, I recommend that you find a coach who is certified by a reputable organization such as the National Board for Health & Wellness Coaching or the American Council on Exercise. You should also interview the coach to make sure that they are a good fit for you and your needs."
Using AI has been invaluable to us, as a health practice trying to learn, to grow and make the most out of the ever changing technology. However, there are dangers. Whilst I had heard that AIs have previously made up information, I was surprised to see such well presented data - study names and links - to be false. Given the problems that we see with online influencers misrepresenting health studies and data - knowingly or not - this is part of the age that we live in. Make the most of AI, with caution.
Stay Healthy,
Alastair
Alastair Hunt NBC-HWC, is the founder of The Whole Health Practice Pte Ltd. a leading lifestyle and whole person health practice, based in Singapore and working with clients globally. www.thewholehealthpractice.com
If you would like to read the original interview with ChatGPT, click on the image below...
Related Studies
McGowan A, Gui Y, Dobbs M, Shuster S, Cotter M, Selloni A, Goodman M, Srivastava A, Cecchi GA, Corcoran CM. ChatGPT and Bard exhibit spontaneous citation fabrication during psychiatry literature search. Psychiatry Res. 2023 Aug;326:115334. doi: 10.1016/j.psychres.2023.115334. Epub 2023 Jul 7. PMID: 37499282; PMCID: PMC10424704.
Kim JK, Chua M, Rickard M, Lorenzo A. ChatGPT and large language model (LLM) chatbots: The current state of acceptability and a proposal for guidelines on utilization in academic medicine. J Pediatr Urol. 2023 Oct;19(5):598-604. doi: 10.1016/j.jpurol.2023.05.018. Epub 2023 Jun 2. PMID: 37328321.
Comments