How ChatGPT Crunched Through 262k Hotel Reviews Before I Could Finish My Coffee

Introducing the #AISummer2023 series, a continuation of the original #30DayAIChallenge in Spring. As the field of AI evolves, this series delves deeper into the latest advancements in AI tools and capabilities. It represents not just my practical tinkering over the Summer, but a metaphorical ‘summer’ of AI growth and evolution.

If you recall, Day 19 of the 30 Day AI Challenge explored the possibility of using Large Language Models (LLMs) like ChatGPT to supercharge our understanding of hotel review sentiment (or restaurants, experiences, products for that matter). This was back in early April and I basically copy-pasted a bunch of hotel reviews from an open source database to run my analysis using GPT4’s public chat interface. This worked pretty well, but wasn’t very scalable…for this follow-up experiment, I used ChatGPT with Code Interpreter to see how well the system would do with a large volume of reviews. Here’s how that went…

The Challenge

Objectives:

• Test using hotel reviews from the real world.
• Practically, we’d deal with thousands of reviews from a single market (and hundreds from a single hotel). I wanted to see if ChatGPT with Code Interpreter could handle the volume and still provide useful, speedy insights.
• Start broad with a market overview, then zoom in on one hotel.
• Play around, explore the data, and create some interesting insights / visuals.

Disclaimer:

• This purely academic test is just to show off what ChatGPT with Code Interpreter can do, while sparking further ideas and dialogue.
• I used a random dataset available publicly on Kaggle for this.
• I didn’t start out with a specific market or set of hotels in mind. It’s all based on what was available in the downloaded dataset…any filtering was done to keep the experiment focused and within tool constraints.
• Important Heads up: The data is from 2016 and a bit of 2017, so it might not match the latest hotel ratings or how people feel now. It’s also possible that some of these hotels no longer exist, have re-branded or renovated since then. Essentially…the objective of this post is not to provide specific or current-day hotel recommendations.

The Process

• Market – I noticed that the dataset contains reviews from various markets…for the purpose of this experiment, I filtered out all everything except for London. You’ll see from the stats below that about 400 unique hotels are represented in the data.
• The File – Code interpreter finally accepts more than one file but I suspect the file size limit is still about 100MB, so I also trimmed my CSV file down by removing the detailed address column and any reviews from 2015. This left us with a list of 262,068 reviews from hotels in London, from 2016 and early 2017. Here’s a general list of stats ChatGPT produced:

• The Prompts – Fairly straightforward, as always…simply going where needed with ChatGPT using natural language questions and dialogue.

An Analysis of Hotels in London – The Results

Here are some selected insights and visualizations produced by ChatGPT with Code Interpreter:

Hotel Reviews Based On Scores

Top 10 Hotels Based On Positive And Fewest Negative Review Counts

Top Nationalities Leaving Reviews

It’s clear that UK for UK travel dominates these reviews…a good flag when further analyzing the data or diving into sentiment for specific audiences.

Average Scores Given By Different Nationalities

Those working in the industry will not find the above differences in sentiment entirely surprising, given the differences in product and service quality across markets, plus source market expectations.

Monthly Trend Of Average Scores

While ChatGPT doesn’t spot a major seasonal trend, a “Summer Dip” seems quite evident to me…I think this would be clearer if we had data spanning at least 2-3 years. Most hotels in London (much like Europe) do see a massive spike in leisure travel in the summer and customer expectations are high…this is where many hotels also see a corresponding dip in satisfaction and ratings.

Positive And Negative Reviews – Word Clouds

Geographical Insights – Top and Bottom 10 Based On Average Scores

I asked ChatGPT to produce a downloadable map, which it did quite well…you can even see the hotel name when you click on specific markers. ChatGPT summarized the findings are follows: The hotels are spread across London, but you might notice certain clusters or patterns. For instance, some of the top-rated hotels are closer to popular attractions or prime locations in the city.

To summarize the market level insights:

1. General Statistics: Most reviews provide a score between 8 and 9, with an average score around 8.35. Most negative reviews are concise, while positive reviews are slightly longer.
2. Hotel Rankings: We identified the top 10 hotels based on average scores and the count of positive/negative reviews.
3. Reviewer Insights: The majority of reviews come from the UK, followed by the USA and Australia. The average score given varies slightly by nationality.
4. Temporal Trends: The number of reviews increased notably from mid-2016 to early 2017. The average score remained relatively stable over time.
5. Review Content Analysis: Common praises are related to service, location, and room comfort. Complaints often relate to room size, bed comfort, and noise.
6. Geographical Insights: The top-rated hotels are spread out but tend to be in or near prime locations.

Zooming in on ONE Hotel – The Results

For this part of the test, I asked ChatGPT to pick the top rated hotel in the dataset and provide key insights:

Analyzing Positive and Negative Reviews

Top Nationalities That Review This Hotel

Recommendations For Improvement

Clearly you can go much further in the analysis, but the above three examples show just how well the system manages to derive insights and make recommendations.

Exploratory Visualizations

I also ran some exploratory analysis, with a few highlights below:

Why Do UAE and KSA Guests Rate London Hotels Lower?

Mapping Which Hotels Are Rated Highest By Specific Nationalities

Here I asked ChatGPT to create a map showing which nationalities prefer which hotels across London, based on review counts as an indication. The dynamic map also includes a filter so you can select specific nationalities. In the example below, it appears guests from the UAE and Qatar tend to favour hotels in the Hyde Park, Park Lane and Knightsbridge areas.

Distribution Of Review Scores

The boxplot provides a visual distribution of review scores for the top and bottom 10 hotels:

• The top hotels tend to have a higher median score, with a majority of scores clustered around the 9-10 range. There are fewer low scores or outliers.
• The bottom hotels have a broader spread of scores, with the median scores generally lower than the top hotels. Some of these hotels also have a significant number of outliers.

Review Length Analysis

From the scatter plots of review lengths vs. scores:

• Positive Reviews: It seems there’s a trend where higher scores (8-10) tend to have shorter positive reviews, while mid-range scores (5-7) have a more varied length of positive reviews.
• Negative Reviews: For lower scores (below 5), guests tend to write longer negative reviews. As the scores increase, the length of negative reviews decreases, which is expected since higher scores would mean fewer negative points to mention.

Temporal Analysis

From the Day of Week Analysis:

• Average Review Score by Day of the Week: The average review score remains relatively consistent across different days of the week, with slight variations.
• Review Counts by Day of the Week: The number of reviews tends to peak around Tuesday and gradually decreases towards the weekend. This could be due to guests checking out after weekend stays and leaving reviews early in the week.

Hotel Facilities Analysis

Thoughts In Closing:

• Impressive Performance: This test exceeded my expectations. ChatGPT tackled a huge dataset and broke it down super fast.
• Deep Dives Made Easy: It’s not just about skimming the surface. You ask, and it dives deep. The analysis goes wherever you steer it, all through casual chat prompts.
• Future of Analytics: Honestly, this feels like a sneak peek into the future of business analytics. Imagine just chatting with your data, making sense of it all in real time. That’s where we’re headed.
• Practical Uses: Think reputation management. With tools like this, we can instantly decode review sentiments, get alerts, and even draft responses for a quick human check before posting. The routine stuff gets a whole lot easier and faster.
• Endless Potential: There’s a thrilling future ahead of us. This is just the beginning, and it’s already so promising!

What did you think? Leave a comment or send me a personal message to share your thoughts, own experiments and suggestions on where we can push the limits even further (or new experiments I should run in the future).