In this week’s episode of “SEOs Getting Coffee,” SEO experts Emina Demiri-Watson and Sean Carroll embark on an in-depth exploration of crawling, internal linking, robot.txt files, and more, unravelling the intricacies of SEO optimisation.
Join us as we delve into the core elements of crawling and the robot.txt file, shedding light on their significance in SEO strategy.
Watch Video
SEOs Getting Coffee - Episode #14: Mastering Crawling & Crawl Budge
Crawling: Why it should never be overlooked
At the heart of SEO lies the intricate process of crawling, where search engine bots systematically navigate through web pages to index content. Sean provides a comprehensive definition of crawling, emphasising its pivotal role in the broader SEO framework. Emina underscores the importance of understanding crawling in the context of other essential SEO processes, such as indexing and ranking, highlighting the nuanced interplay between these elements.
Crawl Budget: Myth or Reality?
Delving deeper into crawling, Sean and Emina unpack the concept of crawl budget, debunking common misconceptions and clarifying its significance. They address concerns about crawl budget optimisation and offer insights into its relevance for websites of varying sizes. Drawing on real-world examples from e-commerce and news sites, they illustrate the potential pitfalls of mismanaging crawl budgets and emphasise the need for strategic planning and optimisation.
The Power of Internal Linking
Transitioning to internal linking, Sean introduces an analogy to elucidate its importance in SEO strategy. He outlines the critical role of internal linking in enhancing website navigation, optimising crawl paths, and distributing link equity. Emina expands on this discussion, sharing practical tips and tools such as Link Whisper for effective internal linking, including insights from her recent article for Screaming Frog. The duo then further explore the benefits and drawbacks of automated internal linking tools and emphasises the value of an integrated approach. Highlighting that a fully automated approach is not the ideal solution.
Robot.txt File: The Gatekeeper of Web Crawling
In their exploration of robot.txt files, Sean and Emina underscore its significance in regulating bot access to website content. They caution against haphazard edits to the robot.txt file and emphasise the importance of including a sitemap for efficient content discovery. Reflecting on recent observations, they discuss the trend of selectively allowing or disallowing bots from crawling specific website sections, highlighting the need for informed decision-making in SEO strategy.
Room 404: Decoding Google's Reddit Partnership
This week, it was unanimously decided that Google and Reddit’s partnership should be sent to Room 404. Sean and Emina evaluate Google’s partnership with Reddit and its potential ramifications for search engine rankings. They express concerns about the partnership’s implications for smaller news publishers. Emphasising the importance of transparency and accountability in search engine partnerships, they advocate for ongoing dialogue and critical engagement within the SEO community.
Closing Thoughts: Mastering Crawling for SEO Success
As the conversation draws to a close, Sean and Emina offer parting insights and reflections on the topics explored. They highlight the importance of understanding crawling and robot.txt management in modern SEO practices. Emphasising the need for awareness and adaptability, they highlight the significance of staying informed about algorithmic changes and strategic partnerships, such as the recent Google and Reddit collaboration.
For more insights and in-depth conversations on the latest in SEO and digital marketing strategies, stay stuned for upcoming episodes of “SEOs Getting Coffee.” Subscribe to our channel for regular updates and expert opinions.
