Create a Simple XML Sitemap With Python.Randomize User-Agent With Python and BeautifulSoup.Web Scraping With Python and Requests-HTML.Find Keyword Cannibalization Using Google Search Console and Python.Recrawl URLs Extracted with Screaming Frog (using Python).Find Rendering Problems On Large Scale Using Python + Screaming Frog.#What you get here is where you should save your CSV crawlsĭfTextonly = pd.DataFrame(pd.read_csv('Text-only-5000-crawl.csv', low_memory=False, header=1))ĭfJS = pd.DataFrame(pd.read_csv('JS-Rendered-5000-crawl.csv', low_memory=False, header=1))ĭfJS = dfJS].copy()Ĭombine the two crawls into one dataframeĭf = pd.merge(dfTextonly, dfJS, left_on='Address', right_on='Address', how='outer')ĭf = np.where((df = df), "yes", "no")ĭf.to_excel("rendering-test.xlsx") Other Technical SEO Guides With Python #Print the path of your current working directory To export your data into Excel, just use the to_excel panda function.ĭf.to_excel("rendering-test.xlsx") Full Python Code Here, you should get a result that looks like this. #Check if canonical links are equivalentĭf = np.where((df = df), "yes", "no") Source: Tobias Willmann We want to flag pages with big differences between each other.īecause, it will mean that a lot of content is hidden behind JavaScript and can’t be accessed from Google’s first wave crawling.ĭf = df - df Here, what we will do is to count the differences in the number of words and the number of links from our “Text Only” crawl and our “JavaScript Rendered” crawl. Note: In Spyder IDE, Step #5: Check Differences Between Crawls Pandas has automatically added the “_x” and “_y” to the data of the first and the second crawls. What you’ll get is a new dataframe with the same column names twice. This is an easy step, just copy the code below.ĭf = pd.merge(dfTextonly, dfJS, left_on='Address', right_on='Address', how='outer') Source: Tobias Willmann Since JS rendering mostly affect SEOs in its capacity to render links and content, we’ll try to see if a bot can load the content by checking the word count and the Link information in the Crawl.ĭfTextonly = pd.DataFrame(pd.read_csv('Text-only-crawl.csv', low_memory=False, header=1))ĭfTextonly = dfTextonly].copy()ĭfJS = pd.DataFrame(pd.read_csv('JS-Rendered-crawl.csv', low_memory=False, header=1))ĭfJS = dfJS].copy() Source: Tobias Willmann Step #4: Combine the Crawls Into One Data Frame Go in Screaming Frog > Export Step #3: Load the Crawl Data Using Python Now that your crawl is complete, you will want to export the Data to CSV. JavaScript Rendering Step #2: Export The Data To CSV You can always recrawl the problematic URLs later on in list mode if you want to see the rendered pages Screen Shots. Make sure that you unselect “Enable Rederred Page Screen Shots” if you have a really large site. Go in Screaming frog > Configuration > Rendering > JavaScript This will mimic which link that Google will find in its second wave, where it renders the JS content after it has available resources. Now, let’s crawl our website including rendered results. Go in Screaming frog > Configuration > Rendering > Text Only Text-only Rendering JavaScript Rendered Crawl In fact, to compare which pages load properly let’s make two crawls:įirst, let’s crawl our website like Googlebot would do in its first wave before it renders the JS. Step #1: Make Two Crawls With Screaming Frog Make two crawls with Screaming Frog, one with “Text Only” rendering and the other with “JavaScript” rendering.Now it is time to put our website’s JavaScript (JS) to the test. How to Test JavaScript Rendering on a Large Scale? (Step-By-Step) This guide will be fully explained using Spyder that is natively installed when you install Python using Anaconda. If you have no idea how Python works, just look at my two guides on the subject: how to install Python with Anaconda and my Python Basics Complete Guide. To understand this guide you will need to have Python installed, and you will need to have at least a basic knowledge of Python. Make sure that Googlebot can find it straight in your HTML code.īut what happens when you have thousands of pages and you don’t know which resources are loaded via JavaScript, and which are not? If you have important content, or important links. ![]() Yes, all your cute little content hidden behind JavaScript will be deferred, until Google finds it suitable to come back and spend money on your site to load your JS resources. Google Rendering Process Why Test Rendering? This means that the first process the HTML, and IF they have more resources, they’ll come back to load your JavaScript content. Google basically says this: The rendering of JS is deferred until Googlebot has resources available to process that content.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |