Website Crawler and XML Sitemap Generator


Your privacy is important to us. We will never share your information with third parties.
If you would like to prevent this free tool from crawling your website, please add the following lines to your robots.txt file:
User-agent:NinjaBot
    Disallow: /

See how to use Website Crawler and XML Sitemap Generator

The Website Crawler Tool, which acts as something of a Google sitemap generator will crawl your whole website, help you spell check, create your Google sitemap and point you to some errors on pages (e.g. broken links, 404 pages, etc) with our Ninja Website Crawler Tool.

Check All of the Links on a Website

A Note on User Privacy: Your privacy is important to us. We will never share your information with third parties.

About the Tool

Links are critical to webpages, in terms of both informing the reader and positive SEO best practices. The Find broken Links, Redirects & Site Crawl Tool allows you to review the status of both external links and internal links on a website. The resulting report generated by this tool provides important insight into the link schema of the site, and identifies link redirects and errors; all of which help in planning a link optimization strategy. The tool also features downloadable results, a sitemap feature, and is free for anyone to use.

Getting Started: How it Works

  • Type in or copy/paste the website’s homepage URL into the specified text box.
  • Select the range of URLs from the input website that you wish to be scanned.
  • Select ‘Check

Keep in mind, this tool generates in real-time and may take up to 30 minutes for data collection and processing, based on the number of URLs being reviewed. For your convenience, this tool also allows you to receive your results via email, once the crawl is complete.

A Note to Tool Users: Due to the resources which this tool utilizes, there is a running limit of 5 tool runs per day, per user. To continue benefiting from this tool, please bookmark this page and come back after 24 hours.

Tool Results Bar

The results table data of this tool is interactive; most of the data is linked, either to the URLs referenced or to details about the data. For most cells that contain non-URL data, position the cursor over the cell to see the full results. While the tool is running, a results bar begins to appear, displaying the following information:

  • Status of the tool (Crawling or Done)
  • Number of Internal URLs crawled
  • Number of External links found
  • Number of Internal HTTP Redirects found
  • Number of External HTTP Redirects found
  • Number of Internal HTTP error codes found
  • Number of External HTTP error codes found

Once the tool has finished crawling the input website, an XML sitemap, as well as downloadable results will be available. Downloadable tool results can be accessed as an Excel spreadsheet or HTML file. These files can be accessed using the buttons of the appropriate title at the top of the page.

The tool outputs six tables providing detailed information regarding the following areas, each of which can be downloaded as XLS and HTML files:

  1. Internal links
  2. External links
  3. Internal errors (a subset of Internal Links)
  4. Internal redirects (another subset of Internal Links)
  5. External errors (a subset of External Links)
  6. External redirects (another subset of External Links)

The following is a breakdown of each of the different tables included.

Internal Links Table

Includes the following fields of information:

  • URLs crawled on the site
  • URL’s level from the domain root
  • URL’s returned HTTP status code
  • Number of internal links the URL has within the site (click to see the list of URLs)
  • Link text used for the URL
  • Number of internal links on the page (click to see the list of URLs)
  • Number of external links on the page (click to see the list of URLs)
  • Size of the page on kilobytes (click to see page load speed test results for this URL from Google)
  • The tag text from the URL’s page
  • The description tag text from the URL’s page
  • The keywords tag text from the URL’s page
  • Contents, if used, of the anchor tag’s “rel=” attribute

External Links Table

Includes the following data fields:

  • URL’s returned HTTP status code
  • Number of times that URL is linked to from within the site (click to see the list of affected URLs)
  • External URL used in the link
  • Link text used for the URL
  • Internal page URL on which the link was first found

Internal HTTP Code Errors Table

The Internal errors table gathers all of the pages returning HTTP code errors (4xx and 5xx level codes) in one place to help organize the effort to resolve the problems. It includes the following data fields:

  • URL’s returned HTTP status code
  • Number of times that URL is linked to from within the site (click to see the list of affected URLs)
  • Internal URL used in the link
  • Link text used for the URL
  • Internal page URL on which the link was first found

The Internal Errors table is a subset of the Internal Links table showing just those pages returning HTTP status code errors.

Internal HTTP Redirects Table

This table combines all of the pages returning HTTP redirects in one list, so you can easily review them. You should not have to rely on redirects internally; instead, you can fix the source code containing the redirected link. This table contains the following data fields:

  • URL’s returned HTTP status code
  • Number of times that URL is linked to from within the site (click to see the list of affected URLs)
  • Internal URL used in the link
  • Link text used for the URL
  • Redirect’s target URL
  • Internal page URL on which the link was first found

The Internal Redirects table is a subset of the Internal Links table, showing just those pages returning 301 and 302 HTTP status code redirects.

External HTTP Code Errors Table

The External errors table gathers all of the pages returning HTTP code errors (4xx and 5xx level codes) in one place to help organize the effort to resolve the problems. It includes the following data fields:

  • URL’s returned HTTP status code
  • Number of times that URL is linked to from within the site (click to see the list of affected URLs)
  • Internal URL used in the link
  • Link text used for the URL
  • Redirect’s target URL
  • Internal page URL on which the link was first found

The External Errors table is a subset of the External Links table showing just those pages returning HTTP status code errors.

External HTTP Redirects Table

The External Redirects table combines all of the pages returning HTTP redirects in one list so you can easily review them. As the redirect to the targeted page does not affect your page, fix these URLs is a lower priority. This table contains the following data fields:

  • URL’s returned HTTP status code
  • Number of times that URL is linked to from within the site (click to see the list of affected URLs)
  • External URL used in the link
  • Link text used for the URL
  • Redirect’s target URL
  • Internal page URL on which the link was first found

The External Redirects table is a subset of the External Links table showing just those pages returning 301 and 302 HTTP status code redirects.

(Idea and specs by Jim Boykin)


Feedback

 

Users comments:

Login or register to post comments and rate

4.6 (12 votes)

Jacob
2018-05-23
Pretty cool site. I'm glad I found this, because its fun to delve deeper into the sites I visit. I'd say there are some things like the comment area that are really poorly designed, but as a programmer myself I know CSS is a hassle, and seeing as thats my biggest complaint so far, I'd let it slide. Thanks for the services my guys.
Olivia Stephen
2018-05-17
Save Your Relationship and Get Your Ex Boyfriend/Girlfriend Back!contact: Unityspelltemple@gmail.com is certainly the best spell caster online and his result is 100% guarantee. My Name Olivia Stephen form Tx,USA. After 12years of marriage, me and my husband has been into one quarrel or the other until he finally left me and moved to California to be with another woman. I felt my life was over and my kids thought they would never see their father again. i tried to be strong just for the kids but i could not control the pains that torments my heart, my heart was filled with sorrows and pains because i was really in love with my husband. Every day and night i think of him and always wish he would come back to me, I was really upset and i needed help, so i searched for help online and I came across a website that suggested that Dr Unity can help get ex back fast. So, I felt I should give him a try. I contacted him and he told me what to do and i did it then he did a Love spell for me. 28 hours later, my husband really called me and told me that he miss me and the kids so much, So Amazing!! So that was how he came back that same day,with lots of love and joy,and he apologized for his mistake,and for the pain he caused me and the kids. Then from that day,our Marriage was now stronger than how it were before, All thanks to Dr Unity. he is so powerful and i decided to share my story on the internet that Dr.Unity real and powerful spell caster who i will always pray to live long to help his children in the time of trouble, if you are here and you need your Ex back or your husband moved to another woman, do not cry anymore, contact this powerful spell caster now. Here’s his contact Call/WhatsApp: +2348071622464 , Email: Unityspelltemple@gmail.com , Website:https://unityspelltemple126848969.wordpress.com .
Lyndon R
2018-04-18
Also having issues with the Upgrade comment request. All I see is Sending and the spinner.
Anonymous
2017-12-21
wow
info@skycareit.com chiman
2017-11-21
Great service so far just fount this site and has already helped . Still have to try out other tools and services .definitely get my recommendation !
David Bailey
2017-10-04
Upgrade comment request isn't working. Have tried several times now.
thomas perdriau
2017-09-11
How to upgrade to be able to crawl up to 1000 pages?
shine sun
2017-07-19
非常好用的網站爬取工具 ! 感謝。
senja kelabu
2017-07-13
“AGEN JUDI POKER”:https://pokergua.com
shelly henderson
2017-05-02
Great service so far just fount this site and has already helped . Still have to try out other tools and services .definitely get my recommendation !