README

Download | Packagist | Issues

Sitemap plugin for Textpattern CMS. Generates Sitemaps.org XML sitemaps for your site, which help Google and other search engines to index your content. Maps your categories, sections, articles and even custom URLs of your choosing. No diving into code required, all configuration is done from a graphical user interface.

Install

Using Composer:

$ composer require rah/rah_sitemap

Or download an installer package.

Basics

Rah_sitemap generates a sitemap for your Textpattern website, listing all of its section-, article category- and article-pages. The generated sitemap follows the XML based Sitemap protocol format and is targeted to search engines, opposed to your visitors. The sitemap is meant to help search engines to index your site as it grows and gets more and more various nested pages.

The Sitemap can be configured directly from Textpattern’s Preferences panel, making rah_sitemap easy to setup and use.

Accessing the sitemap

The generated sitemap becomes publicly accessible from the site’s root. The sitemap can be accessed from two URLs, depending on the site’s permanent link mode. If the site is configured to use clean URLs, the sitemap can be accessed using a clean path like http://example.com/sitemap.xml, where the example.com would be the site’s URL. Additionally a query string version, http://example.com/?rah_sitemap=sitemap, is available in both the messy and the clean URL modes, and can be used if the other one isn’t available.

Immediate boost to search engine visiblity?

Not exactly. Providing Sitemap is to help crawling, finding your site’s pages that could otherwise be hard to discover. Sitemaps itself do not boost content’s visibility.

For a simple, almost a static website with just few easily discoverable pages, a sitemap may not be necessary at all. If search engines already can get to your pages, you do not really need rah_sitemap, or Sitemaps in general. Once a page is indexed, that’s where sitemap’s work ends.

Rah_sitemap is particularly useful when your site has pages that are hard to discover due to being loaded using JavaScript, there is distinct content with little linking, or pages are nested deep in a complex page structures. Rah_sitemap can also help your site to start up when it has very few external links pointing to its pages.

It’s common misconception to think that Sitemaps guarantee that pages will be indexed. This is not the case. Sitemap is a map. Whether a place is marked on a map doesn’t mean someone will actually go there, or when. Like any map, the map is used to find and learn, to increase the future knowledge. Sitemap’s update interval also helps to estimate when your site is updated next and when it should be crawled again.

Normally, you will benefit from submitting a sitemap, but its just one piece in the puzzle. In no case will you get penalized from Sitemap or including wrong content in it.

Configuration

After rah_sitemap is installed, you may want to configure it to fit your site. For instance, you may want to exclude certain irrelevant articles or sections. The plugin’s settings can be configured from Textpattern’s Preferences panel, organized under its own Sitemap section. Sections and Categories can be excluded from the sitemap from their respective editors.

Sending the sitemap to search engines

Once you have a sitemap up and running, you may want to inform search engines about its existence. There are few ways you can do it: Google Search Central for Google, you could use a robots.txt directive or search vendor specific pinging.

The recommended way is by using a robots.txt file. To get robots.txt up and running, you will have to add robots.txt file at root of your site’s domain, so that its accessible from https://example.com/robots.txt. If your Textpattern site has fully functional clean URLs, is installed at the root and you already do not have robots.txt file, rah_sitemap will automatically create the file for you — or well, serve it dynamically.

If not, you will need to create or edit a file named robots.txt at the root of the domain. In that file you would add a Sitemap directive containing an absolute URL to your sitemap:

Sitemap: https://example.com/?rah_sitemap=sitemap

Where the https://example.com/ is your site’s location as defined in Textpattern’s Preferences panel. The directive should be placed on its own line.

Preferences

Rah_sitemap comes with number of preferences which all can be found from your Preferences panel, organized under a Sitemap section. Rah_sitemap allows excluding sections, categories and articles from the XML sitemap. Following options will be present.

Exclude articles based on fields

The field can be used to exclude articles from the sitemap based on any article field and its value. The option takes a comma-separated list of articlefield: value pairs, where the field is the database field and the value is the field’s value that will be excluded. Available fields include Title, AuthorID, Body, Excerpt, Category1, Category2, Section, Keywords, url_title, custom_1 to custom_10 and Image.

Values used in the option support two wildcard characters. An underscore (_) matches exactly one character, and a percent sign (%) matches zero or more characters.

If you wanted to exclude articles posted to sections named as notes and private or by a user mailer, you could use the following in the field:

Section: notes, Section: private, AuthorID: mailer

Additional URLs

Comma-separated list of additional local site URLs added to the sitemap. Note that a Sitemap only allows local URLs, meaning that any URL used, needs to link to the same domain as where the website itself is located. If a URL is relative and doesn’t start with a HTTP or HTTPS protocol, the site’s URL is prepended to the URL.

Include future articles?

If set to Yes, articles with future publishing date are visible in the sitemap. Please note that by default the article tag doesn’t display future articles, unless its time attribute is explicitly set to future or any.

Include published articles?

If set to Yes, published articles are visible in the sitemap. If both this option and Include future articles? are set to No, no articles will be visible in the sitemap.

Include expired articles?

If set to No, expired articles are not visible in the sitemap.

Exclude sticky articles?

If set to Yes, sticky articles are not visible in the sitemap.

For developers

Rah_sitemap offers small set of tools for developers. These tools allow other Textpattern plugins to extend rah_sitemap’s functionality by adding new URLs to the sitemap. The plugin is packaged in a class structure that can be extended if needed, and introduces new Textpattern callback events.

Callback

Rah_sitemap introduces a new public-facing callback event to the Textpattern’s event library named rah_sitemap.urlset. The event is fired before a sitemap is printed out. The callback event can be used with the API to add new URLs to the sitemap.

As with other callback events in Textpattern, hooking to rah_sitemap’s event happens using Textpattern’s callback handling functions, mainly register_callback.

register_callback('abc_function', 'rah_sitemap.urlset', 0, $urls);

fuction abc_function($event, $step, &$urls) {
    $urls['https://example.com/foo/bar'] = 123456789;
    $urls['https://example.com/second'] = null;
}

The above would add https://example.com/foo/bar to the sitemap with UNIX modification timestamp of 123456789, and https://example.com/second with no modification time.

Custom URL functions

If you are supplying a custom URL function for Textpattern, please note that the URLs the function generates need to meet RFC 3986 and RFC 3987. All URLs should also be entity escaped from special syntax characters using Textpattern’s txpspecialchars function. All URLs Textpattern itself generates follow these specifications, and so should your custom URL plugin.

As rah_sitemap integrates well with Textpattern’s core, it uses the same URL functions as Textpattern. If an URL given to the sitemap doesn’t meet those specification, the sitemap will become invalid.

Changelog

Version 4.0.2 – 2023/08/30

Fixed: “Exclude articles based on fields” setting, caused by regression in version 3.0.0. Articles can once again be excluded from the feed using field-value filters.

Version 4.0.1 – 2023/02/18

Fixed: issues with lastmod date generation caused by Textpattern 4.8.8 core issues. Date format string would be passed down to the wrong date function on certain host system, based on it’s supported features. Mitigated the issue by using PHP’s date function directly instead of Textpattern’s safe_strftime.

Version 4.0.0 – 2022/09/25

Fixed: invalid lastmod date format caused by regression in Textpattern 4.8.8.
Now requires Textpattern >= 4.8.8.

Version 3.0.0 – 2022/04/23

Can now handle large websites with hundreds of thousands of articles. Instead of one large sitemap being generated, it now generates a sitemap index, which links to smaller split sitemaps. This reduces sitemap generation memory usage.
Now requires PHP >= 7.4.0.

Version 2.1.0 – 2020/01/01

Added: Global options to exclude categories from the sitemap by type. Thank you, Sebastian Spautz.
Added: German translation. Thank you, Sebastian Spautz.

Version 2.0.2 – 2020/01/01

Fixed: yes-no toggle default-selection rendering. Thank you, Sebastian Spautz.
Fixed: expired article exclusion. Thank you, Sebastian Spautz.

Version 2.0.1 – 2019/11/01

Fixed: option to exclude expired articles.

Version 2.0.0 – 2019/04/20

Fixed: Generates valid /year/month/day/title permanent links. Thank you, Wladimir Palant.
Drop old legacy preference migration code.
Use preference API to create preference options.
Now requires Textpattern 4.7.0 or newer.
Now requires PHP 7.2.0 or newer.

Version 1.3.0 – 2014/03/28

Changed: Integrated preferences to Textpattern’s native preferences panel, and to Section and Category editors.
Added: Language strings, interface is now translatable using Textpacks.
Added: Finnish translation.
Added: French translation by Patrick Lefevre.
Added: Chinese translation by WizJin.
Improved: Compatibility with Textpattern 4.5.0.
Now requires Textpattern 4.5.0 or newer.

Version 1.2 – 2011/03/09

Added: adds site URL to relative article permlinks. Basically a fix for gbp_permanent_links.
Changed: from permlinkurl_id() to permlinkurl(). Greatly reduced the amount of queries generating article permlinks makes.

Version 1.1 – 2010/10/30

Fixed issues appearing with the installer when MySQL is in strict mode. Thank you for reporting, Gallex.

Version 1.0 – 2010/10/29

Slightly changed backend’s installer call; only check for installing if there is no preferences available.

Version 0.9 – 2010/08/25

Fixed: now correctly parses category tags in category URLs. Thank you for reporting), Andreas.

Version 0.8 – 2010/07/27

Now compression level field’s label now links to the correct field id.
Now suppresses E_WARNING/E_STRICT notices in live mode caused by Textpattern’s timezone code when some conditions are met (TXP 4.2.0, PHP 5.1.0+, TXP’s Auto-DST feature disabled, TXP in Live mode). Error suppression will be removed when TXP version is released with fully working timezone settings.
Now generates UNIX timestamps within the SQL query, not with PHP.
Changed sliding panels’ links (a elements) into spans.

Version 0.7 – 2010/05/30

Fixed: now deleting custom url leads back to the list view, not to the editing form.
Removed some leftover inline styles from v0.6.

Version 0.6 – 2010/05/30

Rewritten the code that generates the sitemap.
New admin panel look.
Now custom permlink modes and custom urls are escaped. Users can input unescaped URLs/markup from now on.
Now custom URL list shows the full formatted URL after auto-fill instead of the user input.
Now custom URLs that start with www. are completed with http:// protocol.
Now all urls that do not start with either http, https, www, ftp or ftps protocol are auto-completed with the site’s address.
Custom url editor got own panel. No longer the form is above the URL list.
Added ability to manually turn gzib compression off and change the compression level.
Added setting to set zlib.output_compression off. See here, thank you for reporting superfly.
Preferences are now trimmed during save.
Merged rah_sitemap_update() with rah_sitemap_save().
From now on all new installations have default settings defined that will automatically exclude link, file and image categories from the sitemap. This won’t effect updaters.
Changed sitemap’s callback register from pre pretext to callback after it (callback is now textpattern. Now $pretext is set before the sitemap and thus more plugins might work within permlink settings and custom urls.
When using Textpattern’s clean URLs, requesting /sitemap.xml.gz and /sitemap.xml URLs will return the sitemap, not just the /?rah_sitemap=sitemap. This will of course require existing fully working clean urls.

Version 0.5 – 2010/03/01

Added customizable timestamp formats.
Cleaned backend markup.
Combined individual preference queries.

Version 0.4 – 2009/04/12

Added support for custom permlink rules: Now you can easily set any kind of permlink rules for articles, section and categories.
Added option to exclude future articles.
Added option to exclude past articles.
Added option to exclude expired articles.
Moved Custom URL UI to it’s own page.
Added multi-delete feature to Custom URL UI.
Improved Custom URL UI.
Removed default static appending domain from Custom URL input field.
Changed TXP minimum requirement to version 4.0.7 (and above. Note that the plugin still works with older TXP versions (down to 4.0.5) if the Exclude Expired articles -option is left empty (unset.

Version 0.3.2 – 2008/10/25

Fixed view url that still (from version 0.2) included installation address before link.

Version 0.3 – 2008/10/24

Added option to insert URLs that are outside Textpattern install directory.
Fixed option to exclude categories directly by type: added forgotten link type.

Version 0.2 – 2008/10/22

Added option to exclude/include sticky articles.
Added option to exclude categories directly by type.
Fixed bug: now shows all categories, and not only article-type, in admin panel.
Fixed bug: removed double install query (didn’t do a thing, just checked table status twice.

Version 0.1.2 – 2008/09/12

Fixed article listing bug caused by nasty little typo: now only 4 and 5 statuses are listed.

Version 0.1 – 2008/09/07

First release.

rah / rah_sitemap

Maintainers

Details