---
title: Pressable sites and robots.txt
url: "https://pressable.com/knowledgebase/pressable-sites-and-robots-txt/"
published: 2021-05-19
modified: 2025-09-11
author: Wayne McWilliams
---

The `robots.txt` file gives you control over how automated bots and web crawlers access your site. With the exception of [staging sites](https://pressable.com/blog/how-to-create-a-staging-site-for-wordpress/) (which use a .mystagingwebsite.com address), most bots are allowed to crawl your site.

Adding rules to `robots.txt` allows you to limit or block specific bots from accessing all or part of your site.

## Default Staging Site robots.txt

By default, all `robots.txt` for [staging sites](/knowledgebase/how-sites-staging-websites-and-website-clones-work-at-pressable/) created at Pressable are “hidden” and prevent indexing by the search engine. This is generally a good thing, as you would not want clones of your live site being included in search results.

```
User-agent: *
Disallow: /
```

## Default Live Site robots.txt

For a site created as (or [converted to](https://pressable.com/knowledgebase/how-sites-staging-websites-and-website-clones-work-at-pressable/#converting-sites)) a **live site**, the default `robots.txt` file allows indexing:

```
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.phpSitemap: [URL]/wp-sitemap.xml
```

## Custom Staging Domain robots.txt

The **[Custom Staging Domain](https://pressable.com/knowledgebase/assign-a-custom-staging-domain-to-your-pressable-site/)** can be added to your Pressable Account within the Settings under the [Company](https://my.pressable.com/settings/company) section. When you enter the custom domain, there are two settings that you can adjust when creating the custom domain. Enabling “Existing Staging Sites” allows custom domains to be added to all existing staging sites.

Setting “`Overwrite robots.txt file`” allows the `robots.txt` file to be overwritten when a custom domain is created, ensuring all Staging sites will prevent search engines from indexing the site.

![](https://i0.wp.com/pressable.com/wp-content/uploads/2024/07/new-custom-domain.png?resize=1024%2C797&ssl=1)

## Cloned and Restored Site robots.txt

 Important:  If you [restore](https://pressable.com/knowledgebase/restore-a-site-in-pressable-using-the-automated-restore-tool) or [clone a site](https://pressable.com/knowledgebase/how-to-clone-website-to-staging/) that has a custom robots.txt (added manually or via plugin), the robots.txt file will NOT be changed for the restored or cloned site to reflect the current site’s environment (Staging or Live). You would need to manually adjust the robots.txt file on the restored or cloned site to reflect the site’s needed state.

## Accessing and Customizing robots.txt

If you need to override this functionality, you can do so by uploading your own custom robots.txt file to the root of your site [via SFTP](https://pressable.com/knowledgebase/setup-and-use-sftp-to-connect-to-your-wordpress-site/) or a [file manager plugin](https://pressable.com/knowledgebase/using-wp-file-manager-with-pressable/). When a custom `robots.txt` file exists, it takes precedence over the system-side one.

You can also customize `robots.txt` to disallow GPTBots and AI agents from ingesting your site content by following [this guide](https://pressable.com/knowledgebase/disallowing-gptbots-from-accessing-your-site-content/).