Dynamic robots.txt with Zend Framework

Here’s a quick little tutorial to show you how you can create a dynamic robots.txt file with the Zend Framework.

My robot files aren’t usually complex since I’m really careful about not putting anything that could be indexed in the web root, but there are some cases where you may want to have more control over the robots file at the application level. In the example below, we’ll be serving up different robots content depending on if the file is accessed from HTTPS or not.

You may also want to put your robot settings in your application ini file or control what gets indexed at the Controller level where the other business for your application logic lives.

Step 1 – Create a custom route in your bootstrap

This route will catch requests from robots.txt and route them to the robots action of the RobotsController.

; robots.txt
routes.robots.route = "/robots.txt"
routes.robots.defaults.module = default
routes.robots.defaults.controller = "robots"
routes.robots.defaults.action = "robots"

Step 2 – Create the robots.txt Controller

I opted to not use the default indexAction for my robots.txt action. This would to try and avoid any chance of Google indexing additional versions of the robots controller/action urls. In the controller below, /robots redirects to /robots.txt. It still leaves /robots/robots open, but that’s less likely to get hit that just /robots.

Notice that I’ve also set the content-type to text/plain which is what it would have been had this been an actual text file. I’m not sure it makes any difference to Google, but this will make the page resemble as close as possible an actual robots.txt file.


class Default_RobotsController extends Zend_Controller_Action
{

    public function indexAction() 
    {
            $this->_redirect("/robots.txt", array('code' => 301));
    }

        public function robotsAction()
        {
            $this->_helper->getHelper('layout')->disableLayout();
            $this->getResponse()
                    ->setHeader('Content-type', 'text/plain');

            $this->view->sslEnabled = false;

            if(isset($_SERVER['HTTPS']))
                $this->view->sslEnabled = true;
        }

}

Step 3 – Create the robots.txt view

In the controller above, I assigned a boolean to the view so that I could render a different robots.txt if the page is accessed via SSL. In this case, I want to completely remove the HTTPS site from Google’s index to avoid duplicate content issues.

<?php if($this->sslEnabled):?>
User-agent: *
Disallow: /
<?php else:?>
User-agent: *
Allow: /
<?php endif;?>

Sitemap: http://www.mysite.com/sitemap.xml

Fonte adaptada: http://joshuamcginnis.com/2010/11/29/dynamic-robots-txt-with-zend-framework/

Deixe um comentário

Preencha os seus dados abaixo ou clique em um ícone para log in:

Logotipo do WordPress.com

Você está comentando utilizando sua conta WordPress.com. Sair / Alterar )

Imagem do Twitter

Você está comentando utilizando sua conta Twitter. Sair / Alterar )

Foto do Facebook

Você está comentando utilizando sua conta Facebook. Sair / Alterar )

Foto do Google+

Você está comentando utilizando sua conta Google+. Sair / Alterar )

Conectando a %s