Saturday, September 22, 2012

robots.txt and sitemap.xml on Google App Engine

I added a robots.txt and sitemap.xml to my Google App Engine site (http://twix-mit.appspot.com). Due to this change, I also added "when" pages. From when pages, I added better HTTP caching given a cron cycle.

robots.txt

Source code:
Handler:
Runtime: http://twix-mit.appspot.com/robots.txt

sitemap.xml

Source code:
Handler:
Runtime: http://twix-mit.appspot.com/sitemap.xml

"when" pages

You might note in the sitemap, all the pages are driven by date. I've used some of the nice time delta modules in python.

Source code:
This allows getting the current time, doing delas to tomorrow, yesterday, etc.

"when" caching

Per the "when" stuff, the http response headers are now timed given the cron cycle for the news digestion. The memcache time is also driven by this calculation as well.

Source code:




Share on Twitter