robots.txt and sitemap.xml on Google App Engine

I added a robots.txt and sitemap.xml to my Google App Engine site (http://twix-mit.appspot.com). Due to this change, I also added "when" pages. From when pages, I added better HTTP caching given a cron cycle.

robots.txt

Source code:

• https://github.com/twixmit/twixmit/blob/be5b2ac12c94e10ceea9a3360e1717d6fadf00ba/static/robots.txt

Handler:

• https://github.com/twixmit/twixmit/blob/be5b2ac12c94e10ceea9a3360e1717d6fadf00ba/app.yaml#L29

Runtime: http://twix-mit.appspot.com/robots.txt

sitemap.xml

Source code:

• https://github.com/twixmit/twixmit/blob/master/newmedigester.py#L553

Handler:

• https://github.com/twixmit/twixmit/blob/be5b2ac12c94e10ceea9a3360e1717d6fadf00ba/app.yaml#L37

Runtime: http://twix-mit.appspot.com/sitemap.xml

"when" pages

You might note in the sitemap, all the pages are driven by date. I've used some of the nice time delta modules in python.

Source code:

• https://github.com/twixmit/twixmit/blob/fb0c7973544b093ba65aefde4122c6fe48629dbc/helpers.py#L63

This allows getting the current time, doing delas to tomorrow, yesterday, etc.

"when" caching

Per the "when" stuff, the http response headers are now timed given the cron cycle for the news digestion. The memcache time is also driven by this calculation as well.

Source code:

• https://github.com/twixmit/twixmit/blob/3223be3ac2b0ebf255fdd7d44227f8a8046879c5/newmedigester.py#L532
• https://github.com/twixmit/twixmit/blob/3223be3ac2b0ebf255fdd7d44227f8a8046879c5/newmedigester.py#L465