julieandrews (
julieandrews) wrote2009-01-28 06:05 am
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
Entry tags:
Open Question Wednesday #23
I'm lying in bed thinking, instead of sleeping. Grr. And I got to wondering. So since it also happens to be Wednesday, maybe you all can stop me from wondering. :)
Do online publications, such as e-magazines, generally allow robots to crawl their site and index and cache their stories?
Do online publications, such as e-magazines, generally allow robots to crawl their site and index and cache their stories?
no subject
(Anonymous) 2009-01-28 12:12 pm (UTC)(link)no subject
I was just wondering if the stories would hang around in Google caches and the Wayback Machine, etc even after they were taken down from the site. Or even after the publication folded.
no subject
(Anonymous) 2009-01-28 09:36 pm (UTC)(link)I think (recalling an AbsoluteWrite glitch when stuff that was supposed to be protected, wasn't) it takes about 2 weeks or so to clean out. I don't know if that's absolute or statistical, though.
- Keyan