Open Question Wednesday #23
Jan. 28th, 2009 06:05 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
julieandrewsI'm lying in bed thinking, instead of sleeping. Grr. And I got to wondering. So since it also happens to be Wednesday, maybe you all can stop me from wondering. :)
Do online publications, such as e-magazines, generally allow robots to crawl their site and index and cache their stories?
Do online publications, such as e-magazines, generally allow robots to crawl their site and index and cache their stories?
(no subject)
Date: 2009-01-28 09:36 pm (UTC)I think (recalling an AbsoluteWrite glitch when stuff that was supposed to be protected, wasn't) it takes about 2 weeks or so to clean out. I don't know if that's absolute or statistical, though.
- Keyan