Bots..

[natselrox]

You see Google[bot] and so on.. What are these things? Automated data-collection programs for Google and others?

[naturetalk]

You see Google[bot] and so on.. What are these things? Automated data-collection programs for Google and others?

Yes, software that crawls the web and gathers data of various kinds. Simple bots can be written in just a few minutes.

[Darkchilde]

They are used by the search engines to gather data, about links, words etc.

[naturetalk]

Just to clarify a bit, anybody can use a bot.

As example, I wrote a bot that goes to the National Weather Service every day, grabs the local forecast for requested zip codes, and then mails the forecast to subscribers.

A bot can be really simple. For instance, go grab a remote web page, and save it to your server. That's a dozen lines of code, something like that.

And bots can be very sophisticated too. Imagine trying to keep track of every page on the web! Phew!

[natselrox]

Just to clarify a bit, anybody can use a bot.

As example, I wrote a bot that goes to the National Weather Service every day, grabs the local forecast for requested zip codes, and then mails the forecast to subscribers.

A bot can be really simple. For instance, go grab a remote web page, and save it to your server. That's a dozen lines of code, something like that.

And bots can be very sophisticated too. Imagine trying to keep track of every page on the web! Phew!

Funny to think if there were biological-equivalents of bots! A giant spider sends some small probes around to search for food and they report back to the spider! A colony of ants can be treated as a decentralised system that works on similar bot-based principle!

[naturetalk]

Funny to think if there were biological-equivalents of bots!

That may be coming. I may be happy that I may not be here to see it.

[rustyness]

They are used by the search engines to gather data, about links, words etc.

To elaborate, commonly they'll start on a web page, cache the page and save every link on that page. They then visit a random sample of the links to other web pages or web site. As they visit other pages, the process starts again and they cache the new pages. The bots 'spider' out across the web, caching many web sites.

This is why the more links to your web site, the more often a bot is likely to visit and cache your web site and hence the higher ranking the web site is on Google's search rankings. They also compare the links on the page to the page's content and the content of the page that the link points to so that they can check how relevant the link is. More relevant links will also rank higher in search engines. If, for example, you had 100 photography web sites linking to your photography web site, it would rank higher than another photography web site that had 5000 links from, say, estate agents and online cooking courses.

[Dudely]

They are used by the search engines to gather data, about links, words etc.

To elaborate, commonly they'll start on a web page, cache the page and save every link on that page. They then visit a random sample of the links to other web pages or web site. As they visit other pages, the process starts again and they cache the new pages. The bots 'spider' out across the web, caching many web sites.

This is why the more links to your web site, the more often a bot is likely to visit and cache your web site and hence the higher ranking the web site is on Google's search rankings. They also compare the links on the page to the page's content and the content of the page that the link points to so that they can check how relevant the link is. More relevant links will also rank higher in search engines. If, for example, you had 100 photography web sites linking to your photography web site, it would rank higher than another photography web site that had 5000 links from, say, estate agents and online cooking courses.

To add further: this is why Wikipedia seems to always be close to the top. People link to it when actually talking about the subject involved and it sometimes is linked hundreds of times on other Wikipedia pages as well. This also makes porn hard to rank up nefariously.

[rustyness]

They are used by the search engines to gather data, about links, words etc.

To elaborate, commonly they'll start on a web page, cache the page and save every link on that page. They then visit a random sample of the links to other web pages or web site. As they visit other pages, the process starts again and they cache the new pages. The bots 'spider' out across the web, caching many web sites.

This is why the more links to your web site, the more often a bot is likely to visit and cache your web site and hence the higher ranking the web site is on Google's search rankings. They also compare the links on the page to the page's content and the content of the page that the link points to so that they can check how relevant the link is. More relevant links will also rank higher in search engines. If, for example, you had 100 photography web sites linking to your photography web site, it would rank higher than another photography web site that had 5000 links from, say, estate agents and online cooking courses.

To add further: this is why Wikipedia seems to always be close to the top. People link to it when actually talking about the subject involved and it sometimes is linked hundreds of times on other Wikipedia pages as well. This also makes porn hard to rank up nefariously.

Yeah, I bet this web site has a few hundred links to Wikipedia, and most will be to Wikipedia articles relating to philosophy, atheism, agnosticism, science or religion, the main aspects of this web site.

[natselrox]

To add further: this is why Wikipedia seems to always be close to the top. People link to it when actually talking about the subject involved and it sometimes is linked hundreds of times on other Wikipedia pages as well. This also makes porn hard to rank up nefariously.

I thought they made Wikipedia kinda sticky!

[BeAfraid]

Funny to think if there were biological-equivalents of bots!

That may be coming. I may be happy that I may not be here to see it.

Why does it not surprise me that Naturetalk was/is a luddite?

Matthew Bailey

[natselrox]

Funny to think if there were biological-equivalents of bots!

That may be coming. I may be happy that I may not be here to see it.

Why does it not surprise me that Naturetalk was/is a luddite?

Matthew Bailey

What did he mean by that? I have no clue! naturetalk was really strange!