Programming Spiders, Bots, and Aggregators in Java - Softcover

Heaton, Jeff

3.47

17 ratings by Goodreads

9780782140408: Programming Spiders, Bots, and Aggregators in Java

Softcover

ISBN 10: 0782140408 ISBN 13: 9780782140408

Publisher: Sybex, 2002

View all copies of this ISBN edition:

The content and services available on the web continue to be accessed mostly through direct human control. But this is changing. Increasingly, users rely on automated agents that save them time and effort by programmatically retrieving content, performing complex interactions, and aggregating data from diverse sources. Programming Spiders, Bots, and Aggregators in Java teaches you how to build and deploy a wide variety of these agents–from single–purpose bots to exploratory spiders to aggregators that present a unified view of information from multiple user accounts.

You will quickly build on your basic knowledge of Java to quickly master the techniques that are essential to this specialized world of programming, including parsing HTML, interpreting data, working with cookies, reading and writing XML, and managing high–volume workloads. You′ll also learn about the ethical issues associated with bot use––and the limitations imposed by some websites.

This book offers two levels of instruction, both of which are focused on the library of routines provided on the companion CD. If your main concern is adding ready–made functionality to an application, you′ll achieve your goals quickly thanks to step–by–step instructions and sample programs that illustrate effective implementations. If you′re interested in the technologies underlying these routines, you′ll find in–depth explanations of how they work and the techniques required for customization.

"synopsis" may belong to another edition of this title.

Review:

If you want to surf the Web programmatically, rather than by firing up your favourite browser, then Programming Spiders, Bots and Aggregators in Java is the book for you. Spiders are programs that crawl the Web following hyperlinks, and are used to populate databases for search engines. Aggregators pull together data from several different sites, for example to present and compare prices from a number of online retailers, while "Bot" is a generic term covering any program that pulls data from the Internet.

Readers are expected to have some knowledge of Java, but the author's patient step-by-step approach makes this title accessible to developers of every skill level. He begins by explaining the basics of Internet programming, showing how to send and receive data using sockets, and how to parse HTML to get useful results. Bots need to know how to fill in forms, accept cookies and connect over HTTPS, to mention three of the topics covered here. The book continues with specifics such as how to construct a spider, how to build a bot for multiple sites within a certain category and how to get an aggregated view of many Web sites.

Towards the end, the book touches on SOAP, a protocol for Web services that in theory could make bots redundant. It will be a long time, if ever, before enough sites offer SOAP services to make that happen. In the meantime, bots are essential tools for exploiting Web resources, and this thorough and well-written title is an excellent place to learn how to use them. The bundled CD includes a bot package, which you can incorporate into your own applications, along with Java tools and resources. --Tim Anderson

Synopsis:

Spiders, bots, and aggregators are all so-called intelligent agents, which execute tasks on the Web without the intervention of a human being. Spiders go out on the Web and identify multiple sites with information on a chosen topic and retrieve the information. Bots find information within one site by cataloging and retrieving it. Aggregrators gather data from multiple sites and consolidate it on one page, such as credit card, bank account, and investment account data. As the Web grows more complex, there will be more and more applications of intelligent agents; Java is expected to be one of the principal languages used to build these agents.

"About this title" may belong to another edition of this title.