us.codecraft.webmagic.scheduler

Class QueueScheduler

java.lang.Object

us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

us.codecraft.webmagic.scheduler.QueueScheduler

All Implemented Interfaces:

MonitorableScheduler, Scheduler

public class QueueScheduler
extends DuplicateRemovedScheduler
implements MonitorableScheduler

Basic Scheduler implementation.
Store urls to fetch in LinkedBlockingQueue and remove duplicate urls by HashMap.

Since:

0.1.0

Author:

code4crafter@gmail.com

Field Summary

Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

logger

Constructor Summary

Constructors

Constructor and Description
QueueScheduler()

Method Summary

Modifier and Type	Method and Description
int	getLeftRequestsCount(Task task)
int	getTotalRequestsCount(Task task)
Request	poll(Task task) get an url to crawl
void	pushWhenNoDuplicate(Request request, Task task)

Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface us.codecraft.webmagic.scheduler.Scheduler

push

Constructor Detail

QueueScheduler

public QueueScheduler()

Method Detail

pushWhenNoDuplicate

public void pushWhenNoDuplicate(Request request,
                                Task task)

Overrides:

pushWhenNoDuplicate in class DuplicateRemovedScheduler

poll

public Request poll(Task task)

Description copied from interface: Scheduler

get an url to crawl

Specified by:

poll in interface Scheduler

Parameters:

task - the task of spider

Returns:

the url to crawl

getLeftRequestsCount

public int getLeftRequestsCount(Task task)

Specified by:

getLeftRequestsCount in interface MonitorableScheduler

getTotalRequestsCount

public int getTotalRequestsCount(Task task)

Specified by:

getTotalRequestsCount in interface MonitorableScheduler