Package com.viglet.turing.connector.webcrawler

Class TurWCProcess

Object

TurWCProcess

@Component
public class TurWCProcess
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
static final String	A_HREF
static final String	ABS_HREF
static final String	ID_ATTR
static final String	JAVASCRIPT
static final String	MAILTO
static final String	TEL

Constructor Summary

Constructors

Constructor	Description
TurWCProcess(String turingUrl, String turingApiKey, int timeout, int jobSize, String referrer, TurWCAllowUrlRepository turWCAllowUrlRepository, TurWCNotAllowUrlRepository turWCNotAllowUrlRepository, TurWCFileExtensionRepository turWCFileExtensionRepository, TurWCAttributeMappingRepository turWCAttributeMappingRepository)

Method Summary

Modifier and Type	Method	Description
Map<String,Object>	getJobItemAttributes(TurWCSource turWCSource, Document document, String url)
Locale	getLocale(TurWCSource turWCSource, Document document, String url)
TurSNJobItem	getPage(TurWCSource turWCSource, String url)
void	getPagesFromQueue(TurWCSource turWCSource)
TurWCContext	getTurWCContext(Document document, String url)
void	start(TurWCSource turWCSource)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

ID_ATTR

public static final String ID_ATTR

See Also:

• Constant Field Values

MAILTO

public static final String MAILTO

See Also:

• Constant Field Values

TEL

public static final String TEL

See Also:

• Constant Field Values

JAVASCRIPT

public static final String JAVASCRIPT

See Also:

• Constant Field Values

A_HREF

public static final String A_HREF

See Also:

• Constant Field Values

ABS_HREF

public static final String ABS_HREF

See Also:

• Constant Field Values

Constructor Details

TurWCProcess

@Inject
public TurWCProcess(@Value("${turing.url}")
 String turingUrl,
 @Value("${turing.apiKey}")
 String turingApiKey,
 @Value("${turing.wc.timeout:5000}")
 int timeout,
 @Value("${turing.wc.job.size:50}")
 int jobSize,
 @Value("${turing.wc.referrer:https://www.google.com}")
 String referrer,
 TurWCAllowUrlRepository turWCAllowUrlRepository,
 TurWCNotAllowUrlRepository turWCNotAllowUrlRepository,
 TurWCFileExtensionRepository turWCFileExtensionRepository,
 TurWCAttributeMappingRepository turWCAttributeMappingRepository)

Method Details

start

public void start(TurWCSource turWCSource)

getPagesFromQueue

public void getPagesFromQueue(TurWCSource turWCSource)

getPage

public TurSNJobItem getPage(TurWCSource turWCSource,
 String url)

getJobItemAttributes

public Map<String,Object> getJobItemAttributes(TurWCSource turWCSource,
 Document document,
 String url)

getLocale

public Locale getLocale(TurWCSource turWCSource,
 Document document,
 String url)

getTurWCContext

public TurWCContext getTurWCContext(Document document,
 String url)