Interface IContext

Hierarchy

IContext

Properties

counter

You can have customized counters that collect with each task. You can have up to 100 counters for each task. Counters are not shared between tasks and task retries.

retryContext

retryContext: IRetryContext

task

task: ITask

Methods

addToDom

addToDom(id: string, data: any, options?: IAddToDomOptions): Promise<void>
Adds a div to the dom with data provided by the user.
Parameters
- id: string
- data: any
- Optional options: IAddToDomOptions
Returns Promise<void>

captureRequests

captureRequests(): Promise<void>
Function that starts capturing Network Requests happening in the background while the browser is loading the page. This function is required to be called if you want to use searchForRequest function.

Returns Promise<void>

checkBlocked

checkBlocked(statusCode?: number): Promise<boolean>
Checks to see if current page is getting blocked. Uses internal block detection and can be called at anytime
Parameters
- Optional statusCode: number
Returns Promise<boolean>

click

click(selector: string, options?: {}): Promise<void>
This method fetches an element with selector, scrolls it into view if needed, and then uses page.mouse to click in the center of the element. If there's no element matching selector, the method throws an error.

Bear in mind that if click() triggers a navigation event and there's a separate page.waitForNavigation() promise to be resolved, you may end up with a race condition that yields unexpected results. The correct pattern for click and wait for navigation is the following:
```
const [response] = await Promise.all([
  context.waitForNavigation(waitOptions),
  context.click(selector, clickOptions),
]);
```
Parameters
- selector: string
- Optional options: {}
Returns Promise<void>

clickAndWaitForNavigation

clickAndWaitForNavigation(selector: string, clickOptions?: object, navigationOptions?: IWaitOptions): Promise<void>
Parameters
- selector: string
- Optional clickOptions: object
- Optional navigationOptions: IWaitOptions
Returns Promise<void>

content

content(): Promise<string>
Gets the full HTML contents of the page, may include the doctype.

Returns Promise<string>

cookies

cookies(): Promise<ICookie[]>
Returns all cookies the current page

Returns Promise<ICookie[]>

enableNetworkDebugger

enableNetworkDebugger(): Promise<void>
Enable network debugger

Returns Promise<void>

evaluate

evaluate(pageFunction: (() => any), ...args: Json[]): Promise<Json>
Executes a function in the website's browser context If the function passed to evaluate returns a Promise, then evaluate would wait for the promise to resolve and return its value.
Parameters
- pageFunction: (() => any)
  - - (): any
    - Returns any
- Rest ...args: Json[]
Returns Promise<Json>

evaluateInFrame

evaluateInFrame(context: string, pageFunction: (() => any), ...args: Json[]): Promise<void | Json>
Executes a function in the specified context
Parameters
- context: string
- pageFunction: (() => any)
  - - (): any
    - Returns any
- Rest ...args: Json[]
Returns Promise<void | Json>

extract

extract(id: string, mergeOptions?: IMergeOptions): Promise<any>
Carry out an extraction, and merge with the data that should be returned
Parameters
- id: string
- Optional mergeOptions: IMergeOptions
Returns Promise<any>

generate2FAToken

generate2FAToken(secret: string): string
Function that generates time based Two-Factor Authentication tokens. It uses the secret code as input, to generate those tokens. This code is usually available in form of a QR code * when enabling the two-factor auth on a given website. This can be used to bypass two factor auth on websites.
Parameters
- secret: string
Returns string

goto

goto(url: string, options?: IGoToOptions): Promise<IResponse>
Load a URL
Parameters
- url: string
- Optional options: IGoToOptions
Returns Promise<IResponse>

halt

halt(ignoreData: boolean): void
Immediately throws a caught exception to stop processing without it being an extraction error.
Parameters
- ignoreData: boolean
Returns void

reload

reload(): Promise<void>
Refresh the page

Returns Promise<void>

reportBlocked

reportBlocked(code: number, details?: string): Promise<any>
Report if the extractor got blocked on the target site. When the target site starts blocking connections, or throwing captchas, and there is no way to bypass it, extractor should report that it got blocked. That way the system will know that the proxy used in this connection should be taken out of rotation for the particular domain, and that this extractor should get a new proxy information on the next attempt.
Parameters
- code: number
- Optional details: string
Returns Promise<any>

reportWrongGeocoding

reportWrongGeocoding(): Promise<any>
Report that the site is reporting a different country from expected

Returns Promise<any>

saveJson

saveJson(id: string, data: any): Promise<void>
Adds json in a script tag, with data provided by the user.
Parameters
- id: string
- data: any
Returns Promise<void>

screenshot

screenshot(options?: {
fullPage?: boolean;
type?: ScreenshotType;
}): Promise<undefined | string>
Take a screenshot.

Use this sparingly, or consdier if you need it at all.
Parameters
- Optional options: {
  fullPage?: boolean;
  type?: ScreenshotType;
  }
  - Optional fullPage?: boolean
  - Optional type?: ScreenshotType
Returns Promise<undefined | string>

scrollToBottom

scrollToBottom(options?: IScrollOptions): Promise<void>
Scrolls page to bottom. Can be used for infinite scrolling pages.
Parameters
- Optional options: IScrollOptions
Returns Promise<void>

searchAllRequests

searchAllRequests(urlPattern: string, method: string, pastTimestamp: number): Promise<any>
searchAllRequests will return all matching requests, regardless whether it finished or not.
Parameters
- urlPattern: string
- method: string
- pastTimestamp: number
Returns Promise<any>

searchForRequest

searchForRequest(urlPattern: string, method: string, pastTimestamp: number, timeout: number): Promise<any>
Searches for a Network Request that was performed on the website, and allows gathering details of that request. Requires captureRequests to be called or used during goto
Parameters
- urlPattern: string
- method: string
- pastTimestamp: number
- timeout: number
Returns Promise<any>

select

select(selector: string, ...values: string[]): Promise<void>
Set select element values
Parameters
- selector: string
- Rest ...values: string[]
Returns Promise<void>

setAntiFingerprint

setAntiFingerprint(antiFingerPrint: boolean): void
Mocks browser APIs commonly used to fingerprint a user. Defaults to true. Must be called before goto.
Parameters
- antiFingerPrint: boolean
Returns void

setBlockAds

setBlockAds(enabled: boolean): void
Block advertisments and trackers

Must be set before calling goto.
Parameters
- enabled: boolean
Returns void

setBypassCSP

setBypassCSP(enabled: boolean): void
Must be set before calling goto
Parameters
- enabled: boolean
Returns void

setCssEnabled

setCssEnabled(enabled: boolean): void
Must be set before calling goto
Parameters
- enabled: boolean
Returns void

setDownloadLimit

setDownloadLimit(downloadLimit: number): void
Sets size limit for file downloads, default is 150e6 or 150Mb
Parameters
- downloadLimit: number
Returns void

setDownloadRetries

setDownloadRetries(downloadRetries: number): void
Sets retry attempts for file downloads, default is 3
Parameters
- downloadRetries: number
Returns void

setDownloadTimeout

setDownloadTimeout(downloadTimeout: number): void
Sets timeout for file downloads, default is 240e3 or 4 minutes
Parameters
- downloadTimeout: number
Returns void

setExtraHTTPHeaders

setExtraHTTPHeaders(headers: Record<string, string>): void
Must be set before calling goto
Parameters
- headers: Record<string, string>
Returns void

setFirstRequestTimeout

setFirstRequestTimeout(firstRequestTimeout: number): void
Timeout between start to 'did-navigate'. Defaults to 20 seconds, but can be extended. This method must be called before goto
Parameters
- firstRequestTimeout: number
Returns void

setInputValue

setInputValue(selector: string, value: string): Promise<void>
Set the value of an input element
Parameters
- selector: string
- value: string
Returns Promise<void>

setJavaScriptEnabled

setJavaScriptEnabled(enabled: boolean): void
Must be set before calling goto
Parameters
- enabled: boolean
Returns void

setLoadAllResources

setLoadAllResources(enabled: boolean): void
Must be set before calling goto. Default false.
Parameters
- enabled: boolean
Returns void

setLoadImages

setLoadImages(enabled: boolean): void
Must be set before calling goto. Default false.
Parameters
- enabled: boolean
Returns void

setMixedProxy

setMixedProxy(mixedProxy: boolean | IMixedProxyOptions): void
Specifies mixed proxy configuration
Parameters
- mixedProxy: boolean | IMixedProxyOptions
Returns void

setUseRelayProxy

setUseRelayProxy(useRelayProxy: boolean): void
Specifies whether or not to use relay proxy in this extractor
Parameters
- useRelayProxy: boolean
Returns void

setUserAgent

setUserAgent(ua: string): void
Must be set before calling goto
Parameters
- ua: string
Returns void

setViewPort

setViewPort(viewport: {
height: number;
width: number;
}): void
Must be set before calling goto
Parameters
- viewport: {
  height: number;
  width: number;
  }
  - height: number
  - width: number
Returns void

solveCaptcha

solveCaptcha(options: ISolveCaptchaOptions): Promise<any>
Function allowing solving Captchas found on the page. It takes an options argument that defines an element on the page that contains the Captcha, and the type of Captcha it is.
Parameters
- options: ISolveCaptchaOptions
Returns Promise<any>

stop

stop(): Promise<void>
Stop the page (window.stop)

Returns Promise<void>

waitForFunction

waitForFunction(predicate: string | (() => any), options?: ITimeoutable, ...args: Json[]): Promise<void>
Parameters
- predicate: string | (() => any)
- Optional options: ITimeoutable
- Rest ...args: Json[]
Returns Promise<void>

waitForMutation

waitForMutation(selector: string, options: ITimeoutable): Promise<void>
Wait for a subtree modification
Parameters
- selector: string
- options: ITimeoutable
Returns Promise<void>

waitForNavigation

waitForNavigation(options?: IWaitOptions): Promise<(() => Promise<void>)>
This will return a promise of a promise when the hook is in place to detect the page change
Parameters
- Optional options: IWaitOptions
Returns Promise<(() => Promise<void>)>

waitForSelector

waitForSelector(selector: string, options?: ITimeoutable): Promise<void>
Parameters
- selector: string
- Optional options: ITimeoutable
Returns Promise<void>

waitForXPath

waitForXPath(xpath: string, options?: ITimeoutable): Promise<void>
Parameters
- xpath: string
- Optional options: ITimeoutable
Returns Promise<void>

Interface IContext

Hierarchy

Index

Properties

Methods

Properties

counter

retryContext

task

Methods

addToDom

Parameters

id: string

data: any

Optional options: IAddToDomOptions

Returns Promise<void>

captureRequests

Returns Promise<void>

checkBlocked

Parameters

Optional statusCode: number

Returns Promise<boolean>

click

Parameters

selector: string

Optional options: {}

Returns Promise<void>

clickAndWaitForNavigation

Parameters

selector: string

Optional clickOptions: object

Optional navigationOptions: IWaitOptions

Returns Promise<void>

content

Returns Promise<string>

cookies

Returns Promise<ICookie[]>

enableNetworkDebugger

Returns Promise<void>

evaluate

Parameters

pageFunction: (() => any)

Returns any

Rest ...args: Json[]

Returns Promise<Json>

evaluateInFrame

Parameters

context: string

pageFunction: (() => any)

Returns any

Rest ...args: Json[]

Returns Promise<void | Json>

extract

Parameters

id: string

Optional mergeOptions: IMergeOptions

Returns Promise<any>

generate2FAToken

Parameters

secret: string

Returns string

goto

Parameters

url: string

Optional options: IGoToOptions

Returns Promise<IResponse>

halt

Parameters

ignoreData: boolean

Returns void

reload

Returns Promise<void>

reportBlocked

Parameters

code: number

Optional details: string

Returns Promise<any>

reportWrongGeocoding

Returns Promise<any>

saveJson

`Optional` options: IAddToDomOptions

`Optional` statusCode: number

`Optional` options: {}

`Optional` clickOptions: object

`Optional` navigationOptions: IWaitOptions

`Rest` ...args: Json[]

`Rest` ...args: Json[]

`Optional` mergeOptions: IMergeOptions

`Optional` options: IGoToOptions

`Optional` details: string

`Optional` options: {
fullPage?: boolean;
type?: ScreenshotType;
}

`Optional` fullPage?: boolean

`Optional` type?: ScreenshotType

`Optional` options: IScrollOptions

`Rest` ...values: string[]