import-io-cli

This toolchain allows import.io users and managed service providers to build out scalable extractor definitions by creating a modular Extractor Library.

• Download and installation
• Updating to the latest version
• Getting started
• Context API
• Commands Reference

Download and installation

Pre-requisites

• Download and install Google Chrome if you don't already have it.
• On Mac, HomeBrew is required to install required dependencies. Follow webpage for installation details.

Windows

Windows x64

Windows x86

Mac and Linux

On Mac and Linux, the instal script requires: JQ, wget, tar, sha3sum

#on Mac
% brew install jq wget sha3sum

#on debian based linux
> sudo apt-get install jq tar wget -y

Then to download and install, run the following command:

wget -qO- https://prod-import-io-cli.s3.us-east-2.amazonaws.com/importio-install.sh | bash

CLI-2.3.5 (legacy version)

Mac , Windows x64 , Windows x86 , Linux Debian amd64 , Linux Debian armel

Updating to the latest version

> import-io update

To verify your current installed version, run:

> import-io --version

Please refer to import-io update [CHANNEL] for update command and additional flags.

Getting started

Concepts

An import.io Extractor Library is a git repository, that contains a library of modules and extractors for one or more organizations.

There are multiple types of modules:

• Action
  • A browser control and logic building block
  • Uses a browser context to control the browser - see IContext
  • Action may be used as an interface
    • A default definition may be provided
    • Named parameters (e.g. domain, country)
• Schema
  • A definition of what columns are expected to be returned
• Extraction
  • A definition of what to extract on the page
• Robot
  • An extractor template that is inherited from when creating an org extractor

Each instance of a module maps to a file within the git repository within the top-level src/library folder, and has a URI composed of the type and path, e.g. "robot:product/details".

Configure

> import-io config

This will prompt you for your SAAS API key and your DOC user token

Give it a try

Before going on the next steps verify you can start the browser up and get a REPL to control it by running:

> import-io browser:launch

This will start a browser and give you a REPL interface to control the browser, as well as the dev tools for the browser.

If this fails maybe the CLI cannot find your Chrome instance, set the CHROME_PATH environment variable.

For example on Linux Ubuntu add this to your .profile (with the proper path)

> export CHROME_PATH=/usr/bin/google-chrome

To stop the control simply type-in exit

Context API

Context API uses a browser context to control the browser

To jump straight into the browser context methods available, see Context API.

Commands Reference

• import-io action:compile
• import-io action:implement
• import-io action:interface
• import-io action:new
• import-io action:run:local
• import-io action:run:remote
• import-io browser:launch
• import-io cache:clear
• import-io config [NAME]
• import-io extraction:new FIRST
• import-io extractor:build
• import-io extractor:new
• import-io extractor:run:local
• import-io extractor:run:remote
• import-io extractor:tests:unit
• import-io extractor:tests:update
• import-io function:new PATH
• import-io init
• import-io robot:new
• import-io schema:new PATH
• import-io source:deploy
• import-io source:run
• import-io update [CHANNEL]

import-io action:compile

Compile an action to JS

USAGE
  $ import-io action:compile -P <value> -a <value> [-h]

FLAGS
  -P, --parameters=<value>...  (required) parameter values, key=value
  -a, --action=<value>         (required) action path
  -h, --help                   Show CLI help.

DESCRIPTION
  Compile an action to JS

See code: src/commands/action/compile.ts

import-io action:implement

Implement an action interface

USAGE
  $ import-io action:implement -P <value> -I <value> [-h]

FLAGS
  -I, --interface=<value>      (required) path to where interface is
  -P, --parameters=<value>...  (required) parameter values, key=value
  -h, --help                   Show CLI help.

DESCRIPTION
  Implement an action interface

See code: src/commands/action/implement.ts

import-io action:interface

Create a new interface with default implementation

USAGE
  $ import-io action:interface -p <value> [-h] [-P <value>] [-i <value>]

FLAGS
  -P, --parameters=<value>...  parameter name
  -h, --help                   Show CLI help.
  -i, --inputs=<value>...      input name
  -p, --path=<value>           (required) new action path

DESCRIPTION
  Create a new interface with default implementation

See code: src/commands/action/interface.ts

import-io action:new

Create a new action (not an interface impl)

USAGE
  $ import-io action:new -p <value> [-h] [-P <value>] [-i <value>]

FLAGS
  -P, --parameters=<value>...  parameter name
  -h, --help                   Show CLI help.
  -i, --inputs=<value>...      input name
  -p, --path=<value>           (required) new action path

DESCRIPTION
  Create a new action (not an interface impl)

See code: src/commands/action/new.ts

import-io action:run:local

Run an action locally

USAGE
  $ import-io action:run:local -a <value> [-h] [-P <value>] [-i <value>] [--proxy <value>] [--proxyauth <value>] [-c]
    [-C <value>] [-I]

FLAGS
  -C, --credentials=<value>...  credentials for an auth extractor, username:password
  -I, --incognito               run with an incognito browser
  -P, --parameters=<value>...   [default: ] parameter values, key=value
  -a, --action=<value>          (required) action path
  -c, --compile                 compile down to the code action (cannot use breakpoints)
  -h, --help                    Show CLI help.
  -i, --inputs=<value>...       input values, key=value
  --proxy=<value>               proxy, e.g. http://10.10.10.10:8000
  --proxyauth=<value>           proxyauth, e.g. user:password

DESCRIPTION
  Run an action locally

See code: src/commands/action/run/local.ts

import-io action:run:remote

Run the action locally using an import.io remote browser

USAGE
  $ import-io action:run:remote -a <value> [-h] [-P <value>] [-i <value>] [-C <value>] [-r] [-c <value>] [-I]

FLAGS
  -C, --country=<value>         country code for proxy to checkout
  -I, --incognito               run with an incognito browser
  -P, --parameters=<value>...   [default: ] parameter values, key=value
  -a, --action=<value>          (required) action path
  -c, --credentials=<value>...  Uses credentials for auth extractor, with username:password
  -h, --help                    Show CLI help.
  -i, --inputs=<value>...       input values, key=value
  -r, --residential             Use residential proxies

DESCRIPTION
  Run the action locally using an import.io remote browser

See code: src/commands/action/run/remote.ts

import-io browser:launch

Launch a browser for testing

USAGE
  $ import-io browser:launch [-h] [--proxy <value>] [--proxyauth <value>]

FLAGS
  -h, --help           Show CLI help.
  --proxy=<value>      proxy, e.g. http://10.10.10.10:8000
  --proxyauth=<value>  proxyauth, e.g. user:password

DESCRIPTION
  Launch a browser for testing

See code: src/commands/browser/launch.ts

import-io cache:clear

Clear cache

USAGE
  $ import-io cache:clear

DESCRIPTION
  Clear cache

See code: src/commands/cache/clear.ts

import-io config [NAME]

Intialize developer configuration

USAGE
  $ import-io config [NAME] [-h]

FLAGS
  -h, --help  Show CLI help.

DESCRIPTION
  Intialize developer configuration

See code: src/commands/config.ts

import-io extraction:new FIRST

Create a new extraction

USAGE
  $ import-io extraction:new FIRST -s <value> [-h]

FLAGS
  -h, --help            Show CLI help.
  -s, --schema=<value>  (required) schema path

DESCRIPTION
  Create a new extraction

See code: src/commands/extraction/new.ts

import-io extractor:build

Build extractor (s)

USAGE
  $ import-io extractor:build -o <value> [-h] [-p <value>]

FLAGS
  -h, --help               Show CLI help.
  -o, --org=<value>        (required) org slug
  -p, --prefix=<value>...  path prefix to search in

DESCRIPTION
  Build extractor (s)

See code: src/commands/extractor/build.ts

import-io extractor:new

Create a new robot

USAGE
  $ import-io extractor:new -r <value> -o <value> [-h] [-P <value>] [-p <value>] [-a]

FLAGS
  -P, --parameters=<value>...  [default: ] parameter values, key=value
  -a, --auth                   whether or not to scaffold out auth actions and dependencies
  -h, --help                   Show CLI help.
  -o, --org=<value>            (required) org slug
  -p, --path=<value>           path override
  -r, --robot=<value>          (required) path to robot

DESCRIPTION
  Create a new robot

See code: src/commands/extractor/new.ts

import-io extractor:run:local

Run an extractor locally

USAGE
  $ import-io extractor:run:local -e <value> -o <value> -i <value> [-h] [--proxy <value>] [--proxyauth <value>] [-c]
  [-I]

FLAGS
  -I, --incognito          run with an incognito browser
  -c, --compile            compile down to the code action (cannot use breakpoints)
  -e, --extractor=<value>  (required) path to extractor directory
  -h, --help               Show CLI help.
  -i, --inputs=<value>...  (required) input values, key=value
  -o, --org=<value>        (required) org slug
  --proxy=<value>          proxy, e.g. http://10.10.10.10:8000
  --proxyauth=<value>      proxyauth, e.g. user:password

DESCRIPTION
  Run an extractor locally

See code: src/commands/extractor/run/local.ts

import-io extractor:run:remote

Run an extractor remotely

USAGE
  $ import-io extractor:run:remote -e <value> -o <value> [-h] [-i <value>]

FLAGS
  -e, --extractor=<value>  (required) path to extractor directory
  -h, --help               Show CLI help.
  -i, --inputs=<value>...  input values, key=value
  -o, --org=<value>        (required) org slug

DESCRIPTION
  Run an extractor remotely

See code: src/commands/extractor/run/remote.ts

import-io extractor:tests:unit

Check that the extractions still give the expected data

USAGE
  $ import-io extractor:tests:unit -o <value> [-h] [-p <value>]

FLAGS
  -h, --help               Show CLI help.
  -o, --org=<value>        (required) org slug
  -p, --prefix=<value>...  path prefix to search in

DESCRIPTION
  Check that the extractions still give the expected data

See code: src/commands/extractor/tests/unit.ts

import-io extractor:tests:update

Import a crawl run as a set of test cases

USAGE
  $ import-io extractor:tests:update -e <value> -c <value> -o <value> [-h]

FLAGS
  -c, --crawlRunId=<value>  (required) id of crawl run to import
  -e, --extractor=<value>   (required) path to extractor directory
  -h, --help                Show CLI help.
  -o, --org=<value>         (required) org slug

DESCRIPTION
  Import a crawl run as a set of test cases

See code: src/commands/extractor/tests/update.ts

import-io function:new PATH

Create a new function

USAGE
  $ import-io function:new PATH [-h]

FLAGS
  -h, --help  Show CLI help.

DESCRIPTION
  Create a new function

See code: src/commands/function/new.ts

import-io init

Initialize a library

USAGE
  $ import-io init [-h] [-o <value>]

FLAGS
  -h, --help         Show CLI help.
  -o, --org=<value>  org slug

DESCRIPTION
  Initialize a library

See code: src/commands/init.ts

import-io robot:new

Create a new robot

USAGE
  $ import-io robot:new -s <value> -P <value> -e <value> -r <value> [-h] [-a <value>] [-c <value>]

FLAGS
  -P, --parameters=<value>...        (required) parameters
  -a, --authentication=<value>       authentication action entrypoint
  -c, --checkAuthentication=<value>  validate authentication action entrypoint
  -e, --entryPoint=<value>           (required) action entrypoint
  -h, --help                         Show CLI help.
  -r, --path=<value>                 (required) path to robot
  -s, --schema=<value>               (required) path to schema

DESCRIPTION
  Create a new robot

See code: src/commands/robot/new.ts

import-io schema:new PATH

Create a new schema

USAGE
  $ import-io schema:new PATH [-h]

FLAGS
  -h, --help  Show CLI help.

DESCRIPTION
  Create a new schema

See code: src/commands/schema/new.ts

import-io source:deploy

Deploy an extractor and update the sample inputs

USAGE
  $ import-io source:deploy -e <value> (-c <value> -o <value>) -p <value> -s <value> [-h]

FLAGS
  -c, --collection=<value>  (required) collection to deploy to
  -e, --path=<value>        (required) path to extractor directory
  -h, --help                Show CLI help.
  -o, --org=<value>         (required) org slug
  -p, --project=<value>     (required) project to deploy to
  -s, --source=<value>      (required) source slug

DESCRIPTION
  Deploy an extractor and update the sample inputs

See code: src/commands/source/deploy.ts

import-io source:run

Run an extractor

USAGE
  $ import-io source:run -e <value> -o <value> -p <value> -c <value> -s <value> [-h] [-d] [-w]

FLAGS
  -c, --collection=<value>  (required) collection to deploy to
  -d, --deploy              deploy before running, if false will run the currently saved extractor
  -e, --path=<value>        (required) path to extractor directory
  -h, --help                Show CLI help.
  -o, --org=<value>         (required) org slug
  -p, --project=<value>     (required) project to deploy to
  -s, --source=<value>      (required) source slug
  -w, --wait                whether or not to wait until the crawl run completes

DESCRIPTION
  Run an extractor

See code: src/commands/source/run.ts

import-io update [CHANNEL]

update the import-io CLI

USAGE
  $ import-io update [CHANNEL] [-a] [-v <value> | -i] [--force]

FLAGS
  -a, --available        Install a specific version.
  -i, --interactive      Interactively select version to install. This is ignored if a channel is provided.
  -v, --version=<value>  Install a specific version.
  --force                Force a re-download of the requested version.

DESCRIPTION
  update the import-io CLI

EXAMPLES
  Update to the stable channel:

    $ import-io update stable

  Update to a specific version:

    $ import-io update --version 1.0.0

  Interactively select version:

    $ import-io update --interactive

  See available versions:

    $ import-io update --available

See code: @oclif/plugin-update