serps / cli
Command line utility to scrape from the terminal
Installs: 631
Dependents: 3
Suggesters: 0
Security: 0
Stars: 14
Watchers: 5
Forks: 7
Open Issues: 2
Type:project
Requires
- php: >=5.5.9
- serps/core: ~0.3.0
- serps/http-client-curl: ~0.2.0
- serps/http-client-phantomjs: ~0.2.0
- serps/search-engine-google: ~0.3.0|~0.4.0
- symfony/console: ^3.3
- zendframework/zend-diactoros: ^1.4
Requires (Dev)
This package is auto-updated.
Last update: 2024-11-06 19:29:22 UTC
README
Command line utility to scrape from the terminal
Install
You can install it globally with composer:
composer global require serps/cli
Test installation by invoking the help message
serps help
Search
Search command allows to trigger a google search from the command line
serps google:search "keyword"
example
$ serps google:search "github" { "natural-results": [ { "url": "https://github.com/", "title": "How people build software · GitHub", "types": [ "classical", "classical_large" ] }, { "url": "https://fr.wikipedia.org/wiki/GitHub", "title": "GitHub — Wikipédia", "types": [ "classical" ] }, { "url": "https://en.wikipedia.org/wiki/GitHub", "title": "GitHub - Wikipedia, the free encyclopedia", "types": [ "classical" ] }, { "url": "http://rue89.nouvelobs.com/2015/03/31/quest-tous-les-techos-monde-font-github-258439", "title": "Qu'est-ce que tous les techos du monde font sur GitHub ? - Rue89 - L ...", "types": [ "classical" ] }, { "url": "https://twitter.com/github?lang=fr", "title": "GitHub (@github) | Twitter", "types": [ "classical" ] }, { "url": "https://www.githubarchive.org/", "title": "GitHub Archive", "types": [ "classical" ] }, { "url": "https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin", "title": "GitHub Plugin - Jenkins - Jenkins Wiki", "types": [ "classical" ] } ], "total-count": 1.31e+08, "natural-results-count": 7, "evaluated": true, "http-client": "curl", "url": "https://www.google.fr/search?q=github&gws_rd=cr&ei=kH7OV7LaForeU_yGhtgC", "initial-url": "https://google.com/search?q=github" }
Tip: In the example jq helped to pretty format the outputted json:
$ serps google:search "github" | jq '.'
Advanced usage
$ serps google:search --tld="co.uk" --lr="lang_es" "some keywords" phantomjs
With proxy:
$ serps google:search --proxy="http://proxy:8080" "some keywords"
Page and result per page:
$ serps google:search --page=2 --res-per-page=20 "some keywords"
User Ageng:
$ serps google:search --user-agent="..." "some keywords"
Mobile search
This is an alertnative to the user agent option to use an hard coded mobile user agent
$ serps google:search --mobile "some keywords"
Dump page in a file:
In a addition of printing the results you can save the dom in a file
$ serps google:search --dump="/path/to/file.html" "some keywords"
By default the dump option wont be able to process if the file you specify already exists. The option force-dump makes it able to override an existing file:
$ serps google:search --dump="/path/to/file.html" --force-dump=true "some keywords"
Read and parse a html google file saved locally instead of doing http query
$ serps google:search --file=./google--search-file.html "some keywords"