Judgement Document Spider

Use git clone git@github.com:Thesharing/court-spider.git --recursive to clone this repo into local.

Requires python >= 3.5, better use anaconda.

conda install certifi chardet idna numpy requests six urllib3 werkzeug Flask lxml
pip install opencv-python PyExecJS pymongo redis

Reference

wget http://download.redis.io/releases/redis-4.0.11.tar.gz
tar xzf redis-4.0.11.tar.gz
rm redis-4.0.11.tar.gz
mv redis-4.0.11 redis
cd redis
make

Reference

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
nodejs --version

cd spider
mkdir content data download log temp
cd ../proxy
touch key.txt

Refer to Thesharing/proxy_pool to modify the config of Redis.

Customize the spider in config.py and main.py.

First, run Redis.

Use Ubuntu on bash on Windows if you are using Windows.

(In the path of Redis)

src/redis-server

Then run proxy.

(In court-spider/proxy)

python main.py

Access 127.0.0.1/get_status to check the status of proxy pool.

raw_proxy means unvalidated proxy, useful_proxy means validated proxy. Change the website used to validate the proxies in Util/utilFunction.py.

Make sure the number of useful_proxy > 0, so that the spider can run with available proxies.

Finally, run the spider.

(In court-spider/spider)

python main.py

If you want to update this repo, please push the changes in the submodule, and then go back to the root folder of this project.

git submodule update
git add . 
git commit -m '<Commit Message>'
git push

sixs/wenshu_spider

jhao104/proxy_pool