PyCharm setup scrapy (updated: Dec 2018)

Setup of PyCharm 2018

 

Download the .tar.gz from the official website

 

Download PyCharm from official link and unzip it

 

Remove the old zip file and move the folder to /opt/

 

Create symbolic link for PyCharm for bash

 

Execute PyCharm

 

 

Scrapy basic setup

Install using pip

 

Make the demo project named tutorial

 

File structure

 

Add spiders/quotes_spider.py for testing

The name of the spider should be specified in the spiders/quotes_spider.py using name = "quotes" (NOT file name / class name!)

 

 

Execute the spider

Upon successful execution, two html files will be added (quotes-1.html, quotes-2.html)

 

1. To execute the spider using command line

 

2. To programmatically execute the spider

Create an entry_point.py for PyCharm

 

To execute the spider