PyCharm setup scrapy (updated: Dec 2018)

Setup of PyCharm 2018


Download the .tar.gz from the official website


Download PyCharm from official link and unzip it


Remove the old zip file and move the folder to /opt/


Create symbolic link for PyCharm for bash


Execute PyCharm



Scrapy basic setup

Install using pip


Make the demo project named tutorial


File structure


Add spiders/ for testing

The name of the spider should be specified in the spiders/ using name = "quotes" (NOT file name / class name!)



Execute the spider

Upon successful execution, two html files will be added (quotes-1.html, quotes-2.html)


1. To execute the spider using command line


2. To programmatically execute the spider

Create an for PyCharm


To execute the spider