Python

Hands On Machine Learning – Example 1.1 with CSV

Required CSV oecd_bli_2015.csv gdp_per_capita.csv   prepare_country_stats()

  Completed code for example 1.1

  Plotting using sklearn.linear_model.LinearRegression():                         Plotting using sklearn.neighbors.KNeighborsRegressor()with sampling n = 3:

Fix, Python

[Debug] no name in module error in pylint and flask

Problem The following errors occur in visual code editor

  Solution It should be an import error of from urllib.request import urlopen, which I am trying to use modern python3 library. Similarly, same errors keep popping up during the bootstrap of flask server. It can be concluded that the error is caused by the conflicts of python version, which the…

Continue Reading

Discovery, MS Office, Python

SharePoint URL study

For each SharePoint URL, the pattern can be observed as:

  Root folder is encoded with HTML (%2F), which is /

  FolderCTID is digits of length 38, i.e. 0x { 38 digits }   The View is also encoded with HTML (%7B), which is { (%7D), which is } The format of digits length will be { 8 –…

Continue Reading

MS Office, Python

Pyexcelerate write file

 

Selenium

Vimeo fetching video title

  Login

  Check if xpath exist in current page

  Load all the components (3s pause by default for loading)

  The format of video title will be (XPATH)  

  Fetch all the video titles

 

Python

PyCharm setup scrapy (updated: Dec 2018)

Setup of PyCharm 2018   Download the .tar.gz from the official website   Download PyCharm from official link and unzip it

  Remove the old zip file and move the folder to /opt/

  Create symbolic link for PyCharm for bash

  Execute PyCharm

    Scrapy basic setup Install using pip

  Make the demo…

Continue Reading

Linux, Python, Windows

Scrapy setup

Start a scrapy project at current directory

  File structure

  Set up the first spider file (tutorial/tutorial/spiders/quotes_spider.py)

 

Linux, Python, Windows

Scrapy (web crawling) installation [Windows / Kali Linx]

For Python 2.7 / 3.3+ Windows Install scrapy using pip

    Kali Linux Since I have both of the Python 2.7 (python) and 3.5.3+ (python3) installed, in order to install the scrapy on python3: Install the pip3 for python3, instead of using pip for old python

  Try install scrapy using pip3

  However, the dependency…

Continue Reading

Python

Make python 3.6 executable in Windows

It is noted that for current version of pyinstaller, python 3.6 is not supported, however, support can be added by the installation of the developer version. Solution: (Install the developer version of pyinstaller) Clone from the official repo

  To generate the desired .exe (add –onefile to generate standalone exe)

  The target.exe generated will be located at

Before the discovery…

Continue Reading

Selenium

Fetch stock price from Yahoo Finance

The url for stock number #1234

Pattern: https://hk.finance.yahoo.com/quote/STOCKNUM.HK/?p=STOCKNUM.HK   The stock number in url should be in 4 digits with leading zero, therefore, when reading input: