Tech Blog of Edward – Page 3 – Remember the steps, do be do be do

Bitwise operation

admin / 12 December, 2017

Initialize char with zeros

char testing = 0;
// 0000 0000 (testing)

1 2	char testing = 0; // 0000 0000 (testing)

Align bits from the right

testing |= 3
// 0000 0000 (before)
// 0000 0011 (after)

testing |= 3

// 0000 0000 (before)

// 0000 0011 (after)

Shift bits to the left

testing <<= 2
// 0000 0011 (before)
// 0000 1100 (after)

testing <<= 2

// 0000 0011 (before)

// 0000 1100 (after)

Extract bits using bit mask

char mask = 12  // 0000 1100
char temp = 255 // 1111 1111

mask & temp     // 0000 1100

char mask = 12 // 0000 1100

char temp = 255 // 1111 1111

mask & temp // 0000 1100

C++

fscanf problems with double

admin / 28 November, 2017

Trying to get input from the FILE* stream with image processing filter. Example One: Sample input (.txt)

2
0.7 0
0 0

0.7 0

0 0

Code snippet for scanning the double

fscanf(filterFile, "%lf", &filter.m[i][j]);

1	fscanf(filterFile, "%lf", &filter.m[i][j]);

Setting breakpoint in visual studio 2017, the filter.m returns set of values as the following

m[0][0] = 0.69999999999999996
m[0][1] = 0.00000000000000000
m[1][0] = 0.00000000000000000
m[1][1] = 0.00000000000000000

m[0][0] = 0.69999999999999996

m[0][1] = 0.00000000000000000

m[1][0] = 0.00000000000000000

m[1][1] = 0.00000000000000000

Which the first value, 0.7, wrongly being scanned as 0.69999999999999996. Example Two: Sample input (.txt)

5
0.04 0.04 0.04 0.04 0.04 
0.04 0.04 0.04 0.04 0.04 
0.04 0.04 0.04 0.04 0.04 
0.04 0.04 0.04 0.04 0.04 
0.04 0.04 0.04 0.04 0.04

0.04 0.04 0.04 0.04 0.04

…

Continue Reading→

Algorithms, C++, std

std::map

admin / 29 July, 2017

Simple insert

map<string, int> dataMap;

// Two methods to insert
// 1. Insert with pair
dataMap.insert(pair<string, int>("TESTING", 1));

// 2. Insert with array
dataMap["TESTING"] = 1;

map<string, int> dataMap;

// Two methods to insert

// 1. Insert with pair

dataMap.insert(pair<string, int>("TESTING", 1));

// 2. Insert with array

dataMap["TESTING"] = 1;

Access to second element using key (C++ 11)

cout << dataMap.at("TESTING");

// Console output = 1

cout << dataMap.at("TESTING");

// Console output = 1

Iterate

map<string, int>::iterator iter;

for (iter = dataMap.begin(); iter != dataMap.end(); iter++) {
	string first = iter->first;
	int second = iter->second;
}

map<string, int>::iterator iter;

for (iter = dataMap.begin(); iter != dataMap.end(); iter++) {

string first = iter->first;

int second = iter->second;

}

Find element

if (dataMap.find("TESTING") != dataMap.end()) 
	{...} // Found
else
	{...} // NOT Found

if (dataMap.find("TESTING") != dataMap.end())

{...} // Found

else

{...} // NOT Found

std

std::set

admin / 29 July, 2017

Simple insert

set<int> s;
s.insert(52);

1 2	set<int> s; s.insert(52);

Quick match

const bool found = s.find(123) != s.end();

1	const bool found = s.find(123) != s.end();

Store search result

set<int>::iterator it;

it = s.find(123)

set<int>::iterator it;

it = s.find(123)

Check if duplicate insert

s.insert(1);

auto a = s.insert(1);
auto b = s.insert(4);

// Fail insertion
// a.second == 0
cout << a.second;

// Success insertion
// b.second == 1
cout << b.second;

s.insert(1);

auto a = s.insert(1);

auto b = s.insert(4);

// Fail insertion

// a.second == 0

cout << a.second;

// Success insertion

// b.second == 1

cout << b.second;

Discovery, MS Office, Python

SharePoint URL study

admin / 24 July, 2017

For each SharePoint URL, the pattern can be observed as:

https://yourSiteURL.aspx
?RootFolder = YourRootFolder
&FolderCTID = 0xYourIDHere
&View = YourViewIDHere

https://yourSiteURL.aspx

?RootFolder = YourRootFolder

&FolderCTID = 0xYourIDHere

&View = YourViewIDHere

Root folder is encoded with HTML (%2F), which is /

some%2Frandom%2Fdirectory
# EQUAL TO
some/random/directory

some%2Frandom%2Fdirectory

# EQUAL TO

some/random/directory

FolderCTID is digits of length 38, i.e. 0x { 38 digits } The View is also encoded with HTML (%7B), which is { (%7D), which is } The format of digits length will be { 8 –…

Continue Reading→

MS Office, Python

Pyexcelerate write file

admin / 19 July, 2017

import pyexcelerate as pyex

dataWB = pyex.Workbook()
dataWS = dataWB.new_sheet("Database")

##### ROW & COL are NOT zero-based, i.e. 1, 2, 3 #####
dataWS.set_cell_value(row, col, "FOOBAR")

dataWB.save("OUTPUT.xlsx")

import pyexcelerate as pyex

dataWB = pyex.Workbook()

dataWS = dataWB.new_sheet("Database")

##### ROW & COL are NOT zero-based, i.e. 1, 2, 3 #####

dataWS.set_cell_value(row, col, "FOOBAR")

dataWB.save("OUTPUT.xlsx")

Selenium

Vimeo fetching video title

admin / 19 July, 2017

import selenium
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('https://vimeo.com/manage/videos')

import selenium

from selenium import webdriver

driver = webdriver.Chrome()

driver.get('https://vimeo.com/manage/videos')

# Login 
driver.find_element_by_xpath('//*[@id="signup_email"]').send_keys("inputTheUserName")
driver.find_element_by_xpath('//*[@id="login_password"]').send_keys("inputThePassword")
driver.find_element_by_xpath('//*[@id="login_form"]/div[5]/input').click()

# Login

driver.find_element_by_xpath('//*[@id="signup_email"]').send_keys("inputTheUserName")

driver.find_element_by_xpath('//*[@id="login_password"]').send_keys("inputThePassword")

driver.find_element_by_xpath('//*[@id="login_form"]/div[5]/input').click()

Check if xpath exist in current page

def check_exists_by_xpath(xpath):
    try:
        driver.find_element_by_xpath(xpath)
    except NoSuchElementException:
        return False
    return True

def check_exists_by_xpath(xpath):

try:

driver.find_element_by_xpath(xpath)

except NoSuchElementException:

return False

return True

Load all the components (3s pause by default for loading)

# Load all components
while(check_exists_by_xpath('//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/div/span')):
    try:
        time.sleep(3)
        more = driver.find_element_by_xpath('//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/div/span')
        more.click()
    except:
        break

# Load all components

while(check_exists_by_xpath('//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/div/span')):

try:

time.sleep(3)

more = driver.find_element_by_xpath('//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/div/span')

more.click()

except:

break

The format of video title will be (XPATH)

//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/table/tbody/tr[1]/td[3]/div/a
//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/table/tbody/tr[2]/td[3]/div/a
//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/table/tbody/tr[N]/td[3]/div/a

//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/table/tbody/tr[1]/td[3]/div/a

//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/table/tbody/tr[2]/td[3]/div/a

//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/table/tbody/tr[N]/td[3]/div/a

Fetch all the video titles

i = 1
while(True):
    try:
        temp = driver.find_element_by_xpath('//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/table/tbody/tr[{0}]/td[3]/div/a'.format(i))
        print(temp.text)
        i += 1
    except Exception as e:
        print("Extraction completed with {0} titles fetched".format(i))
        break

i = 1

while(True):

try:

temp = driver.find_element_by_xpath('//*[@id="wrap"]/div[2]/main/div/div/div[1]/div/table/tbody/tr[{0}]/td[3]/div/a'.format(i))

print(temp.text)

i += 1

except Exception as e:

print("Extraction completed with {0} titles fetched".format(i))

break

Python

PyCharm setup scrapy (updated: Dec 2018)

admin / 10 July, 2017

Setup of PyCharm 2018 Download the .tar.gz from the official website Download PyCharm from official link and unzip it

wget https://download-cf.jetbrains.com/python/pycharm-community-2018.3.1.tar.gz

tar -xvf pycharm-community-2018.3.1.tar.gz

wget https://download-cf.jetbrains.com/python/pycharm-community-2018.3.1.tar.gz

tar -xvf pycharm-community-2018.3.1.tar.gz

Remove the old zip file and move the folder to /opt/

rm pycharm-community-2018.3.1.tar.gz
sudo mv pycharm-community-2018.3.1/ /opt

1 2	rm pycharm-community-2018.3.1.tar.gz sudo mv pycharm-community-2018.3.1/ /opt

Create symbolic link for PyCharm for bash

sudo ln -s /opt/pycharm-community-2018.3.1/bin/pycharm.sh /usr/bin/pycharm

1	sudo ln -s /opt/pycharm-community-2018.3.1/bin/pycharm.sh /usr/bin/pycharm

Execute PyCharm

pycharm

pycharm

Scrapy basic setup Install using pip

pip3 install scrapy

1	pip3 install scrapy

Make the demo…

Continue Reading→

Linux, Python, Windows

Scrapy setup

admin / 7 July, 2017

Start a scrapy project at current directory

scrapy startproject tutorial

1	scrapy startproject tutorial

File structure

tutorial/
    scrapy.cfg            # deploy configuration file
    tutorial/             # project's Python module
        __init__.py
        items.py          # project items definition file
        pipelines.py      # project pipelines file
        settings.py       # project settings file
        spiders/          # spiders directory
            __init__.py

tutorial/

scrapy.cfg # deploy configuration file

tutorial/ # project's Python module

__init__.py

items.py # project items definition file

pipelines.py # project pipelines file

settings.py # project settings file

spiders/ # spiders directory

__init__.py

Set up the first spider file (tutorial/tutorial/spiders/quotes_spider.py)

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)

import scrapy

class QuotesSpider(scrapy.Spider):

name = "quotes"

def start_requests(self):

urls = [

'http://quotes.toscrape.com/page/1/',

'http://quotes.toscrape.com/page/2/',

]

for url in urls:

yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):

page = response.url.split("/")[-2]

filename = 'quotes-%s.html' % page

with open(filename, 'wb') as f:

f.write(response.body)

self.log('Saved file %s' % filename)

Linux, Python, Windows

Scrapy (web crawling) installation [Windows / Kali Linx]

admin / 7 July, 2017

For Python 2.7 / 3.3+ Windows Install scrapy using pip

pip install scrapy

1	pip install scrapy

Kali Linux Since I have both of the Python 2.7 (python) and 3.5.3+ (python3) installed, in order to install the scrapy on python3: Install the pip3 for python3, instead of using pip for old python

apt-get install python3-pip

1	apt-get install python3-pip

Try install scrapy using pip3

pip3 install scrapy

1	pip3 install scrapy

However, the dependency…

Continue Reading→