Understanding Generators in Python.

Generators is a function in which objects are created at once but not all code is executed at once as done in normal function. In normal function execution from top to the return statement. A function that consists of a yield statement is called a generators function. The execution of the generator function happens differently, in which the code execution stops at the yield statement rather than a return statement, to move to the next statement next() method is called which will start the execution of the code from where it is left over. If no yield statement is found a StopIteration exception is raised.

So lets see how to create, execute a Generators in python.

def fib(n):
    a, b = 0, 1
    while a <= n:
        yield a   # yield statement.
        a, b = b, a + b

Now let execute the method fib().

fib_fun = fib(10)
next(fib_fun) # 0
next(fib_fun) # 1
next(fib_fun) # 1
.
.
.
next(fib_fun) # 8
next(fib_fun) # reached the end will raise StopIteration Error.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
 

# Else you can use for loop which call next() in the background.

for fib_value in fib(10):
    print(fib)

# Output
0
1
1
2
3
5
8

So here we today understand the Generators concept in python. Now you would be thinking where we can use this, let me state some use cases.

  • Can be used for memory management, where we pass the whole list as once, we can use Generator to pass data one by one so that less load comes on memory.
  • Generator can be used to define infinite streams.

If you know any more use case, please do share in the comments and if want to share something else or talk about Generators feel free to ping me on twitter

Till then Cheers 🙂
Happy Digging.

Scrapy The Tool

scrapy logo

As part of my job, I have to scrape some website to help our sales team with data on the market, as of now they were doing it manually which is a bit of tedious job to do and consumes lot of their productive time. So on bit searching and going through different tools and framework came across a framework named Scrapy. So here I am going to share how to set up and use Scrapy.

Scrapy is a free and open source web-crawling framework written in python which is used to extract data from a website without much of hassle. They have a very nice documentation you can check out here.

Steps to Install Scrapy

sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
pip install Scrapy

Steps to Create New Project

To create a Scrapy project type this command in your terminal.scrapy startproject <project name>. Project structure will look like this

now go ahead and create a python file at path /spiders and paste below code.

#!/usr/bin/env python3
import scrapy

class RedditSpider(scrapy.Spider):
    # name of the scrapper, it should be unique.
    name = "reddit"
    # list of the URL need to be iterated.
    start_urls = ['https://www.reddit.com/']

    # Called to do any operation on the response of the above URL.
    def parse(self, response):
       # css selector of the anchor tag which contains the headers
       top_post = response.css("a.SQnoC3ObvgnGjWt90zD9Z")
       for post in top_post:
           self.log(post.css('::text').extract_first())

To start scrapping, type

scrapy crawl reddit

Here we are scrapping the Reddit website for the latest post and getting the header of all the post. The output of the above code will look like this.

  • Trump Organization ‘Sold Property to Shell Company Linked to Maduro Regime,’ Says Report
  • Blind people of Reddit, what do you find sexually attractive?
  • A “caravan” of Americans is crossing the Canadian border to get affordable medical care
  • A “caravan” of Americans is crossing the Canadian border to get affordable medical care
  • [Post Game Thread] The Houston Rockets defeat the Golden State Warriors, 112-108, behind Harden’s 38 points to level the series 2-2, despite the continued brilliance of Kevin Durant 18, my friend here is failing biology and thinks she’s unroastable. Go for it guys, and go hard If you strike me down, I shall become more powerful than you can possibly imagine. [BOTW]
  • ELI5: Why are all economies expected to “grow”? Why is an equilibrium bad?
    ….

Now the best part of Scrapy is if you want to experiment around any website before creating any project you can easily do that.

scrapy shell 'https://www.reddit.com/'

and then can try different CSS selector on the response . Though there is a lot more you can do with Scrapy like saving the result in JSON, CSV format and even integrate with Django project might show that in next post, till then good bye.

Cheers

Python Tutorial -1

As wiki state

Python is a widely used high-level, general-purpose, interpreted, dynamic programming language.Its design philosophy emphasizes code readability, and its syntax allows programmers to express concepts in fewer lines of code than possible in languages such as C++ or Java.

Setting your PC to run Python

  • Window user can download Python from here
  • GNU/Linux can download with simple command :
    dnf install python
    Or you can download straight from here
  • Mac user head over straight to this Link

Now as you all are ready with python now let’s have some fun
I am using on Fedora 23, but it same for other OS
So, GNU/Linux user head straight to terminal and type python3 and python interpreter will open.

To declare a variable and print it’s value

This will declare variable a with value 10
a=10
To print the value type a and press enter
In real world(when we write our program in file) we use print(a) to print the value of variable
now further you can play around python and learn more from resources on internet.

Let me show you some Pythonic way of coding.

  • Pythonic way to swap two number’s
    C++ way to swap two numbers
    int a=10;
    int b=20;
    int temp=a;
    a=b;
    b=temp;

    But in python you can do this in one line
    a=10
    b=20
    a,b=b,a

    Yea this way only one line can do this
    Python is amazingly great soon will share more pythonic way of coding.

Till than cheers!
Happy Coding