Screen Scrapping Using Linux Commands

     Screen Scrapping is a very funny exercise to practice. You can select any website and scrap it down to get the necessary or to be precise the information that you needed. During my practice I selected the following website:

http://www.espncricinfo.com/australia-v-india-2011/engine/match/518950.html

     Initially we can use curl command to fetch the source code of a website.

curl 'http://www.espncricinfo.com/australia-v-india-2011/engine/match/518950.html'>cricinfo

In this example I am sending the output to cricinfo file.

Now in the bigger source code obtained we have to segregate the necessary information.


By analyzing the source code carefully, I finally came up with the following commands...

grep 'colspan="2">A\|colspan="2">I\|width="19\|width="25\|<td class="b' cricinfo     |     sed 's/.*colspan=\"2\">\([AI].*\)<span.*/\1/g'     |     sed 's/.*width=\"19.*playerName\">\(.*\)<\/a>.*/\1/g'     |     sed 's/.*width=\"25.*battingDismissal\">\(.*\)<\/td>/\1/g'     |     sed 's/<td class=\"\(.*\)\">\(.*\)<\/td>/\1\t\2/g'     |     sed 's/battingRuns\">\([0-9]*\)<\/td>.*/battingRuns\t\t\1/'     |     sed 's/bowlingDetails\".*//'>out

Finally the data will get stored in out file.






Output:



Australia 1st innings
EJM Cowan
c &dagger;Dhoni b Ashwin
  battingRuns   68
  battingDetails        294
  battingDetails        177
  battingDetails        7
  battingDetails        0
  battingDetails        38.41
DA Warner
c &dagger;Dhoni b Yadav
  battingRuns   37
  battingDetails        59
  battingDetails        49
  battingDetails        4
  battingDetails        1
  battingDetails        75.51
SE Marsh
c Kohli b Yadav
  battingRuns   0
  battingDetails        11
  battingDetails        6
  battingDetails        0
  battingDetails        0
  battingDetails        0.00
RT Ponting
c Laxman b Yadav
  battingRuns   62
  battingDetails        148
  battingDetails        94
  battingDetails        6
  battingDetails        0
  battingDetails        65.95
MJ Clarke
 b Khan
  battingRuns   31
  battingDetails        57
  battingDetails        68
  battingDetails        5
  battingDetails        0
    battingDetails        45.58
MEK Hussey
c &dagger;Dhoni b Khan
  battingRuns   0
  battingDetails        2
  battingDetails        1
  battingDetails        0
  battingDetails        0
  battingDetails        0.00
BJ Haddin
c Sehwag b Khan
  battingRuns   27
  battingDetails        119
  battingDetails        70
  battingDetails        1
  battingDetails        0
  battingDetails        38.57
PM Siddle
c &dagger;Dhoni b Khan
  battingRuns   41
  battingDetails        131
  battingDetails        99
  battingDetails        4
  battingDetails        0
  battingDetails        41.41
JL Pattinson
not out
  battingRuns   18
  battingDetails        86
  battingDetails        54
  battingDetails        2
  battingDetails        0
  battingDetails        33.33
BW Hilfenhaus
c Kohli b Ashwin
  battingRuns   19
  battingDetails        46
  battingDetails        32
  battingDetails        3
  battingDetails        0

  battingDetails        59.37

NM Lyon

 b Ashwin

  battingRuns   6

  battingDetails        15

  battingDetails        11

  battingDetails        1

  battingDetails        0

  battingDetails        54.54

  battingDismissal      (lb 21, w 2, nb 1)

  battingRuns           24

  battingDismissal      (all out; 110 overs; 488 mins)

  battingRuns   <b>333</b>

  battingRate" colspan="5       (3.02 runs per over)

  bowlingDetails        6

  bowlingDetails        77

  bowlingDetails        4

  bowlingDetails        2.48

  bowlingDetails        (1nb, 1w)

  bowlingDetails        7

  bowlingDetails        48

  bowlingDetails        0

  bowlingDetails        2.00

  bowlingDetails

  bowlingDetails        5

  bowlingDetails        106

  bowlingDetails        3

  bowlingDetails        4.07

  bowlingDetails        (1w)

  bowlingDetails        3

  bowlingDetails        81

  bowlingDetails        3

  bowlingDetails        2.79

  bowlingDetails

India 1st innings

G Gambhir

c &dagger;Haddin b Hilfenhaus

  battingRuns   3

  battingDetails        35

  battingDetails        23

  battingDetails        0

  battingDetails        0

  battingDetails        13.04

V Sehwag

 b Pattinson

  battingRuns   67

  battingDetails        127

  battingDetails        83

  battingDetails        7

  battingDetails        0

  battingDetails        80.72

R Dravid

 b Hilfenhaus

  battingRuns   68

  battingDetails        249

  battingDetails        187

  battingDetails        6

  battingDetails        0

  battingDetails        36.36

SR Tendulkar

 b Siddle

  battingRuns   73

  battingDetails        151

  battingDetails        98

  battingDetails        8

  battingDetails        1

  battingDetails        74.48

I Sharma

c &dagger;Haddin b Hilfenhaus

  battingRuns   11

  battingDetails        106

  battingDetails        69

  battingDetails        0

  battingDetails        0

  battingDetails        15.94

VVS Laxman

c &dagger;Haddin b Siddle

  battingRuns   2

  battingDetails        30

  battingDetails        22

  battingDetails        0

  battingDetails        0

  battingDetails        9.09

V Kohli

c &dagger;Haddin b Hilfenhaus

  battingRuns   11

  battingDetails        28

  battingDetails        21

  battingDetails        1

  battingDetails        0

  battingDetails        52.38

MS Dhoni

c Hussey b Hilfenhaus

  battingRuns   6

  battingDetails        13

  battingDetails        8

  battingDetails        0

  battingDetails        0

  battingDetails        75.00

R Ashwin

c &dagger;Haddin b Siddle

  battingRuns   31

  battingDetails        57

  battingDetails        35

  battingDetails        3

  battingDetails        1

  battingDetails        88.57

Z Khan

 b Pattinson

  battingRuns   4

  battingDetails        7

  battingDetails        6

  battingDetails        1

  battingDetails        0

  battingDetails        66.66

U Yadav

not out

  battingRuns   2

  battingDetails        22

  battingDetails        16

  battingDetails        0

  battingDetails        0

  battingDetails        12.50

  battingDismissal      (w 1, nb 3)

  battingRuns           4

  battingDismissal      (all out; 94.1 overs; 417 mins)

  battingRuns   <b>282</b>

  battingRate" colspan="5       (2.99 runs per over)

  bowlingDetails        6

  bowlingDetails        55

  bowlingDetails        2

  bowlingDetails        2.39

  bowlingDetails        (1w)

  bowlingDetails        5

  bowlingDetails        75

  bowlingDetails        5

  bowlingDetails        2.88

  bowlingDetails        (1nb)

  bowlingDetails        21.1

  bowlingDetails        2

  bowlingDetails        63

  bowlingDetails        3

  bowlingDetails        2.97

  bowlingDetails        (2nb)

  bowlingDetails        2

  bowlingDetails        66

  bowlingDetails        0

  bowlingDetails        3.88

  bowlingDetails

  bowlingDetails        0

  bowlingDetails        15

  bowlingDetails        0

  bowlingDetails        3.00

  bowlingDetails

  bowlingDetails        0

  bowlingDetails        8

  bowlingDetails        0

  bowlingDetails        4.00

  bowlingDetails

Australia 2nd innings

DA Warner

 b Yadav

  battingRuns   5

  battingDetails        28

  battingDetails        27

  battingDetails        0

  battingDetails        0

  battingDetails        18.51

EJM Cowan

lbw b Yadav

  battingRuns   8

  battingDetails        33

  battingDetails        17

  battingDetails        1

  battingDetails        0

  battingDetails        47.05

SE Marsh

 b Yadav

  battingRuns   3

  battingDetails        25

  battingDetails        11

  battingDetails        0

  battingDetails        0

  battingDetails        27.27

RT Ponting

c Sehwag b Khan

  battingRuns   60

  battingDetails        151

  battingDetails        97

  battingDetails        3

  battingDetails        0

  battingDetails        61.85

MJ Clarke

 b Sharma

  battingRuns   1

  battingDetails        6

  battingDetails        4

  battingDetails        0

  battingDetails        0

  battingDetails        25.00

MEK Hussey

c &dagger;Dhoni b Khan

  battingRuns   89

  battingDetails        227

  battingDetails        151

  battingDetails        9

  battingDetails        0

  battingDetails        58.94

BJ Haddin

c Laxman b Khan

  battingRuns   6

  battingDetails        11

  battingDetails        14

  battingDetails        0

  battingDetails        0

  battingDetails        42.85

PM Siddle

c &dagger;Dhoni b Yadav

  battingRuns   4

  battingDetails        29

  battingDetails        22

  battingDetails        0

  battingDetails        0

  battingDetails        18.18

NM Lyon

lbw b Ashwin

  battingRuns   0

  battingDetails        9

  battingDetails        11

  battingDetails        0

  battingDetails        0

  battingDetails        0.00

JL Pattinson

not out

  battingRuns   37

  battingDetails        97

  battingDetails        81

  battingDetails        4

  battingDetails        0

  battingDetails        45.67

BW Hilfenhaus

c Laxman b Sharma

  battingRuns   14

  battingDetails        45

  battingDetails        29

  battingDetails        2

  battingDetails        0

  battingDetails        48.27

  battingDismissal      (b 5, lb 2, w 1, nb 5)

  battingRuns           13

  battingDismissal      (all out; 76.3 overs; 335 mins)

  battingRuns   <b>240</b>

  battingRate" colspan="5       (3.13 runs per over)

  bowlingDetails        4

  bowlingDetails        53

  bowlingDetails        3

  bowlingDetails        2.65

  bowlingDetails        (3nb, 1w)

  bowlingDetails        4

  bowlingDetails        70

  bowlingDetails        4

  bowlingDetails        3.50

  bowlingDetails

  bowlingDetails        12.3

  bowlingDetails        0

  bowlingDetails        43

  bowlingDetails        2

  bowlingDetails        3.44

  bowlingDetails        (2nb)

  bowlingDetails        4

  bowlingDetails        60

  bowlingDetails        1

  bowlingDetails        2.72

  bowlingDetails

  bowlingDetails        0

  bowlingDetails        7

  bowlingDetails        0

  bowlingDetails        3.50

  bowlingDetails

India 2nd innings

G Gambhir

c Ponting b Siddle

  battingRuns   13

  battingDetails        52

  battingDetails        46

  battingDetails        1

  battingDetails        0

  battingDetails        28.26

V Sehwag

c Hussey b Hilfenhaus

  battingRuns   7

  battingDetails        28

  battingDetails        12

  battingDetails        1

  battingDetails        0

  battingDetails        58.33

R Dravid

 b Pattinson

  battingRuns   10

  battingDetails        59

  battingDetails        29

  battingDetails        1

  battingDetails        0

  battingDetails        34.48

SR Tendulkar

c Hussey b Siddle

  battingRuns   32

  battingDetails        73

  battingDetails        46

  battingDetails        4

  battingDetails        0

  battingDetails        69.56

VVS Laxman

c Cowan b Pattinson

  battingRuns   1

  battingDetails        22

  battingDetails        14

  battingDetails        0

  battingDetails        0

  battingDetails        7.14

V Kohli

lbw b Hilfenhaus

  battingRuns   0

  battingDetails        3

  battingDetails        1

  battingDetails        0

  battingDetails        0

  battingDetails        0.00

MS Dhoni

 b Pattinson

  battingRuns   23

  battingDetails        88

  battingDetails        44

  battingDetails        0

  battingDetails        1

  battingDetails        52.27

R Ashwin

c Cowan b Siddle

  battingRuns   30

  battingDetails        38

  battingDetails        35

  battingDetails        5

  battingDetails        0

  battingDetails        85.71

Z Khan

c Cowan b Pattinson

  battingRuns   13

  battingDetails        28

  battingDetails        22

  battingDetails        1

  battingDetails        1

  battingDetails        59.09

I Sharma

not out

  battingRuns   6

  battingDetails        33

  battingDetails        14

  battingDetails        0

  battingDetails        0

  battingDetails        42.85

U Yadav

c Warner b Lyon

  battingRuns   21

  battingDetails        26

  battingDetails        25

  battingDetails        2

  battingDetails        1

  battingDetails        84.00

  battingDismissal      (lb 10, w 2, nb 1)

  battingRuns           13

  battingDismissal      (all out; 47.5 overs; 228 mins)

  battingRuns   <b>169</b>

  battingRate" colspan="5       (3.53 runs per over)

  bowlingDetails        2

  bowlingDetails        53

  bowlingDetails        4

  bowlingDetails        3.53

  bowlingDetails        (1nb, 1w)

  bowlingDetails        4

  bowlingDetails        39

  bowlingDetails        2

  bowlingDetails        2.16

  bowlingDetails        (1w)

  bowlingDetails        1

  bowlingDetails        42

  bowlingDetails        3

  bowlingDetails        4.66

  bowlingDetails

  bowlingDetails        5.5

  bowlingDetails        0

  bowlingDetails        25

  bowlingDetails        1

  bowlingDetails        4.28

  bowlingDetails