Comments for Shut up and code http://lepunk.co.uk lePunk's thoughts about life, universe and everything Sun, 21 Apr 2013 16:15:10 +0000 hourly 1 http://wordpress.org/?v=3.5 Comment on Scraping the sh*t out of the interwebz – Part #1 by Scraping the sh*t out of the interwebz – Part #2 | Shut up and code http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-1690 Scraping the sh*t out of the interwebz – Part #2 | Shut up and code Sun, 21 Apr 2013 16:15:10 +0000 http://lepunk.co.uk/?p=83#comment-1690 [...] you haven’t read the first part of my php curl tutorial you should do so here. As in the previous one we will use my open source curl.class.php but this time we will do [...]

]]> Comment on Scraping the sh*t out of the interwebz – Part #1 by Gary Teh http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-247 Gary Teh Wed, 13 Mar 2013 15:34:58 +0000 http://lepunk.co.uk/?p=83#comment-247 Yup, the engine allows for the processing of RSS feeds. Inappropriate of web scraping does impose a huge bandwidth penalty on the content provider.

]]> Comment on Scraping the sh*t out of the interwebz – Part #1 by David Bradbury http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-219 David Bradbury Mon, 11 Mar 2013 19:44:56 +0000 http://lepunk.co.uk/?p=83#comment-219 Thanks for posting this. I’ve been working on scraping data recently so that I can keep track of trends in certain products I use. Remember to use RSS feeds or API’s when available to be more polite and decrease processing time! In the case of eBay, they have ebay dot com/sch/rss/. You can customize the results as well and it will be nicely organized for processing.

]]> Comment on Scraping the sh*t out of the interwebz – Part #1 by Gary Teh http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-210 Gary Teh Mon, 11 Mar 2013 02:41:57 +0000 http://lepunk.co.uk/?p=83#comment-210 Cool! Me and my team mate thought to make the entire scraping layer abstract and used this instead

https://krake.io/scrap-gm

{
origin_url: ‘http://www.ebay.com/sch/?_nkw=GM%20Part&_pgn=1′,
columns: [
{
col_name: 'item_name',
dom_query: 'h4 a'
}, {
col_name: 'item_detail_url',
dom_query: 'h4 a',
required_attribute: 'href',
options : {
columns: [{
col_name: 'description',
dom_query: '#desc_div'
},{
col_name: 'seller_name',
dom_query: '.mbg a[[0]]’
},{
col_name: ‘seller_profile_url’,
dom_query: ‘.mbg a[[0]]’,
required_attribute: ‘href’
}]
}
}, {
col_name: ‘item_image’,
dom_query: ‘.img img’,
required_attribute: ‘src’
}
],
next_page: {
dom_query: ‘.next’
}
};

]]> Comment on Scraping the sh*t out of the interwebz – Part #1 by dibbz http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-209 dibbz Sun, 10 Mar 2013 21:24:26 +0000 http://lepunk.co.uk/?p=83#comment-209 I prefer to use this class (below), it’s clean and lean code, I use curl instead of the built in as it is more resilient and I also like/need to follow redirects.
http://simplehtmldom.sourceforge.net/index.htm

Badly structured html? Look at Tidy.
http://www.php.net/manual/en/intro.tidy.php

]]> Comment on Scraping the sh*t out of the interwebz – Part #1 by lePunk http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-208 lePunk Sun, 10 Mar 2013 20:46:42 +0000 http://lepunk.co.uk/?p=83#comment-208 Yeah, good suggestion. However when I was trying out phpQuery i found that it fails to work on badly structured html

]]> Comment on Scraping the sh*t out of the interwebz – Part #1 by Phillip http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-207 Phillip Sun, 10 Mar 2013 20:36:17 +0000 http://lepunk.co.uk/?p=83#comment-207 If you looking to grab something odly specific phpQuery can make it easier than the DOMDocument can, it lets you use jQuery style selectors to grab elements. Because there jQuery based you can easily use the console to build them out with too. Just something to keep in find if you want the 4rd header containing the word “Trunip” that has the class “farkle”.

]]> Comment on Scraping the sh*t out of the interwebz – Part #1 by lePunk http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-205 lePunk Sun, 10 Mar 2013 16:57:07 +0000 http://lepunk.co.uk/?p=83#comment-205 Yes Emil, you are absolutely right. Will make sure to cover this topic in one of the following tutorials

]]> Comment on Scraping the sh*t out of the interwebz – Part #1 by Emil Vikström http://lepunk.co.uk/scraping-the-sht-out-of-the-interwebz-part-1/#comment-203 Emil Vikström Sun, 10 Mar 2013 16:42:56 +0000 http://lepunk.co.uk/?p=83#comment-203 This lacks a discussion of robots.txt and user agents, stuff that all web crawlers/scrapers should adhere to. In short: Make sure your user-agent is set to something sensible, preferrably with some contact information, and make sure you follow the rules set up by the target site’s robots.txt file.

]]> Comment on Tech support tips for developers by HK http://lepunk.co.uk/tech-support-tips-for-developers/#comment-121 HK Wed, 27 Feb 2013 11:50:18 +0000 http://lepunk.co.uk/?p=68#comment-121 I’m perfectly calm, thanks. Perhaps my faith in humanity is slightly dampened knowing that a cretin like you exists, but at least people like you make the rest of us look better.

So all you have to say is: “You don’t have the foggiest idea what you’re talking about”? …Which is just what I said to you repeated back to me?! Your intellect astounds me.
Basically, you have nothing more to say on the subject matter than “OMG, this is so wrong, everyone is a sociopath!!!1111″ with ZERO to back it up with. There is absolutely no point in wasting any time on you.

Tip for the future: Don’t just disagree with posts for the sake of it. IT DOES NOT MAKE YOU LOOK COOL.

Good luck finding a company stupid enough to employ you, especially as you’ve “forgotten” about management and development. Bahaha!

]]>