curl doesnt quite cut it

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

curl doesnt quite cut it

Postby chopficaro » Wed Aug 22, 2012 8:59 pm

i am using curl to grab search results from web pages
i want to use it on this site
http://www.twitch.tv/search?query=dan+d ... dinh&stp=1
but the source code returned does not have the title, description etc in the source code
instead they have variables which look like {{title}}, {{description}} in its place
i have searched all of the java script files and all the other files in the source code for these variable values and cannot find them
but if the web browser can translate these variables into their values, there must be a way i can do it in php
does anyone know if there are other functions like curl that have the functionality i am looking for?
chopficaro
Forum Commoner
 
Posts: 68
Joined: Fri Jan 01, 2010 1:56 am

Re: curl doesnt quite cut it

Postby Christopher » Thu Aug 23, 2012 10:22 am

You will need to have a browser render the page after you fetch it. cURL just sends/gets requests -- it does not know how to render different data formats.
(#10850)
User avatar
Christopher
Site Administrator
 
Posts: 12666
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Re: curl doesnt quite cut it

Postby chopficaro » Thu Aug 23, 2012 1:26 pm

is there a way to simulate browser rendering?
chopficaro
Forum Commoner
 
Posts: 68
Joined: Fri Jan 01, 2010 1:56 am

Re: curl doesnt quite cut it

Postby tr0gd0rr » Fri Aug 24, 2012 1:54 pm

chopficaro wrote:is there a way to simulate browser rendering?

CasperJS is an awesome way to use a virtual browser to open a site, manipulate dom, take screenshots, click elements, submit forms, etc. On Ubuntu, you can install it with `sudo apt-get install casperjs` or OSX `brew install casperjs`. It uses a full V8 browser engine and executes JavaScript like a normal browser.
User avatar
tr0gd0rr
Forum Contributor
 
Posts: 305
Joined: Thu May 11, 2006 8:58 pm
Location: Utah, USA


Return to PHP - Code

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 12 guests