Get text from webpage

[QM_Expert]

Hey can you show me an example? What i would like is to get the text thats on this webpage.

page: www.helphelphelp.com.net.sc

Thanks for the help!

[Gintaras]

Getting Text From Webpage and Seperating It

[QM_Expert]

ok thats not really what i needed. What i need is it to get it while its not open. like matt's temperature macro did. Just to get it and out it though. While the page is not open.

Thanks

[Gintaras]

This gets html

Macro

Code:

Copy      Help

str s
IntGetFile "http://www.quickmacros.com" s
out s

This gets text

Macro

Code:

Copy      Help

HtmlDoc d.InitFromWeb("http://www.quickmacros.com")
str s=d.GetText
out s

[QM_Expert]

perfect thanks a ton!

[QM_Expert]

Hey just a question out of curiousity....will this work on older versions of QM? If not what is the newest one needed for it to work?

[Gintaras]

IntGetFile works with all QM versions.

HtmlDoc added in QM 2.2.1 (January 1, 2008). Older versions could extract text from html (downloaded with IntGetFile) using regular expressions (quite difficult), or could use MSHTML functions (HtmlDoc functions use them).

[QM_Expert]

Can you show me an example with older QM because my other computer cant handle the newer versions of QM not sure why lol. waiting to get some money to buy a new one thanks

[Gintaras]

See how HtmlDoc functions do it. Or try to export HtmlDoc folder (\System\Functions\HtmlDoc) and import in the old QM version. Can be anywhere.

[donald stein]

using the example from initfromweb, is it possible to get it to log in so i can get past you are not logged in, so i can get to the page which im trying to do?
I.E have it click on the login button using initfromweb?
thanks in advance!

[QM Assistant]

If you need a button to be pressed, why not just use HTML Element Action?

[donald stein]

I like this code, I was just wanting to know if It could obtain secure info as well, without the browser being visible,which in most cases means I would have to be logged into the page which I am obtaining the text from, which would require me to type in my login info then click on the login button, is it possible for this code to input my details and also click on a button while page is not visible?

example, say I was watching a post in qm forum and wanted to get all the text from that page, well this code can do that, but say I was logged out last time I was on this forum, in order for this code to work , it would need to input my login details ,click login, then it could obtain the text which I seek. is this possible?

[QM Assistant]

I do not believe that would be possible with HTML Element Actions. Initfromweb, I'm not sure since i've never tried it.

[Gintaras]

try to post login form using Http.PostFormData
but it is quite difficult and not always works
i tried with qm forum, and it works, partially

Macro

Code:

Copy      Help

str username="x"
str password="y"

str s
Http h.Connect("www.quickmacros.com")

;get sid because it is different everytime
h.FileGet("forum/ucp.php?mode=login" s)
str sid
if(findrx(s "name=''sid'' value=''(.+?)''" 0 0 sid 1)<0)
,goto g1
,ret
out sid

;post form
ARRAY(POSTFIELD) a.create(4)
a[0].name="username"; a[0].value=username
a[1].name="password"; a[1].value=password
a[2].name="sid"; a[2].value=sid
a[3].name="login"; a[3].value="Login"
h.PostFormData("forum/ucp.php?mode=login" a s)

;see what we have
h.FileGet("forum\index.php" s)
;out s
;g1
str f="$temp$\qm forum.htm"
s.setfile(f)
run f

to see what form data firefox is sending, i installed 'live http headers' ff addition

[donald stein]

thanks gin will give it a go!

[QM Assistant]

Is it possible to use something like this to get text from a document, without the document being opened?

[QM Assistant]

Any ideas?

[Gintaras]

no

[QM Assistant]

All right, thanks.

[QM Assistant]

I got it.

Macro

Code:

Copy      Help

HtmlDoc a.InitFromFile("C:\Users\Info\Info\Info\Info\Info\Info.html")
str b=a.GetText
out b

[Lucas]

I try get text of cgi web consult.

"http://web/folder/consult.cgi?name1=value1&name2=value2"

the problem is Usernameassword.

when run:

out
str s
Http h.Connect("web" "username" "password")
IntPost "http://web/folder/consult.cgi" "name1=value1&name2=value2" s
out s

the result:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Authorization Required</title>
</head><body>
<h1>Authorization Required</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
<hr>
<address>Apache/2.2.3 (Red Hat) Server at web Port 80</address>
</body></html>

I tried PostFormData but doesn't work.

Using a Inspector (external program Fiddler):

No Proxy-Authorization Header is present.

Authorization Header is present: Basic ZjE4MDpwd2QxODA=
Decoded Usernameassword= xxx:yyy

Can you help?

[Gintaras]

Don't use Http.Connect with IntPost.

http://www.devx.com/tips/Tip/5604

Or use Http.Connect with Http.PostFormData or with Http.Post.

[Lucas]

I tried

out
str s
IntPost "http://username:password@web/folder/consult.cgi" "name1=value1&name2=value2" s
out s

and doesn't work.

[Gintaras]

Then try
Http h.Connect("web" "username" "password")
h.Post("folder/consult.cgi" "name1=value1&name2=value2" s)

[Lucas]

Now, it works!

How can I open a web in Internet Explorer using Http Class?

[Gintaras]

Http class does not open IE. Do you want to find a link in downloaded file and open the link in IE?

[Lucas]

No,

When I use h.GetUrl("http://web" s2) ; out s2 ----> works

How can I open "http://web"

[Gintaras]

Don't know. What is "web"?

[Lucas]

Is possible keep the Connection alive with the host and then to use the command web?

[Lucas]

I tried

h.Post("/cgi-bin/login.pl" _s s 0 INTERNET_FLAG_KEEP_CONNECTION s4)
out s
out "----"
out s4
out "-----"

but

s4=

HTTP/1.1 200 OK
Date: Tue, 17 Mar 2009 18:08:29 GMT
Server: Apache/2.0.46 (Red Hat)
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=ISO-8859-1


any way to Connection: Keep-Alive ?

note: using
HTTP Debugging Proxy (external program).
Cookies / Login
Cookie
SESSION_ID=a2b099362464360d6d909b478edcdf4d