.\" Automatically generated by Pod::Man 2.27 (Pod::Simple 3.28) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. \*(C+ will .\" give a nicer C++. Capital omega is used to do unbreakable dashes and .\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff, .\" nothing in troff, for use with C<>. .tr \(*W- .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is turned on, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{ . if \nF \{ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "WWW::Mechanize::FAQ 3" .TH WWW::Mechanize::FAQ 3 "2020-02-20" "perl v5.16.3" "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH "NAME" WWW::Mechanize::FAQ \- Frequently Asked Questions about WWW::Mechanize .SH "VERSION" .IX Header "VERSION" version 1.96 .SH "How to get help with WWW::Mechanize" .IX Header "How to get help with WWW::Mechanize" If your question isn't answered here in the \s-1FAQ,\s0 please turn to the communities at: .IP "\(bu" 4 StackOverflow .IP "\(bu" 4 #lwp on irc.perl.org .IP "\(bu" 4 .IP "\(bu" 4 The libwww-perl mailing list at .SH "JavaScript" .IX Header "JavaScript" .SS "I have this web page that has JavaScript on it, and my Mech program doesn't work." .IX Subsection "I have this web page that has JavaScript on it, and my Mech program doesn't work." That's because WWW::Mechanize doesn't operate on the JavaScript. It only understands the \s-1HTML\s0 parts of the page. .SS "I thought Mech was supposed to work like a web browser." .IX Subsection "I thought Mech was supposed to work like a web browser." It does pretty much, but it doesn't support JavaScript. .PP I added some basic attempts at picking up URLs in \f(CW\*(C`window.open()\*(C'\fR calls and return them in \f(CW\*(C`$mech\->links\*(C'\fR. They work sometimes. .PP Since Javascript is completely visible to the client, it cannot be used to prevent a scraper from following links. But it can make life difficult. If you want to scrape specific pages, then a solution is always possible. .PP One typical use of Javascript is to perform argument checking before posting to the server. The \s-1URL\s0 you want is probably just buried in the Javascript function. Do a regular expression match on \&\f(CW\*(C`$mech\->content()\*(C'\fR to find the link that you want and \f(CW\*(C`$mech\->get\*(C'\fR it directly (this assumes that you know what you are looking for in advance). .PP In more difficult cases, the Javascript is used for \s-1URL\s0 mangling to satisfy the needs of some middleware. In this case you need to figure out what the Javascript is doing (why are these URLs always really long?). There is probably some function with one or more arguments which calculates the new \s-1URL.\s0 Step one: using your favorite browser, get the before and after URLs and save them to files. Edit each file, converting the argument separators ('?', '&' or ';') into newlines. Now it is easy to use diff or comm to find out what Javascript did to the \s-1URL.\s0 Step 2 \- find the function call which created the \s-1URL \-\s0 you will need to parse and interpret its argument list. The Javascript Debugger in the Firebug extension for Firefox helps with the analysis. At this point, it is fairly trivial to write your own function which emulates the Javascript for the pages you want to process. .PP Here's another approach that answers the question, \*(L"It works in Firefox, but why not Mech?\*(R" Everything the web server knows about the client is present in the \s-1HTTP\s0 request. If two requests are identical, the results should be identical. So the real question is \*(L"What is different between the mech request and the Firefox request?\*(R" .PP The Firefox extension \*(L"Tamper Data\*(R" is an effective tool for examining the headers of the requests to the server. Compare that with what \s-1LWP\s0 is sending. Once the two are identical, the action of the server should be the same as well. .PP I say \*(L"should\*(R", because this is an oversimplification \- some values are naturally unique, e.g. a SessionID, but if a SessionID is present, that is probably sufficient, even though the value will be different between the \s-1LWP\s0 request and the Firefox request. The server could use the session to store information which is troublesome, but that's not the first place to look (and highly unlikely to be relevant when you are requesting the login page of your site). .PP Generally the problem is to be found in missing or incorrect \s-1POSTDATA\s0 arguments, Cookies, User-Agents, Accepts, etc. If you are using mech, then redirects and cookies should not be a problem, but are listed here for completeness. If you are missing headers, \f(CW\*(C`$mech\->add_header\*(C'\fR can be used to add the headers that you need. .SS "Which modules work like Mechanize and have JavaScript support?" .IX Subsection "Which modules work like Mechanize and have JavaScript support?" In no particular order: Gtk2::WebKit::Mechanize, Win32::IE::Mechanize, WWW::Mechanize::Firefox, WWW::Scripter, WWW::Selenium .SH "How do I do X?" .IX Header "How do I do X?" .SS "Can I do [such\-and\-such] with WWW::Mechanize?" .IX Subsection "Can I do [such-and-such] with WWW::Mechanize?" If it's possible with LWP::UserAgent, then yes. WWW::Mechanize is a subclass of LWP::UserAgent, so all the wondrous magic of that class is inherited. .SS "How do I use WWW::Mechanize through a proxy server?" .IX Subsection "How do I use WWW::Mechanize through a proxy server?" See the docs in LWP::UserAgent on how to use the proxy. Short version: .PP .Vb 1 \& $mech\->proxy([\*(Aqhttp\*(Aq, \*(Aqftp\*(Aq], \*(Aqhttp://proxy.example.com:8000/\*(Aq); .Ve .PP or get the specs from the environment: .PP .Vb 1 \& $mech\->env_proxy(); \& \& # Environment set like so: \& gopher_proxy=http://proxy.my.place/ \& wais_proxy=http://proxy.my.place/ \& no_proxy="localhost,my.domain" \& export gopher_proxy wais_proxy no_proxy .Ve .SS "How can I see what fields are on the forms?" .IX Subsection "How can I see what fields are on the forms?" Use the mech-dump utility, optionally installed with Mechanize. .PP .Vb 6 \& $ mech\-dump \-\-forms http://search.cpan.org \& Dumping forms \& GET http://search.cpan.org/search \& query= \& mode=all (option) [*all|module|dist|author] \& =CPAN Search (submit) .Ve .SS "How do I get Mech to handle authentication?" .IX Subsection "How do I get Mech to handle authentication?" .Vb 1 \& use MIME::Base64; \& \& my $agent = WWW::Mechanize\->new(); \& my @args = ( \& Authorization => "Basic " . \& MIME::Base64::encode( USER . \*(Aq:\*(Aq . PASS ) \& ); \& \& $agent\->credentials( ADDRESS, REALM, USER, PASS ); \& $agent\->get( URL, @args ); .Ve .PP If you want to use the credentials for all future requests, you can also use the LWP::UserAgent \f(CW\*(C`default_header()\*(C'\fR method instead of the extra arguments to \f(CW\*(C`get()\*(C'\fR .PP .Vb 2 \& $mech\->default_header( \& Authorization => \*(AqBasic \*(Aq . encode_base64( USER . \*(Aq:\*(Aq . PASSWORD ) ); .Ve .SS "How can I get WWW::Mechanize to execute this JavaScript?" .IX Subsection "How can I get WWW::Mechanize to execute this JavaScript?" You can't. JavaScript is entirely client-based, and WWW::Mechanize is a client that doesn't understand JavaScript. See the top part of this \s-1FAQ.\s0 .SS "How do I check a checkbox that doesn't have a value defined?" .IX Subsection "How do I check a checkbox that doesn't have a value defined?" Set it to the value of \*(L"on\*(R". .PP .Vb 1 \& $mech\->field( my_checkbox => \*(Aqon\*(Aq ); .Ve .SS "How do I handle frames?" .IX Subsection "How do I handle frames?" You don't deal with them as frames, per se, but as links. Extract them with .PP .Vb 1 \& my @frame_links = $mech\->find_link( tag => "frame" ); .Ve .SS "How do I get a list of \s-1HTTP\s0 headers and their values?" .IX Subsection "How do I get a list of HTTP headers and their values?" All HTTP::Headers methods work on a HTTP::Response object which is returned by the \fI\fIget()\fI\fR, \fI\fIreload()\fI\fR, \fI\fIresponse()\fI/\fIres()\fI\fR, \fI\fIclick()\fI\fR, \&\fI\fIsubmit_form()\fI\fR, and \fI\fIrequest()\fI\fR methods. .PP .Vb 6 \& my $mech = WWW::Mechanize\->new( autocheck => 1 ); \& $mech\->get( \*(Aqhttp://my.site.com\*(Aq ); \& my $response = $mech\->response(); \& for my $key ( $response\->header_field_names() ) { \& print $key, " : ", $response\->header( $key ), "\en"; \& } .Ve .SS "How do I enable keep-alive?" .IX Subsection "How do I enable keep-alive?" Since WWW::Mechanize is a subclass of LWP::UserAgent, you can use the same mechanism to enable keep-alive: .PP .Vb 3 \& use LWP::ConnCache; \& ... \& $mech\->conn_cache(LWP::ConnCache\->new); .Ve .SS "How can I change/specify the action parameter of an \s-1HTML\s0 form?" .IX Subsection "How can I change/specify the action parameter of an HTML form?" You can access the action of the form by utilizing the HTML::Form object returned from one of the specifying form methods. .PP Using \f(CW\*(C`$mech\->form_number($number)\*(C'\fR: .PP .Vb 4 \& my $mech = WWW::mechanize\->new; \& $mech\->get(\*(Aqhttp://someurlhere.com\*(Aq); \& # Access the form using its Zero\-Based Index by DOM order \& $mech\->form_number(0)\->action(\*(Aqhttp://newAction\*(Aq); #ABS URL .Ve .PP Using \f(CW\*(C`$mech\->form_name($number)\*(C'\fR: .PP .Vb 4 \& my $mech = WWW::mechanize\->new; \& $mech\->get(\*(Aqhttp://someurlhere.com\*(Aq); \& #Access the form using its Zero\-Based Index by DOM order \& $mech\->form_name(\*(AqtrgForm\*(Aq)\->action(\*(Aqhttp://newAction\*(Aq); #ABS URL .Ve .SS "How do I save an image? How do I save a large tarball?" .IX Subsection "How do I save an image? How do I save a large tarball?" An image is just content. You get the image and save it. .PP .Vb 2 \& $mech\->get( \*(Aqphoto.jpg\*(Aq ); \& $mech\->save_content( \*(Aq/path/to/my/directory/photo.jpg\*(Aq ); .Ve .PP You can also save any content directly to disk using the \f(CW\*(C`:content_file\*(C'\fR flag to \f(CW\*(C`get()\*(C'\fR, which is part of LWP::UserAgent. .PP .Vb 2 \& $mech\->get( \*(Aqhttp://www.cpan.org/src/stable.tar.gz\*(Aq, \& \*(Aq:content_file\*(Aq => \*(Aqstable.tar.gz\*(Aq ); .Ve .ie n .SS "How do I pick a specific value from a ""\fP list?" .IX Subsection "How do I pick a specific value from a