3 0fn@srdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z dddddd d d d d ddddddddgZ dZ dZdZdZdZdZejejjddejjjDZdZdZd Zejd!jZejd"jZ ejd#Z!ejd$Z"d%d&d'hZ#dCd)d*Z$Gd+d,d,ej%j&Z'd-d.Z(e'fd/d0Z)Gd1ddej*Z+Gd2ddZ,y ddl-Z-Wne.k rlYnXGd3d4d4e,Z/e j0d4Gd5dde1Z2Gd6dde2Z3Gd7d d e2Z4Gd8dde2Z5Gd9d d e2Z6Gd:d d e2Z7Gd;d d e2Z8Gddde9Z;Gd?dde9ZGdBdde?e=Z@e2ZAdS)Da HTTP/1.1 client library HTTPConnection goes through a number of "states", which define when a client may legally make another request or fetch the response for a particular request. This diagram details these state transitions: (null) | | HTTPConnection() v Idle | | putrequest() v Request-started | | ( putheader() )* endheaders() v Request-sent |\_____________________________ | | getresponse() raises | response = getresponse() | ConnectionError v v Unread-response Idle [Response-headers-read] |\____________________ | | | response.read() | putrequest() v v Idle Req-started-unread-response ______/| / | response.read() | | ( putheader() )* endheaders() v v Request-started Req-sent-unread-response | | response.read() v Request-sent This diagram presents the following rules: -- a second request may not be started until {response-headers-read} -- a response [object] cannot be retrieved until {request-sent} -- there is no differentiation between an unread response body and a partially read response body Note: this enforcement is applied by the HTTPConnection class. The HTTPResponse class does not enforce this state machine, which implies sophisticated clients may accelerate the request/response pipeline. Caution should be taken, though: accelerating the states beyond the above pattern may imply knowledge of the server's connection-close behavior for certain requests. For example, it is impossible to tell whether the server will close the connection UNTIL the response headers have been read; this means that further requests cannot be placed into the pipeline until it is known that the server will NOT be closing the connection. Logical State __state __response ------------- ------- ---------- Idle _CS_IDLE None Request-started _CS_REQ_STARTED None Request-sent _CS_REQ_SENT None Unread-response _CS_IDLE Req-started-unread-response _CS_REQ_STARTED Req-sent-unread-response _CS_REQ_SENT N)urlsplit HTTPResponseHTTPConnection HTTPException NotConnectedUnknownProtocolUnknownTransferEncodingUnimplementedFileModeIncompleteRead InvalidURLImproperConnectionStateCannotSendRequestCannotSendHeaderResponseNotReady BadStatusLine LineTooLongRemoteDisconnectederror responsesPiZUNKNOWNZIdlezRequest-startedz Request-sentcCsi|] }|j|qS)phrase).0vrr/usr/lib64/python3.6/client.py ksriids[^:\s][^:\r\n]*s\n(?![ \t])|\r(?![ \t\n])z[- ]z[-]ZPATCHZPOSTZPUTdatacCsfy |jdStk r`}z:t|j|j|j|jd|j||j|j|fdWYdd}~XnXdS)zZhstringrrr parse_headerss rEcseZdZd@ddZddZddZd d Zd d Zfd dZfddZ ddZ ddZ dAddZ ddZ ddZddZddZdd Zd!d"Zd#d$Zd%d&ZdCd(d)ZdEd*d+ZdGfd,d- Zd.d/Zd0d1Zd2d3ZdHd4d5Zd6d7Zd8d9Zd:d;Zdd?Z Z!S)IrrNcCsR|jd|_||_||_d|_|_t|_t|_t|_ t|_ t|_ t|_ t|_ dS)Nrb)makefiler= debuglevel_methodr>msg_UNKNOWNversionstatusreasonchunked chunk_leftlength will_close)r0sockrHmethodurlrrr__init__s  zHTTPResponse.__init__cCst|jjtdd}t|tkr*td|jdkrBtdt||sNt dy|j dd\}}}WnFt k ry|j dd\}}d}Wnt k rd}YnXYnX|j d s|j t|y$t|}|d ks|d krt|Wnt k rt|YnX|||fS) Nr*z iso-8859-1z status linerzreply:z-Remote end closed connection without responsezHTTP/ri)strr=r:r;r,rrHprintreprrsplit ValueError startswith _close_connrint)r0r2rLrMrNrrr _read_statuss2    zHTTPResponse._read_statusc Cs|jdk rdSx<|j\}}}|tkr(Pt|j}|jdkrFtd|~qW||_|_|j |_ |dkrrd|_ n|j drd|_ nt |t|j|_|_|jdkrx&|jD]}td|d |jj|qW|jjd }|r|jd krd |_d|_nd |_|j|_d|_|jjd}|jjd }|rx|j rxyt||_Wntk rbd|_YnX|jdkr~d|_nd|_|tks|tksd|kodkns|jdkrd|_|j r|j r|jdkrd |_dS)Nrzheaders:HTTP/1.0HTTP/0.9 zHTTP/1. zheader:r)ztransfer-encodingrOTFzcontent-lengthrHEAD)rbrc)r>raZCONTINUEr?r=rHrZcoderMstriprNrLr^rrErJgetr+rOrP _check_closerRrQr`r]Z NO_CONTENTZ NOT_MODIFIEDrI)r0rLrMrNZskipped_headershdrZtr_encrQrrrbegin9s\                   zHTTPResponse.begincCs|jjd}|jdkr:|jjd}|r6d|jkr6dSdS|jjdrJdS|r^d|jkr^dS|jjd}|r~d|jkr~dSdS)NZ connectionrecloseTFz keep-alivezproxy-connection)r>rjrLr+)r0ZconnZpconnrrrrks     zHTTPResponse._check_closecCs|j}d|_|jdS)N)r=rn)r0r=rrrr_szHTTPResponse._close_connc s$ztjWd|jr|jXdS)N)superrnr=r_)r0) __class__rrrnszHTTPResponse.closecstj|jr|jjdS)N)roflushr=)r0)rprrrqs zHTTPResponse.flushcCsdS)zAlways returns TrueTr)r0rrrreadableszHTTPResponse.readablecCs |jdkS)z!True if the connection is closed.N)r=)r0rrrisclosedszHTTPResponse.isclosedc Cs|jdkrdS|jdkr$|jdS|dk rRt|}|j|}t|d|jS|jr`|jS|j dkrv|jj }n6y|j |j }Wnt k r|jYnXd|_ |j|SdS)Nr9rgr) r=rIr_ bytearrayreadinto memoryviewtobytesrO_readall_chunkedrQread _safe_readr )r0amtbr1srrrrys*     zHTTPResponse.readcCs|jdkrdS|jdkr$|jdS|jr4|j|S|jdk r^t||jkr^t|d|j}|jj|}| r~|r~|jn&|jdk r|j|8_|js|j|S)z^Read up to len(b) bytes into bytearray b and return the number of bytes read. Nrrg) r=rIr_rO_readinto_chunkedrQr,rvru)r0r|r1rrrrus$        zHTTPResponse.readintoc Csp|jjtd}t|tkr$td|jd}|dkrB|d|}y t|dStk rj|jYnXdS)Nr*z chunk size;r) r=r:r;r,rfindr`r]r_)r0r2irrr_read_next_chunk_size s    z"HTTPResponse._read_next_chunk_sizecCs>x8|jjtd}t|tkr&td|s,P|dkrPqWdS)Nr*z trailer line r8r9)rr8r9)r=r:r;r,r)r0r2rrr_read_and_discard_trailers z&HTTPResponse._read_and_discard_trailerc Csl|j}|sh|dk r|jdy |j}Wntk rDtdYnX|dkrb|j|jd}||_|S)NrWr9r)rPrzrr]r rr_)r0rPrrr_get_chunk_left(s  zHTTPResponse._get_chunk_leftc Csr|jtkstg}y8x,|j}|dkr(P|j|j|d|_qWdj|Stk rltdj|YnXdS)Nrr9) rOrKAssertionErrorrr/rzrPrAr )r0valuerPrrrrx@s  zHTTPResponse._readall_chunkedc Cs|jtkstd}t|}yvxp|j}|dkr2|St||krZ|j|}|||_||S|d|}|j|}||d}||7}d|_qWWn(tk rtt |d|YnXdS)Nr) rOrKrrvrr,_safe_readintorPr bytes)r0r| total_bytesmvbrPr1temp_mvbrrrr~Ns&      zHTTPResponse._readinto_chunkedcCsXg}xH|dkrL|jjt|t}|s4tdj|||j||t|8}qWdj|S)aVRead the number of bytes requested, compensating for partial reads. Normally, we have a blocking socket, but a read() can be interrupted by a signal (resulting in a partial read). Note that we cannot distinguish between EOF and an interrupt when zero bytes have been read. IncompleteRead() will be raised in this situation. This function should be used when bytes "should" be present for reading. If the bytes are truly not available (due to EOF), then the IncompleteRead exception can be used to detect the problem. rr9)r=rymin MAXAMOUNTr rAr/r,)r0r{r}chunkrrrrzfs  zHTTPResponse._safe_readcCsd}t|}xt|t|krtt|kr@|dt}|jj|}n |jj|}|sjtt|d|t|||d}||7}qW|S)z2Same as _safe_read, but for reading into a buffer.rN)rvr,rr=rur r)r0r|rrrr1rrrr}s     zHTTPResponse._safe_readintor*c Cs|jdks|jdkrdS|jr(|j|S|jdk rJ|dksD||jkrJ|j}y|jj|}Wn*tk r|dkrt|jjd}YnX| r|r|jn|jdk r|jt|8_|S)zvRead with at most one underlying system call. If at least one byte is buffered, return that instead. Nrgr9rrii@) r=rIrO_read1_chunkedrQread1r]r_r,)r0r1resultrrrrs"    zHTTPResponse.read1cCs4|jdks|jdkrdS|jr(|j|S|jj|S)Nrgr9)r=rIrO _peek_chunkedpeek)r0r1rrrrs  zHTTPResponse.peekcs|jdks|jdkrdS|jr*tj|S|jdk rL|dksF||jkrL|j}|jj|}| rl|rl|jn|jdk r|jt|8_|S)Nrgr9r)r=rIrOror:rQr_r,)r0limitr)rprrr:s     zHTTPResponse.readlinecCsf|j}|dks|dkrdSd|ko.|kns8|}|jj|}|jt|8_|sbtd|S)Nrr9)rr=rrPr,r )r0r1rPryrrrrs zHTTPResponse._read1_chunkedc CsBy |j}Wntk r dSX|dkr.dS|jj|d|S)Nr9)rr r=r)r0r1rPrrrrs zHTTPResponse._peek_chunkedcCs |jjS)N)r=fileno)r0rrrrszHTTPResponse.filenocCsH|jdkrt|jj|p|}t|ts6t|d r:|Sdj|SdS)axReturns the value of the header matching *name*. If there are multiple matching headers, the values are combined into a single string separated by commas and spaces. If no matching header is found, returns *default* or None if the *default* is not specified. If the headers are unknown, raises http.client.ResponseNotReady. N__iter__z, )r>rZget_all isinstancerYhasattrrA)r0r%defaultr>rrr getheaders zHTTPResponse.getheadercCs|jdkrtt|jjS)z&Return list of (header, value) tuples.N)r>rlistitems)r0rrr getheaderss zHTTPResponse.getheaderscCs|S)Nr)r0rrrrszHTTPResponse.__iter__cCs|jS)ajReturns an instance of the class mimetools.Message containing meta-information associated with the URL. When the method is HTTP, these headers are those returned by the server at the head of the retrieved HTML page (including Content-Length and Content-Type). When the method is FTP, a Content-Length header will be present if (as is now usual) the server passed back a file length in response to the FTP retrieval request. A Content-Type header will be present if the MIME type can be guessed. When the method is local-file, returned headers will include a Date representing the file's last-modified time, a Content-Length giving file size, and a Content-Type containing a guess at the file's type. See also the description of the mimetools module. )r>)r0rrrinfoszHTTPResponse.infocCs|jS)aZReturn the real URL of the page. In some cases, the HTTP server redirects a client to another URL. The urlopen() function handles this transparently, but in some cases the caller needs to know which URL the client was redirected to. The geturl() method can be used to get at this redirected URL. )rU)r0rrrgeturls zHTTPResponse.geturlcCs|jS)zuReturn the HTTP status code that was sent with the response, or None if the URL is not an HTTP URL. )rM)r0rrrgetcodeszHTTPResponse.getcode)rNN)N)rr)rr)r)N)"r4r5r6rVrarmrkr_rnrqrrrsryrurrrrxr~rzrrrr:rrrrrrrrr __classcell__rr)rprrs< !K  "     c@seZdZdZdZeZeZdZ dZ e ddZ e ddZ d ejd fd d Zd0d d ZddZddZddZddZddZddZddZddZd1dd Zd2d!d"Zd#d$Zd%d&Zd3dd'd(d)Zd ifdd'd*d+Zd,d-Z d.d/Z!d S)4rrezHTTP/1.1r*rcCs t|tjS)zFTest whether a file-like object is a text or a binary stream. )rio TextIOBase)streamrrr _is_textIO0szHTTPConnection._is_textIOc Csd|dkr|jtkrdSdSt|dr*dSyt|}|jStk rLYnXt|tr`t|SdS)aGet the content-length based on the body. If the body is None, we set Content-Length: 0 for methods that expect a body (RFC 7230, Section 3.3.2). We also set the Content-Length for any method if the body is a str or bytes-like object and not a file. Nrry) upper_METHODS_EXPECTING_BODYrrvnbytes TypeErrorrrYr,)bodyrTZmvrrr_get_content_length6s   z"HTTPConnection._get_content_lengthNcCs\||_||_d|_g|_d|_t|_d|_d|_d|_ i|_ |j ||\|_ |_ tj|_dS)N)timeoutsource_addressrS_buffer_HTTPConnection__response_CS_IDLE_HTTPConnection__staterI _tunnel_host _tunnel_port_tunnel_headers _get_hostporthostportsocketZcreate_connection_create_connection)r0rrrrrrrrVVszHTTPConnection.__init__cCs<|jrtd|j||\|_|_|r.||_n |jjdS)aDSet up host and port for HTTP CONNECT tunnelling. In a connection that uses HTTP CONNECT tunneling, the host passed to the constructor is used as a proxy server that relays all communication to the endpoint passed to `set_tunnel`. This done by sending an HTTP CONNECT request to the proxy server when the connection is established. This method must be called before the HTML connection has been established. The headers argument should be a mapping of extra HTTP headers to send with the CONNECT request. z.Can't set up tunnel for established connectionN)rS RuntimeErrorrrrrclear)r0rrr>rrr set_tunnelis zHTTPConnection.set_tunnelc Cs|dkr|jd}|jd}||kryt||dd}WnHtk r||dddkrh|j}ntd||ddYnX|d|}n|j}|r|ddkr|ddkr|dd }||fS) Nr)]r*rXznonnumeric port: '%s'r[rr)rfindr`r] default_portr )r0rrrjrrrrs   zHTTPConnection._get_hostportcCs ||_dS)N)rH)r0levelrrrset_debuglevelszHTTPConnection.set_debuglevelc Csd|j|jf}|jd}|j|x6|jjD](\}}d||f}|jd}|j|q0W|jd|j|j|jd}|j \}} } | t j j kr|j td| | jfxP|jjtd} t| tkrtd | sP| dkrP|jd krtd | jqWdS)NzCONNECT %s:%d HTTP/1.0 asciiz%s: %s zlatin-1 )rTzTunnel connection failed: %d %sr*z header liner8r9rzheader:)rr8r9)rrrsendrrresponse_classrSrIrahttp HTTPStatusZOKrnOSErrorrir=r:r;r,rrHrZrB) r0Z connect_strZ connect_bytesheaderrZ header_strZ header_bytesresponserLrhmessager2rrr_tunnels2         zHTTPConnection._tunnelcCsB|j|j|jf|j|j|_|jjtjtj d|j r>|j dS)z3Connect to the host and port specified in __init__.r*N) rrrrrrSZ setsockoptrZ IPPROTO_TCPZ TCP_NODELAYrr)r0rrrconnects zHTTPConnection.connectc CsBt|_z|j}|r d|_|jWd|j}|rrrrrrszHTTPConnection.requestc Cstdd|D}i}d|kr&d|d<d|kr6d|d<|j||f|d|krd |krd }|j||}|dkr|dk r|jd krtd |d }|jddq|jdt|nd }x |jD]\} } |j| | qWt|trt |d}|j ||ddS)Ncss|]}|jVqdS)N)r+)rkrrr sz/HTTPConnection._send_request..rr*rzaccept-encodingrzcontent-lengthztransfer-encodingFrzUnable to determine size of %rTzTransfer-EncodingrOzContent-Lengthr)r) frozensetrrrHrZrrYrrr'r) r0rTrUrr>rZ header_namesZskipsZcontent_lengthrlrrrrrs0      zHTTPConnection._send_requestcCs|jr|jjrd|_|jtks&|jr0t|j|jdkrR|j|j|j|jd}n|j|j|jd}yZy |j Wnt k r|j YnX|j t kstt|_|j r|j n||_|S|j YnXdS)a)Get the response from the server. If the HTTPConnection is in the correct state, returns an instance of HTTPResponse or of whatever object is returned by the response_class variable. If a request has not been sent or if a previous response has not be handled, ResponseNotReady is raised. If the HTTP response indicates that the connection should be closed, then it will be closed before the response is returned. When the connection is closed, the underlying socket is closed. Nr)rT)rrsrrrrHrrSrIrmConnectionErrorrnrRrKrr)r0rrrr getresponse)s.      zHTTPConnection.getresponse)NN)NF)FF)N)"r4r5r6rrrr HTTP_PORTrrrH staticmethodrrr_GLOBAL_DEFAULT_TIMEOUTrVrrrrrrnrrrrrrrrrrrrrrrr&s<     ' 6   .csFeZdZdZeZdddejdfdddfdd ZfddZ Z S)HTTPSConnectionz(This class allows communication via SSL.N)contextcheck_hostnamec stt|j|||||dk s.|dk s.|dk rDddl} | jdtd||_||_|dkr`tj }|j tj k} |dkrz|j }|r| rt d|s|r|j||||_||_dS)NrzTkey_file, cert_file and check_hostname are deprecated, use a custom context instead.rWzMcheck_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED)rorrVwarningswarnDeprecationWarningkey_file cert_filesslZ_create_default_https_contextZ verify_modeZ CERT_NONErr]Zload_cert_chain_context_check_hostname) r0rrr r rrrrrZ will_verify)rprrrVts(   zHTTPSConnection.__init__c stj|jr|j}n|j}|jj|j|d|_|jj r|jryt j |jj |Wn.t k r|jj tj|jjYnXdS)z(Connect to a host on a given (SSL) port.)server_hostnameN)rorrrr Z wrap_socketrSrr r Zmatch_hostnameZ getpeercert ExceptionZshutdownrZ SHUT_RDWRrn)r0r)rprrrs    zHTTPSConnection.connect) r4r5r6__doc__ HTTPS_PORTrrrrVrrrr)rprrmsrc@s eZdZdS)rN)r4r5r6rrrrrsc@s eZdZdS)rN)r4r5r6rrrrrsc@s eZdZdS)r N)r4r5r6rrrrr sc@seZdZddZdS)rcCs|f|_||_dS)N)argsrL)r0rLrrrrVszUnknownProtocol.__init__N)r4r5r6rVrrrrrsc@s eZdZdS)rN)r4r5r6rrrrrsc@s eZdZdS)r N)r4r5r6rrrrr sc@s&eZdZdddZddZddZdS) r NcCs|f|_||_||_dS)N)rpartialexpected)r0rrrrrrVszIncompleteRead.__init__cCs2|jdk rd|j}nd}d|jjt|j|fS)Nz, %i more expectedrXz%s(%i bytes read%s))rrpr4r,r)r0errr__repr__s   zIncompleteRead.__repr__cCst|S)N)r[)r0rrr__str__szIncompleteRead.__str__)N)r4r5r6rVrrrrrrr s c@s eZdZdS)r N)r4r5r6rrrrr sc@s eZdZdS)r N)r4r5r6rrrrr sc@s eZdZdS)rN)r4r5r6rrrrrsc@s eZdZdS)rN)r4r5r6rrrrrsc@seZdZddZdS)rcCs|s t|}|f|_||_dS)N)r[rr2)r0r2rrrrVszBadStatusLine.__init__N)r4r5r6rVrrrrrsc@seZdZddZdS)rcCstj|dt|fdS)Nz&got more than %d bytes when reading %s)rrVr;)r0Z line_typerrrrVszLineTooLong.__init__N)r4r5r6rVrrrrrsc@seZdZddZdS)rcOs"tj|dtj|f||dS)NrX)rrVConnectionResetError)r0poskwrrrrVs zRemoteDisconnected.__init__N)r4r5r6rVrrrrrs)r)BrZ email.parserrCZ email.messagerrosrerrZ urllib.parser__all__rrrKrrrglobalsupdater __members__rrrr;r<compile fullmatchrrrrrrr'rZMessager(r?rEBufferedIOBaserrr  ImportErrorrr/rrrr rrr r r r rrrrrrrrrrrEs        9F 6