관리-도구
편집 파일: keepalive.pyc
� Hq�Qc @ s� d Z d d l Z d d l Z d d l Z d d l Z d d l Z d a d d l Z e j d k rl d a n d a d e f d � � YZ d e f d � � YZ d e e j f d � � YZ d e j f d � � YZ d � Z d � Z d e j f d � � YZ d � Z d � a d � Z d � Z d d � Z d � Z d d � Z e d k r�d d l Z d d l Z y$ e e j d � Z e j d Z Wn' e e! f k r�d e j d GHq�Xe e e � n d S( s An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive. >>> import urllib2 >>> from keepalive import HTTPHandler >>> keepalive_handler = HTTPHandler() >>> opener = urllib2.build_opener(keepalive_handler) >>> urllib2.install_opener(opener) >>> >>> fo = urllib2.urlopen('http://www.python.org') If a connection to a given host is requested, and all of the existing connections are still in use, another connection will be opened. If the handler tries to use an existing connection but it fails in some way, it will be closed and removed from the pool. To remove the handler, simply re-run build_opener with no arguments, and install that opener. You can explicitly close connections by using the close_connection() method of the returned file-like object (described below) or you can use the handler methods: close_connection(host) close_all() open_connections() NOTE: using the close_connection and close_all methods of the handler should be done with care when using multiple threads. * there is nothing that prevents another thread from creating new connections immediately after connections are closed * no checks are done to prevent in-use connections from being closed >>> keepalive_handler.close_all() EXTRA ATTRIBUTES AND METHODS Upon a status of 200, the object returned has a few additional attributes and methods, which should not be used if you want to remain consistent with the normal urllib2-returned objects: close_connection() - close the connection to the host readlines() - you know, readlines() status - the return status (i.e. 404) reason - english translation of status (i.e. 'File not found') If you want the best of both worlds, use this inside an AttributeError-catching try: >>> try: status = fo.status >>> except AttributeError: status = None Unfortunately, these are ONLY there if status == 200, so it's not easy to distinguish between non-200 responses. The reason is that urllib2 tries to do clever things with error codes 301, 302, 401, and 407, and it wraps the object upon return. For python versions earlier than 2.4, you can avoid this fancy error handling by setting the module-level global HANDLE_ERRORS to zero. You see, prior to 2.4, it's the HTTP Handler's job to determine what to handle specially, and what to just pass up. HANDLE_ERRORS == 0 means "pass everything up". In python 2.4, however, this job no longer belongs to the HTTP Handler and is now done by a NEW handler, HTTPErrorProcessor. Here's the bottom line: python version < 2.4 HANDLE_ERRORS == 1 (default) pass up 200, treat the rest as errors HANDLE_ERRORS == 0 pass everything up, error processing is left to the calling code python version >= 2.4 HANDLE_ERRORS == 1 pass up 200, treat the rest as errors HANDLE_ERRORS == 0 (default) pass everything up, let the other handlers (specifically, HTTPErrorProcessor) decide what to do In practice, setting the variable either way makes little difference in python 2.4, so for the most consistent behavior across versions, you probably just want to use the defaults, which will give you exceptions on errors. i����Ni i i i t ConnectionManagerc B sG e Z d Z d � Z d � Z d � Z d � Z d � Z d d � Z RS( sV The connection manager must be able to: * keep track of all existing c C s. t j � | _ i | _ i | _ i | _ d S( N( t threadt allocate_lockt _lockt _hostmapt _connmapt _readymap( t self( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt __init__� s c C ss | j j � zQ | | j k r/ g | j | <n | j | j | � | | j | <| | j | <Wd | j j � Xd S( N( R t acquireR t appendR R t release( R t hostt connectiont ready( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt add� s c C s� | j j � zk y | j | } Wn t k r4 nC X| j | =| j | =| j | j | � | j | sw | j | =n Wd | j j � Xd S( N( R R R t KeyErrorR R t removeR ( R R R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR � s c C s) y | | j | <Wn t k r$ n Xd S( N( R R ( R R R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt set_ready� s c C sz d } | j j � zR | | j k rd x< | j | D]* } | j | r3 d | j | <| } Pq3 q3 Wn Wd | j j � X| S( Ni ( t NoneR R R R R ( R R t connt c( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt get_ready_conn� s c C s0 | r t | j j | g � � St | j � Sd S( N( t listR t gett dict( R R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt get_all� s N( t __name__t __module__t __doc__R R R R R R R ( ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR } s t KeepAliveHandlerc B se e Z d � Z d � Z d � Z d � Z d � Z d d � Z d � Z d � Z d � Z d � Z RS( c C s t � | _ d S( N( R t _cm( R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR � s c C s8 g | j j � j � D] \ } } | t | � f ^ q S( st return a list of connected hosts and the number of connections to each. [('foo.com:80', 2), ('bar.org', 1)]( R R t itemst len( R R t li( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt open_connections� s c C s; x4 | j j | � D] } | j j | � | j � q Wd S( s� close connection(s) to <host> host is the host:port spec, as in 'www.cnn.com:8080' as passed in. no error occurs if there is no connection to that host.N( R R R t close( R R t h( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt close_connection� s c C sU xN | j j � j � D]7 \ } } x( | D] } | j j | � | j � q) Wq Wd S( s close all open connectionsN( R R t iteritemsR R$ ( R R t connsR% ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt close_all� s " c C s | j j | d � d S( se tells us that this request is now closed and that the connection is ready for another requesti N( R R ( R t requestR R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt _request_closed� s i c C s' | r | j � n | j j | � d S( N( R$ R R ( R R R R$ ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt _remove_connection� s c C s | j t | � S( N( t do_opent HTTPConnection( R t req( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt http_open� s c C s� | j � } | s$ t j d � � n y� | j j | � } x� | r� | j | | | � } | ra Pn | j � | j j | � | j j | � } q< W| | � } t r� t j d | t | � � n | j j | | d � | j | | � | j � } Wn. t j t j f k r"} t j | � � n X| j r?| j j | � n t rat j d | j | j � n | | _ | | _ | j � | _ | | _ | j | _ | j | _ | j | _ | j d k s�t r�| S| j j d | | | j | j | j � Sd S( Ns no host givens" creating new connection to %s (%d)i s STATUS: %s, %si� t http( t get_hostt urllib2t URLErrorR R t _reuse_connectionR$ R t DEBUGt infot idR t _start_transactiont getresponset sockett errort httplibt HTTPExceptiont will_closet statust reasont _handlert _hostt get_full_urlt _urlt _connectiont codet msgt headerst HANDLE_ERRORSt parent( R t http_classR/ R R% t rt err( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR- � sH c C s� y | j | | � | j � } Wnh t j t j f k rE d } nF t rj t j d | t | � � n | j j | � | j � � n X| d k s� | j d k r� t r� t j d | t | � � n d } n" t r� t j d | t | � � n | S( sG start the transaction with a re-used connection return a response object (r) upon success or None on failure. This DOES not close or remove bad connections in cases where it returns. However, if an unexpected exception occurs, it will close and remove the connection before re-raising. s4 unexpected exception - closing connection to %s (%d)i s& failed to re-use connection to %s (%d)s re-using connection to %s (%d)N( R9 R: R; R< R= R> R R6 R8 R R R$ t versionR7 ( R R% R/ R RM ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR5 s( c C s� | j j � } t j d k r1 | j | j � n | j | j j � t d � | j � D� � } i } x7 d D]/ } | | k rm d | d | j d d � <qm qm Wy� | j � r | j � } | j d | j � | � d | k r� | j d d � n d | k r8| j d d t | � � q8n | j d | j � | � Wn% t j k r`} t j | � � n Xx* | j � D] \ } } | j | | � qnW| j � | j � r�| j | � n d S( Ni i c s s' | ] \ } } | j � | f Vq d S( N( t lower( t .0t nt v( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pys <genexpr>L s R s accept-encodingi t skip_t -t _t POSTs content-types Content-types! application/x-www-form-urlencodeds content-lengths Content-lengths %dt GET( i i ( s hosts accept-encoding( RI t copyt syst version_infot updatet unredirected_hdrsRK t addheadersR R t replacet has_datat get_datat putrequestt get_selectort putheaderR! R; R< R3 R4 t endheaderst send( R R% R/ RI t skipheadersRR t dataRN t kRS ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR9 E s4 ! ( R R R R# R& R) R+ R, R0 R- R5 R9 ( ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR � s 4 .t HTTPHandlerc B s e Z RS( ( R R ( ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyRj d s t HTTPResponsec B sw e Z d d d d � Z e j j Z d � Z d � Z d � Z d � Z d d � Z d � Z d d � Z d d � Z RS( i c C sh t j j | | | | � | j | _ d | _ d | _ d | _ d | _ d | _ d | _ d | _ d S( Nt i� ( R= Rk R t filenoR RG t _rbuft _rbufsizeRB RC RE RF ( R t sockt debuglevelt strictt method( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR { s c C sN | j rJ | j j � d | _ | j rJ | j j | | j | j � qJ n d S( N( t fpR$ R RB R+ RC RF ( R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR$ � s c C s- | j j | j | j d d �| j � d S( NR$ i ( RB R, RC RF R$ ( R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR& � s c C s | j S( N( RI ( R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR7 � s c C s | j S( N( RE ( R ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt geturl� s c C s� | j ra | d k ra t | j � } | | k r= | | 8} qa | j | } | j | | _ | Sn | j | j | � } d | _ | S( NRl ( Rn R R! t _raw_read( R t amtt Lt s( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt read� s c C s� | j } d } x[t rl| d k r� | j j � } | j d � } | d k r[ | | } n y t | d � } Wn* t k r� | j � t j | � � n X| d k r� Pq� n | d k r� | | j | � 7} n� | | k r | | j | � 7} | | | _ | S| | k r9| | j | � 7} | j d � d | _ | S| | j | � 7} | | 8} | j d � d } q Wx3 t r�| j j � } | s�Pn | d k rpPqpqpW| j � | S( NRl t ;i i i s ( t chunk_leftt TrueR Rt t readlinet findt intt ValueErrorR$ R= t IncompleteReadt _safe_read( R Rw R| t valuet linet i( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt _read_chunked� sL i����c C s | j j d � } x� | d k r� d | k oA t | j � k n r� | j | j � } | sc Pn | j d � } | d k r� | t | j � } n | j | | _ q W| d k r� t | j � } n | d } d | k o� t | j � k n r� | } n | j | | j | } | _ | S( Ns i i ( Rn R R! Rv Ro ( R t limitR� t newRh ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR~ � s 5 % c C sf d } g } xS t ra | j � } | s+ Pn | j | � | t | � 7} | r | | k r Pq q W| S( Ni ( R} R~ R R! ( R t sizehintt totalR R� ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt readlines� s N( R R R R R= Rk Rz Rv R$ R& R7 Ru R� R~ R� ( ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyRk g s 8 c C sw t | d d � d k r d S| j d k rM | j rA | j � qM t j � n | j d k rn d Gt | � GHn y� d } t | d d � } | d k r� | j d k r� d GHn | | � } x9 | r� | j j | � | | � } q� Wn | j j | � Wn{ t j k rr} t } | d t j k rc| j t j k rVd | _ | j � | _ t } n | j � n | rs� qsn Xd S( s` Send `str' to the server. Shamelessly ripped off from httplib to patch a bad behavior. t _broken_pipe_respNi s send:i Rz s sending a read()able( t getattrR Rp t auto_opent connectR= t NotConnectedRq t reprt sendallR; R< R} t errnot EPIPEt _HTTPConnection__statet _CS_REQ_SENTR� R: t FalseR$ ( R t strt blocksizeRz Rh RS t reraise( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt safesend s: c s" � f d � } � j j | _ | S( s> Wraps getresponse in cls with a broken-pipe sane version. c s/ t | d d � } | d k r" | S� j | � S( NR� ( R� R R: ( R RM ( t cls( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt safegetresponse9 s ( R: R ( R� R� ( ( R� s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt wrapgetresponse6 s R. c B s# e Z e Z e Z e e j � Z RS( ( R R Rk t response_classR� Rf R� R= R. R: ( ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR. E s c C s! t } t � } t j | � } t j | � i d d 6d d 6} x� d D]� } d | | | f GH| a y^ t j | � } | j � | j � y | j | j } } Wn t k r� d \ } } n XWn t k r� } d | GH� qF Xd | | f GHqF W| a | j � } d G| GH| j � d S( Nt offi t oni s. fancy error handling %s (HANDLE_ERRORS = %i)s EXCEPTION: %ss status = %s, reason = %ss open connections:( i i ( NN( RJ Rj R3 t build_openert install_openert urlopenRz R$ R@ RA t AttributeErrorR t IOErrorR# R) ( t urlt origt keepalive_handlert openert posR� t foR@ RA t et hosts( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt error_handlerP s0 c C sH y d d l m } Wn! t k r7 d d l m } n X| a | | � S( Ni����( t md5( t hashlibR� t ImportError( Ry t _md5( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR� l s c C sD d } t j � } t j | � t j | � } | j � } | j � t j | � } | d | j � f GHt j t � � } t j | � t j | � } | j � } | j � t j | � } | d | j � f GHt j | � } d } x* t r| j � } | r | | } q� Pq� W| j � t j | � } | d | j � f GHd S( Ns %25s: %ss normal urllibs keepalive readRl s keepalive readline( R3 R� R� R� Rz R$ R� R� t hexdigestRj R} R~ ( R� t formatR� R� t foot mt f( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt continuityu s2 c C s� d | | f GHt j j d � t j � } t j | � t | | � } d | GHt j j d � t j t � � } t j | � t | | � } d | GHd | | GHd S( Ns making %i connections to: %ss( first using the normal urllib handlerss TIME: %.3f ss( now using the keepalive handler s improvement factor: %.2f( RZ t stdoutt writeR3 R� R� t fetchRj ( t NR� R� t t1t t2( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt comp� s c C s� d d l } g } | j � } xn t | � D]` } | rS | d k rS | j | � n t j | � } | j � } | j � | j t | � � q+ W| j � | } d } x> | d D]2 } | d } | | d k s� d | | f GHq� q� W| S( Ni����i i s+ WARNING: inconsistent length on read %i: %i( t timet ranget sleepR3 R� Rz R$ R R! ( R� R� t delayR� t lenst starttimeR� R� R� t difft j( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR� � s" c C s t } d t f d � � Y} | � a d GHt j | � } | j � } | j � d } d | GHxH | d k r� t j j d | � t j j � t j d � | d 8} qa Wt j j d � d GHt j | � } | j � } | j � | | k r� d GHn d GH| a d S( Nt FakeLoggerc B s e Z d � Z e Z Z Z RS( c W s | | GHd S( N( ( R RH t args( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt debug� s ( R R R� R7 t warningR< ( ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyR� � s s- fetching the file to establish a connectioni s; waiting %i seconds for the server to close the connectioni s %2ii s s! fetching the file a second times data are identicals ERROR: DATA DIFFER( R6 t objectR3 R� Rz R$ RZ R� R� t flushR� R� t stderr( R� t dbbackupR� R� t data1R� t data2( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt test_timeout� s. i c C sm d GHy t | � Wn t k r5 d GHt j � n XHd GHt | � Hd GHt | | � Hd GHt | � d S( Ns- checking error handler (do this on a non-200)s. exiting - exception will prevent further testss> performing continuity test (making sure stuff isn't corrupted)s performing speed comparisons# performing dropped-connection check( R� R� RZ t exitR� R� R� ( R� R� ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt test� s t __main__s %s <integer> <url>( i i (" R R� R= R; R R3 R R6 RZ R[ RJ R� R R Rj Rk R� R� R. R� R� R� R� R� R� R� R R� R� t argvR� R� t IndexErrorR� ( ( ( s9 /usr/lib64/python2.7/site-packages/mercurial/keepalive.pyt <module>l sB ?� � 3 ! #