[Th60dZddlZddlmZmZddlmZmZddlmZddl m Z ddl m Z dd l m Zdd l mZdd l mZdd l mZej"d ZGd dZdedej*j,j.defdZdedefdZGddZy)z)[Experimental] Text Only Local Tokenizer.N)AnyIterable)OptionalUnion)sentencepiece_model_pb2)_common)_local_tokenizer_loader) _transformers)types) t_contentszgoogle_genai.local_tokenizerceZdZdZddZdeefdZdeejddfdZ dejddfd Z d ejddfd Z d ejdejfd ZdeejddfdZdeej"ddfdZdej"ddfdZdej(dej(fdZdej,dej,fdZdeeefdeeeffdZdedefdZy)_TextsAccumulatoraAccumulates countable texts from `Content` and `Tool` objects. This class is responsible for traversing complex `Content` and `Tool` objects and extracting all the text content that should be included when calculating token counts. A key feature of this class is its ability to detect unsupported fields in `Content` objects. If a user provides a `Content` object with fields that this local tokenizer doesn't recognize (e.g., new fields added in a future API update), this class will log a warning. The detection mechanism for `Content` objects works by recursively building a "counted" version of the input object. This "counted" object only contains the data that was successfully processed and added to the text list for tokenization. After traversing the input, the original `Content` object is compared to the "counted" object. If they don't match, it signifies the presence of unsupported fields, and a warning is logged. returnNcg|_yN_textsselfs a/home/www/backend.miabetepe.com/venv/lib/python3.12/site-packages/google/genai/local_tokenizer.py__init__z_TextsAccumulator.__init__6s DKc|jSrrrs r get_textsz_TextsAccumulator.get_texts9s ;;rcontentsc4|D]}|j|yr) add_content)rrcontents r add_contentsz_TextsAccumulator.add_contents<s  w rrchtjg|j}|jrB|jD]2}|jJtj}|j |j td|j|j|_|j,|j|j|j|_ |j,|j|j|j|_ |j6|j|_ |jj|j|jj|5|j!d|j!dk7rt"j%d|d|dyy)N)partsrolez6LocalTokenizers do not support non-text content types.T) exclude_nonezHContent contains unsupported types for token counting. Supported fields z. Got .)r Contentr#r"Part file_data inline_data ValueErrorvideo_metadata function_calladd_function_callfunction_responseadd_function_responsetextrappend model_dumploggerwarning)rrcounted_contentpart counted_parts rrz_TextsAccumulator.add_content@smm"7<<@O}}--3$$$000zz| >> %)9)9)EF      *(,(;(;, %    )  !3!3 4'+'9'9, $  ! ! -  $ $T%;%; <+/+A+A, ( 99 "ii,  ++  TYY '$$\2%3(t,0J0J1K1 nn $%VG9A 7rr,c|jr%|jj|jtj|j}|j r#|j |j }||_yy)zProcesses a function call and adds relevant text to the accumulator. Args: function_call: The function call to process. )nameN)r9rr1r FunctionCallargs_dict_traverse)rr,counted_function_call counted_argss rr-z#_TextsAccumulator.add_function_call_si  kk++,!..M4F4FG((););"+ ''rschemactj}|jr|j|_|jr|j|_|j|j|_|j r6|j j|j |j |_|jr6|j j|j|j|_|jr6|j j|j|j|_ |jr6|j j|j|j|_ |jr|j|_ |jr"|j|j}||_ |jr\i}|jjD]6\}}|j j||j|}|||<8||_|j r"|j#|j }||_|S)zProcesses a schema and adds relevant text to the accumulator. Args: schema: The schema to process. Returns: The new schema object with only countable fields. )r Schematypetitledefaultformatrr1rUenumextendrequiredproperty_orderingitemsrW propertiesexample _any_traverse) rrYcounted_schemacounted_schema_itemsdkeyvalue counted_valuecounted_schema_examples rrWz_TextsAccumulator.add_schemas\\^N {{"KKn ||#\\n ~~!%~~n }} kk'$mmn  kk++,#)#5#5n  {{ kk%"KKn  kk) &n )/)A)An& ||!__V\\:1n  a))//1*#u 3. ##$n ~~#11&..A5n rrjci}|jjt|j|j D]\}}|j |||<|S)zProcesses a dict and adds relevant text to the accumulator. Args: d: The dict to process. Returns: The new dict object with only countable fields. )rralistkeysrdrg)rrj counted_dictrkvals rr<z _TextsAccumulator._dict_traversesYLKKtAFFH~&GGI2S,,S1l32 rrlc t|tr|jj||St|tr|j |St|t r|Dcgc]}|j|c}S|Scc}w)zProcesses a value and adds relevant text to the accumulator. Args: value: The value to process. Returns: The new value with only countable fields. ) isinstancestrrr1dictr<rprg)rrlitems rrgz_TextsAccumulator._any_traversesp% kk l E4   '' E4 38 94d  & 99 l:s#B)rN)__name__ __module__ __qualname____doc__rrrvrr r&r rr:r-rBrGrJrOrMr/rTrCr[rWrwrr<rgrrrr"sM& # 8EMM#: t 4> 0U-?-? 0D 0 5:: %** Xejj1d4 ()?)? @4 4 <$55 <  <("'";";(   ((+u||+ +Z d38n c3h rrtokenr\rc|tjjjjk(rt |j ddS|jddjdS)Nrbig)length byteorderu▁ zutf-8) r ModelProto SentencePieceTypeBYTE_parse_hex_byteto_bytesreplaceencode)r~r\s r_token_str_to_bytesrs\ $ / / = = B B G GG 5 ! * *!u * EE == $ + +G 44rct|dk7rtd||jdr|jdstd| t |ddd}|d k\rtd ||S#t$rtd |wxYw) zParses a hex byte string of the form '<0xXX>' and returns the integer value. Raises ValueError if the input is malformed or the byte value is invalid. zInvalid byte length: z<0x>zInvalid byte format: zInvalid hex value: zByte value out of range: )lenr* startswithendswithint)r~rss rrrs  Z1_ ,UG4 55   % s(; ,UG4 554 eAaj" C CZ 08 99 * 4 *5'2 334s A22B c@eZdZdZdefdZejddddee je jfde e jd e jfd Zejddee je jfd e j fd Zy) LocalTokenizera[Experimental] Text Only Local Tokenizer. This class provides a local tokenizer for text only token counting. LIMITATIONS: - Only supports text based tokenization and no multimodal tokenization. - Forward compatibility depends on the open-source tokenizer models for future Gemini versions. - For token counting of tools and response schemas, the `LocalTokenizer` only supports `types.Tool` and `types.Schema` objects. Python functions or Pydantic models cannot be passed directly. model_namectj||_tj|j|_tj |j|_yr)loaderget_tokenizer_name_tokenizer_nameload_model_proto _model_protoget_sentencepiece _tokenizer)rrs rrzLocalTokenizer.__init__sF!44Z@D//0D0DED..t/C/CDDOrzThe SDK's local tokenizer implementation is experimental and may change in the future. It only supports text based tokenization.N)configrrrctj|}t}tjj |xsi}|j ||jr|j|j|jr;|jjr%|j|jj|jr/|j tj|jg|jjt|j!}tj"t%d|DS)aCounts the number of tokens in a given text. Args: contents: The contents to tokenize. config: The configuration for counting tokens. Returns: A `CountTokensResult` containing the total number of tokens. Usage: .. code-block:: python from google import genai tokenizer = genai.LocalTokenizer(model_name='gemini-2.0-flash-001') result = tokenizer.count_tokens("What is your name?") print(result) # total_tokens=5 c32K|]}t|ywr)r).0tokenss r z.LocalTokenizer.count_tokens..Ms?V?s) total_tokens)tr rr CountTokensConfigmodel_validater rHrJgeneration_configresponse_schemarWsystem_instructionrrrprCountTokensResultsum)rrrprocessed_contentstext_accumulator tokens_lists r count_tokenszLocalTokenizer.count_tokens$s:h/(*  $ $ 3 3FLb AF!!"45 ||  . F$<$<$L$L!!&":":"J"JK   ##ALL&2K2K1L$MN//((.>.H.H.J)KLK  " "?;?? rctj|}t}|D]}|j||jj |j }g}|D];}|js|jD]}|j|j=g}t||D]\} } |jtj| jD cgc]} | jc} | jD cgc]C} t| j |j"j| jj$Ec} | tj&|Scc} wcc} w)a,Computes the tokens ids and string pieces in the input. Args: contents: The contents to tokenize. Returns: A `ComputeTokensResult` containing the token information. Usage: .. code-block:: python from google import genai tokenizer = genai.LocalTokenizer(model_name='gemini-2.0-flash-001') result = tokenizer.compute_tokens("What is your name?") print(result) # tokens_info=[TokensInfo(token_ids=[279, 329, 1313, 2508, 13], tokens=[b' What', b' is', b' your', b' name', b'?'], role='user')] ) token_idsrr#) tokens_info)rr rrrEncodeAsImmutableProtorr"r1r#zipr TokensInfopiecesidrpiecerr\ComputeTokensResult) rrrrr tokens_protosroles_ token_infos tokens_protor#rs rcompute_tokenszLocalTokenizer.compute_tokensPs[4h/(*%,""7+,OO::""$M E%%  %A ,,w|| $ %% K!-7  d   /;/B/BCeC ,22 &kk4#4#4#;#;EHH#E#J#J      $ $ ==Ds E/AE4)ryrzr{r|rvrr experimental_warningrr ContentListUnionContentListUnionDictrCountTokensConfigOrDictrrrrr}rrrrs EE  7B9= &e,,e.H.HHI&u445 &   & &P 7B2>e,,e.H.HHI2>   2> 2>rr)r|loggingtypingrrrr sentencepiecerr r rr rr r getLoggerr3rrvrrrbytesrrrrr}rrrs 0 "1/ %   9 :MM`5 5-88FFKK5 5 3 3 ,u>u>r