l$eI ! d dl mZ d dlZd dlZd dlZd dlmZ d dlm Z d dl
m Z m Z m
Z
mZmZ d dlmZmZ d dlmZmZmZmZmZ ddlmZ dd lm Z dd
l m Z m Z m Z e j! e" Z# e rd dl$Z%d,d Z&d
Z'd Z( d-dede)dee* fdZ+ d.dee,ef dee e,e f de)de)deee-e,f f
dZ.d/d Z/e dd ddddddddddddd d e,dee* d e,d!e)d"ee, d#ee, d$ee, d%ee) d&eee
e, e,f d'eee
e, e,f d(eee
e, e,f d)ee, de)deee-e,f de)f d* Z0 G d+ de Z1dS )0 N)Path)copytree)AnyDictListOptionalUnion)
ModelHubMixinsnapshot_download)get_tf_versionis_graphviz_availableis_pydot_availableis_tf_available yaml_dump ) CONFIG_NAME)HfApi)SoftTemporaryDirectoryloggingvalidate_hf_hub_args c < g }| D ]w\ }}|r| d| n|}t |t j r6| t || `| ||f xt | S )a Flatten a nested dictionary.
Reference: https://stackoverflow.com/a/6027615/10319735
Args:
dictionary (`dict`):
The nested dictionary to be flattened.
parent_key (`str`):
The parent key to be prefixed to the children keys.
Necessary for recursing over the nested dictionary.
Returns:
The flattened dictionary.
.)items
isinstance collectionsMutableMappingextend
_flatten_dictappenddict)
dictionary
parent_keyr keyvaluenew_keys ;lib/python3.11/site-packages/huggingface_hub/keras_mixin.pyr r s
E &&((
+
+
U+5>Z''#'''3 e[ 7 8 8 + LL %''
LL'5) * * * * ;; c | j || j }t | }t j j j |d< d}| D ]\ }}|d| d| dz
}nd}|S )z6Parse hyperparameter dictionary into a markdown table.Ntraining_precisionz*| Hyperparameters | Value |
| :-- | :-- |
z| z | z |
) optimizer
get_configr tfkerasmixed_precision
global_policynamer )modeloptimizer_paramstabler$ r% s r' _create_hyperparameter_tabler5 : s " ?5577()9::131I1W1W1Y1Y1^-.>*0022 . .JC -#--%--- -EE . Lr( c f t j j | | dddddddd d S )N
/model.pngFTTB` )to_file show_shapes
show_dtypeshow_layer_namesrankdir
expand_nesteddpi layer_range)r- r. utils
plot_model)r2 save_directorys r'
_plot_networkrE I sQ HN
!---
r( Trepo_dirrC metadatac t | }|r,t r t rt | | |i }| d}d|d< d}|t |d z
}|dz
}|dz
}|d z
}|d
z
}||d z
}|d z
}|d
z
}||z
}|dz
}|rAt
j | d r |dz
}|dz
}|dz
}d}|d| dz
}|dz
}t
j | r?t |dd 5 }| } ddd n # 1 swxY w Y n|} t |dd 5 }|
| ddd dS # 1 swxY w Y dS ) z2
Creates a model card for the repository.
Nz
/README.mdr. library_namez---
F)default_flow_stylez/
## Model description
More information needed
z9
## Intended uses & limitations
More information needed
z:
## Training and evaluation data
More information needed
z
## Training procedure
z
### Training hyperparameters
z;
The following hyperparameters were used during training:
r7 z
## Model Plot
z
z$
View Model Plot
z ./model.pngz
z
rutf8encodingwutf-8) r5 r
r rE r ospathexistsopenreadwrite)
r2 rF rC rG hyperparameters readme_path
model_card path_to_plotfreadmes
r' _create_model_cardr^ W sl 3599O '+-- '2D2F2F 'eX&&& )))K &H^ J)H????J'JGGJQQJRRJ"11
88
VV
o%
d
%bgnn%<%<%<== %**
m#
>>
$ ;,;;;;
n$
w~~k""
+sV
4
4
4 VVXXF
k3 1 1 1 Q s$ D&&D*-D*E))E-0E-FrD configinclude_optimizertagsc t t rddl}nt d | j st d t | }| dd |rt |t s t dt | d |t z d 5 }t j || ddd n # 1 swxY w Y i } t |t r|| d
< nt |t r|g| d
< | d d }
|
@t% j d t( d
| v r | d
|
n|
g| d
< | j | j j i k r|d
z } | rt% j dt0 | d d 5 }t j | j j |dd ddd n # 1 swxY w Y t3 | ||| |j j j | |fd|i| dS )aE
Saves a Keras model to save_directory in SavedModel format. Use this if
you're using the Functional or Sequential APIs.
Args:
model (`Keras.Model`):
The [Keras
model](https://www.tensorflow.org/api_docs/python/tf/keras/Model)
you'd like to save. The model must be compiled and built.
save_directory (`str` or `Path`):
Specify directory in which you want to save the Keras model.
config (`dict`, *optional*):
Configuration object to be saved alongside the model weights.
include_optimizer(`bool`, *optional*, defaults to `False`):
Whether or not to include optimizer in serialization.
plot_model (`bool`, *optional*, defaults to `True`):
Setting this to `True` will plot the model and put it in the model
card. Requires graphviz and pydot to be installed.
tags (Union[`str`,`list`], *optional*):
List of tags that are related to model or string of a single tag. See example tags
[here](https://github.com/huggingface/hub-docs/blame/main/modelcard.md).
model_save_kwargs(`dict`, *optional*):
model_save_kwargs will be passed to
[`tf.keras.models.save_model()`](https://www.tensorflow.org/api_docs/python/tf/keras/models/save_model).
r Nz>Called a Tensorflow-specific function but could not import it.z+Model should be built before trying to saveT)parentsexist_okzAProvided config to save_pretrained_keras should be a dict. Got: ''rP ra task_namez>`task_name` input argument is deprecated. Pass `tags` instead.z history.jsonzZ`history.json` file already exists, it will be overwritten by the history of this version.rQ rN )indent sort_keysr` ) r
tensorflow ImportErrorbuilt
ValueErrorr mkdirr r! RuntimeErrortyper rU jsondumpliststrpopwarningswarn
FutureWarningr historyrT UserWarningr^ r. models
save_model) r2 rD r_ r` rC ra model_save_kwargsr- r\ rG rf rS s r' save_pretrained_kerasr~ s D \ Z[[[ ; HFGGG.))N555 !&$'' t rcghncoco r r rss s {*
0
0
5
5 ! Ifa ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! H$ " D# " 6!%%k488I
L
X V # #I . . . . ){HV } = B & &!N2D{{}}
p 311
NQ %-/1MMMM
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N unj(CCC BHO unooHYo]nooooos$ 3CC C#H HHreturnKerasModelHubMixinc $ t j | i |S )a
Instantiate a pretrained Keras model from a pre-trained model from the Hub.
The model is expected to be in `SavedModel` format.
Args:
pretrained_model_name_or_path (`str` or `os.PathLike`):
Can be either:
- A string, the `model id` of a pretrained model hosted inside a
model repo on huggingface.co. Valid model ids can be located
at the root-level, like `bert-base-uncased`, or namespaced
under a user or organization name, like
`dbmdz/bert-base-german-cased`.
- You can add `revision` by appending `@` at the end of model_id
simply like this: `dbmdz/bert-base-german-cased@main` Revision
is the specific model version to use. It can be a branch name,
a tag name, or a commit id, since we use a git-based system
for storing models and other artifacts on huggingface.co, so
`revision` can be any identifier allowed by git.
- A path to a `directory` containing model weights saved using
[`~transformers.PreTrainedModel.save_pretrained`], e.g.,
`./my_model_directory/`.
- `None` if you are both providing the configuration and state
dictionary (resp. with keyword arguments `config` and
`state_dict`).
force_download (`bool`, *optional*, defaults to `False`):
Whether to force the (re-)download of the model weights and
configuration files, overriding the cached versions if they exist.
resume_download (`bool`, *optional*, defaults to `False`):
Whether to delete incompletely received files. Will attempt to
resume the download if such a file exists.
proxies (`Dict[str, str]`, *optional*):
A dictionary of proxy servers to use by protocol or endpoint, e.g.,
`{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The
proxies are used on each request.
token (`str` or `bool`, *optional*):
The token to use as HTTP bearer authorization for remote files. If
`True`, will use the token generated when running `transformers-cli
login` (stored in `~/.huggingface`).
cache_dir (`Union[str, os.PathLike]`, *optional*):
Path to a directory in which a downloaded pretrained model
configuration should be cached if the standard cache should not be
used.
local_files_only(`bool`, *optional*, defaults to `False`):
Whether to only look at local files (i.e., do not try to download
the model).
model_kwargs (`Dict`, *optional*):
model_kwargs will be passed to the model during initialization
Passing `token=True` is required when you want to use a private
model.
)r from_pretrained)argskwargss r' from_pretrained_kerasr s p -t >v > >>r( z'Push Keras model using huggingface_hub.)r_ commit_messageprivate api_endpointtokenbranch create_prallow_patternsignore_patternsdelete_patternslog_dirr` ra rC repo_idr r r r r r r r r r c
t | }| |||d j }t 5 }t | |z }t | |f||
||d| | F| g nt
| t r| gn| } | d t | |dz |
d||||||| |
|
cddd S # 1 swxY w Y dS )
a
Upload model checkpoint to the Hub.
Use `allow_patterns` and `ignore_patterns` to precisely filter which files should be pushed to the hub. Use
`delete_patterns` to delete existing remote files in the same commit. See [`upload_folder`] reference for more
details.
Args:
model (`Keras.Model`):
The [Keras model](`https://www.tensorflow.org/api_docs/python/tf/keras/Model`) you'd like to push to the
Hub. The model must be compiled and built.
repo_id (`str`):
ID of the repository to push to (example: `"username/my-model"`).
commit_message (`str`, *optional*, defaults to "Add Keras model"):
Message to commit while pushing.
private (`bool`, *optional*, defaults to `False`):
Whether the repository created should be private.
api_endpoint (`str`, *optional*):
The API endpoint to use when pushing the model to the hub.
token (`str`, *optional*):
The token to use as HTTP bearer authorization for remote files. If
not set, will use the token set when logging in with
`huggingface-cli login` (stored in `~/.huggingface`).
branch (`str`, *optional*):
The git branch on which to push the model. This defaults to
the default branch as specified in your repository, which
defaults to `"main"`.
create_pr (`boolean`, *optional*):
Whether or not to create a Pull Request from `branch` with that commit.
Defaults to `False`.
config (`dict`, *optional*):
Configuration object to be saved alongside the model weights.
allow_patterns (`List[str]` or `str`, *optional*):
If provided, only files matching at least one pattern are pushed.
ignore_patterns (`List[str]` or `str`, *optional*):
If provided, files matching any of the patterns are not pushed.
delete_patterns (`List[str]` or `str`, *optional*):
If provided, remote files matching any of the patterns will be deleted from the repo.
log_dir (`str`, *optional*):
TensorBoard logging directory to be pushed. The Hub automatically
hosts and displays a TensorBoard instance if log files are included
in the repository.
include_optimizer (`bool`, *optional*, defaults to `False`):
Whether or not to include optimizer during serialization.
tags (Union[`list`, `str`], *optional*):
List of tags that are related to model or string of a single tag. See example tags
[here](https://github.com/huggingface/hub-docs/blame/main/modelcard.md).
plot_model (`bool`, *optional*, defaults to `True`):
Setting this to `True` will plot the model and put it in the model
card. Requires graphviz and pydot to be installed.
model_save_kwargs(`dict`, *optional*):
model_save_kwargs will be passed to
[`tf.keras.models.save_model()`](https://www.tensorflow.org/api_docs/python/tf/keras/models/save_model).
Returns:
The url of the commit of your model in the given repository.
)endpointT)r r r rd )r_ r` ra rC Nzlogs/*logsr2 )
repo_typer folder_pathr r revisionr r r r ) r create_repor r r r~ r rt r r
upload_folder)r2 r r_ r r r r r r r r r r r` ra rC r} apitmp
saved_paths r' push_to_hub_kerasr s \
&
&
&CoogUGVZo[[cG
! ! %
S#YY(
/!
#* "/377)_%%(
" "8 , , , Wj6 1 2 2 2 ") ) + + !
5%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
s B CC Cc . e Zd ZdZd Zed ZdS )r aA
Implementation of [`ModelHubMixin`] to provide model Hub upload/download
capabilities to Keras models.
```python
>>> import tensorflow as tf
>>> from huggingface_hub import KerasModelHubMixin
>>> class MyModel(tf.keras.Model, KerasModelHubMixin):
... def __init__(self, **kwargs):
... super().__init__()
... self.config = kwargs.pop("config", None)
... self.dummy_inputs = ...
... self.layer = ...
... def call(self, *args):
... return ...
>>> # Initialize and compile the model as you normally would
>>> model = MyModel()
>>> model.compile(...)
>>> # Build the graph by training it or passing dummy inputs
>>> _ = model(model.dummy_inputs)
>>> # Save model weights to local directory
>>> model.save_pretrained("my-awesome-model")
>>> # Push model weights to the Hub
>>> model.push_to_hub("my-awesome-model")
>>> # Download and initialize weights from the Hub
>>> model = MyModel.from_pretrained("username/super-cool-model")
```
c & t | | d S )N)r~ )selfrD s r' _save_pretrainedz#KerasModelHubMixin._save_pretrained s dN33333r( c 8 t rddl}
nt d | dd } t j | s!t |||dt } n|} |
j j
j | fi | }
| |
_ |
S )a Here we just call [`from_pretrained_keras`] function so both the mixin and
functional APIs stay in sync.
TODO - Some args above aren't used since we are calling
snapshot_download instead of hf_hub_download.
r Nz>Called a TensorFlow-specific function but could not import it.r_ r. )r r cache_dirrI library_version)
r rj rk ru rR rS isdirr r r. r{
load_modelr_ )clsmodel_idr r force_downloadproxiesresume_downloadlocal_files_onlyr model_kwargsr- cfgstorage_folderr2 s r' _from_pretrainedz#KerasModelHubMixin._from_pretrained s & ` # # # # # ^__ _ x.. w}}X&& & . !# $ . 0 0 NN &N**>JJ\JJ r( N)__name__
__module__ __qualname____doc__r classmethodr r( r' r r sI ! !F4 4 4 + + [+ + +r( )r )TN)NFTN)r r )2collections.abcabcr rq rR rv pathlibr shutilr typingr r r r r huggingface_hubr
r huggingface_hub.utilsr r
r r r constantsr hf_apir rB r r r
get_loggerr loggerrj r- r r5 rE boolr! r^ rt rs r~ r r r r r( r' r s % % % % % % 3 3 3 3 3 3 3 3 3 3 3 3 3 3 < < < < < < < < # " " " " " H H H H H H H H H H
H % %? : " # * ** * tn * * * *` (, #'+
Rp Rp#t)$Rp
T#s(^ $Rp Rp
Rp 5s#
$
Rp Rp Rp Rpj8? 8? 8? 8?v
"C"& $6:7;7; ! #'+#v
v
v
v
TN v
v
v
3-v
C=v
SMv
~v
U49c> 23v
eDIsN 34v
eDIsN 34v
c] v
v
5s#
$!v
" #v
v
v
v
rS S S S S S S S S Sr(