Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

time series forecasting model #17965

Merged
merged 172 commits into from Sep 30, 2022
Merged

time series forecasting model #17965

merged 172 commits into from Sep 30, 2022

Conversation

kashif
Copy link
Contributor

@kashif kashif commented Jun 30, 2022 •
edited by NielsRogge

What does this PR do?

This PR implements a vanilla encoder-decoder Transformer for time-series forecasting.

@kashif kashif marked this pull request as draft June 30, 2022 15:38
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Jun 30, 2022 •
edited

The documentation is not available anymore as the PR was closed or merged.

kashif added 26 commits June 30, 2022 18:50
@kashif
initial model via cli
cc0b364
@kashif
typos
cf60b95
@kashif
make a start on the model config
b18d345
@kashif
ready with configuation
eb328f5
@kashif
remove tokenizer ref.
8e664f4
@kashif
init the transformer
e078e09
@kashif
added initial model forward to return dec_output
972cc89
@kashif
require gluonts
cf74608
@kashif
update dep. ver table and add as extra
c7b8158
@kashif
fixed typo
dba022c
@kashif
Merge remote-tracking branch 'upstream/main' into time-series
44c9944
@kashif
add type for prediction_length
29bcc72
@kashif
use num_time_features
6f3f627
@kashif
use config
97fbcf4
@kashif
more config
62286aa
@kashif
typos
88e071b
@kashif
opps another typo
3e01a69
@kashif
freq can be none
202dab0
@kashif
default via transformation is 1
489d7f1
@kashif
initial transformations
d5f6eb3
@kashif
fix imports
302a387
@kashif
added transform_start_field
669aeeb
@kashif
add helper to create pytorch dataloader
6c339d8
@kashif
added inital val and test data loader
43ffcf1
@kashif
added initial distr head and loss
1525a1a
@kashif
training working
3fa6d48
@NielsRogge
Copy link
Contributor

cc'ing @mishig25 here - there seems to be an issue with the docs being built. The model is added to the toctree, but it's saying:

Traceback (most recent call last):
  File "/usr/local/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.8/site-packages/doc_builder/commands/build.py", line 96, in build_command
    build_doc(
  File "/usr/local/lib/python3.8/site-packages/doc_builder/build_doc.py", line 427, in build_doc
    sphinx_refs = check_toc_integrity(doc_folder, output_dir)
  File "/usr/local/lib/python3.8/site-packages/doc_builder/build_doc.py", line 482, in check_toc_integrity
    raise RuntimeError(
RuntimeError: The following files are not present in the table of contents:
- model_doc/time_series_transformer
Add them to ../transformers/docs/source/en/_toctree.yml.

@kashif @sgugger
Update docs/source/en/_toctree.yml
cb66357 
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
sgugger
sgugger reviewed Sep 30, 2022
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still a couple of comments on docstrings to address, and one last nit. Then we can merge :-)

src/transformers/models/time_series_transformer/configuration_time_series_transformer.py Outdated Show resolved Hide resolved
src/transformers/models/time_series_transformer/modeling_time_series_transformer.py Outdated Show resolved Hide resolved
src/transformers/models/time_series_transformer/modeling_time_series_transformer.py Outdated

encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*):
Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`)
`last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) is a sequence of
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here too

src/transformers/models/time_series_transformer/modeling_time_series_transformer.py Outdated Show resolved Hide resolved
@kashif @sgugger
Update src/transformers/models/time_series_transformer/configuration_…
4fe63fe 
…time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
@sgugger
Copy link
Collaborator

sgugger commented Sep 30, 2022

Mishig the failure on the doc was due to a typo (comment is hidden now since the suggestion was accepted) nothing to do for you :-)

kashif and others added 6 commits September 30, 2022 20:13
@kashif @sgugger
Update src/transformers/models/time_series_transformer/modeling_time_…
b28a2f9 
…series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
@kashif @sgugger
Update src/transformers/models/time_series_transformer/modeling_time_…
8a3e430 
…series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
@kashif
Update modeling_time_series_transformer.py
f7b0ab5 
fix style
@kashif
fixed typo
4fe8164
@kashif
fix typo and grammer
48a2be4
@kashif
fix style
43dd269
@sgugger sgugger merged commit 5cd16f0 into huggingface:main Sep 30, 2022
@kashif kashif deleted the time-series branch October 2, 2022 08:01
@kashif kashif restored the time-series branch October 2, 2022 14:01
amyeroberts pushed a commit to amyeroberts/transformers that referenced this pull request Oct 18, 2022
@kashif @NielsRogge
@sgugger @amyeroberts
time series forecasting model (huggingface#17965)
1432d4c 
* initial files

* initial model via cli

* typos

* make a start on the model config

* ready with configuation

* remove tokenizer ref.

* init the transformer

* added initial model forward to return dec_output

* require gluonts

* update dep. ver table and add as extra

* fixed typo

* add type for prediction_length

* use num_time_features

* use config

* more config

* typos

* opps another typo

* freq can be none

* default via transformation is 1

* initial transformations

* fix imports

* added transform_start_field

* add helper to create pytorch dataloader

* added inital val and test data loader

* added initial distr head and loss

* training working

* remove TimeSeriesTransformerTokenizer

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update src/transformers/__init__.py

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/__init__.py

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* fixed copyright

* removed docs

* remove time series tokenizer

* fixed docs

* fix text

* fix second

* fix default

* fix order

* use config directly

* undo change

* fix comment

* fix year

* fix import

* add additional arguments for training vs. test

* initial greedy inference loop

* fix inference

* comment out token inputs to enc dec

* Use HF encoder/decoder

* fix inference

* Use Seq2SeqTSModelOutput output

* return Seq2SeqTSPredictionOutput

* added default arguments

* fix return_dict true

* scale is a tensor

* output static_features for inference

* clean up some unused bits

* fixed typo

* set return_dict if none

* call model once for both train/predict

* use cache if future_target is none

* initial generate func

* generate arguments

* future_time_feat is required

* return SampleTSPredictionOutput

* removed unneeded classes

* fix when params is none

* fix return dict

* fix num_attention_heads

* fix arguments

* remove unused shift_tokens_right

* add different dropout configs

* implement FeatureEmbedder, Scaler and weighted_average

* remove gluonts dependency

* fix class names

* avoid _variable names

* remove gluonts dependency

* fix imports

* remove gluonts from configuration

* fix docs

* fixed typo

* move utils to examples

* add example requirements

* config has no freq

* initial run_ts_no_trainer

* remove from ignore

* fix output_attentions and removed unsued getters/setters

* removed unsed tests

* add dec seq len

* add test_attention_outputs

* set has_text_modality=False

* add config attribute_map

* make style

* make fix-copies

* add encoder_outputs to TimeSeriesTransformerForPrediction forward

* Improve docs, add model to README

* added test_forward_signature

* More improvements

* Add more copied from

* Fix README

* Fix remaining quality issues

* updated encoder and decoder

* fix generate

* output_hidden_states and use_cache are optional

* past key_values returned too

* initialize weights of distribution_output module

* fixed more tests

* update test_forward_signature

* fix return_dict outputs

* Update src/transformers/models/time_series_transformer/configuration_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/configuration_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/configuration_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/configuration_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/modeling_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/modeling_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/modeling_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* removed commented out tests

* added neg. bin and normal output

* Update src/transformers/models/time_series_transformer/configuration_time_series_transformer.py

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* move to one line

* Add docstrings

* Update src/transformers/models/time_series_transformer/configuration_time_series_transformer.py

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* add try except for assert and raise

* try and raise exception

* fix the documentation formatting

* fix assert call

* fix docstring formatting

* removed input_ids from DOCSTRING

* Update input docstring

* Improve variable names

* Update order of inputs

* Improve configuration

* Improve variable names

* Improve docs

* Remove key_length from tests

* Add extra docs

* initial unittests

* added test_inference_no_head test

* added test_inference_head

* add test_seq_to_seq_generation

* make style

* one line

* assert mean prediction

* removed comments

* Update src/transformers/models/time_series_transformer/modeling_time_series_transformer.py

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/modeling_time_series_transformer.py

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* fix order of args

* make past_observed_mask optional as well

* added Amazon license header

* updated utils with new fieldnames

* make style

* cleanup

* undo position of past_observed_mask

* fix import

* typo

* more typo

* rename example files

* remove example for now

* Update docs/source/en/_toctree.yml

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/configuration_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/modeling_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/time_series_transformer/modeling_time_series_transformer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update modeling_time_series_transformer.py

fix style

* fixed typo

* fix typo and grammer

* fix style

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: NielsRogge <niels.rogge1@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NielsRogge NielsRogge NielsRogge left review comments

@sgugger sgugger sgugger approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@kashif @HuggingFaceDocBuilderDev @NielsRogge @sgugger