Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source code missing base dir on the remote storage when deploy at other path(subdir) #13140

Open
3 tasks done
Ox0400 opened this issue Apr 28, 2024 · 1 comment · May be fixed by #13141
Open
3 tasks done

source code missing base dir on the remote storage when deploy at other path(subdir) #13140

Ox0400 opened this issue Apr 28, 2024 · 1 comment · May be fixed by #13141
Labels
enhancement An improvement of an existing feature needs:triage Needs feedback from the Prefect product team

Comments

@Ox0400
Copy link

Ox0400 commented Apr 28, 2024

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar request and didn't find it.
  • I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

In current, default upload work dir ($PWD) files to remote storage block base path when deploy a pipeline and upload the source code.

But we edit some file on sub dir path or other path, So the flow source code not uploaded to remote or only upload sub dir files to remote base path.

$ pwd
/var/work/
$ ls
flows/flow_a.py flows/flow_b.py common/pgsql.py

The remote storage have two dir flows and common, and three files is flows/flow_a.py flows/flow_b.py common/pgsql.py.

Cleanup the remote storage files,
cd flows first, then deploy the flow.

$ pwd
/var/work/flows
$ ls
flow_a.py flow_b.py

The remote storage not have any dir, only have two files is flow_a.py flow_b.py. But i hope have two subdir flows and common and them files.

Describe the proposed behavior

Hope we can support specify a local location when deploy the pipeline.
The mean that's deploy and upload the flows or pipelines on anywhere. Not need cd always.

Example Use

e.g:

# create and upload flow
Deployment.build_from_flow(
        flow=projects_dedup_flow,
        name=name,
        apply=False,
        parameters = parameters,
        infra_overrides=infra_overrides,
        work_queue_name=ENV.WORK_QUEUE_NAME,
        work_pool_name=ENV.WORK_POOL_NAME,
        storage=RemoteFileSystem.load('codes'),
        # path='',
        storage_local_path='/data/codes/projects/dedup',
    )

# deployments/deployments.py
    @classmethod
    @sync_compatible
    async def build_from_flow(
        cls,
        flow: Flow,
        name: str,
        output: str = None,
        skip_upload: bool = False,
        ignore_file: str = ".prefectignore",
        apply: bool = False,
        load_existing: bool = True,
        storage_local_path: str = None,
        **kwargs,
    ) -> "Deployment":
        ...
        deployment.upload_to_storage(ignore_file=ignore_file, local_path=storage_local_path)
        ...

    @sync_compatible
    async def upload_to_storage(
        self, storage_block: str = None, ignore_file: str = ".prefectignore", local_path: str = None,
    ) -> Optional[int]:
        ...
        file_count = await self.storage.put_directory(
                ignore_file=ignore_file, to_path=self.path, local_path=local_path,
            )
        ...

Additional context

No response

@Ox0400 Ox0400 added enhancement An improvement of an existing feature needs:triage Needs feedback from the Prefect product team labels Apr 28, 2024
@Ox0400 Ox0400 linked a pull request Apr 28, 2024 that will close this issue
6 tasks
@Ox0400 Ox0400 changed the title deploy a pipeline from other path, the code upload to remote storage be changed. deploy a pipeline from other path, the source code missing base dir at remote storage Apr 28, 2024
@Ox0400 Ox0400 changed the title deploy a pipeline from other path, the source code missing base dir at remote storage deploy a pipeline from other path(subdir), the source code missing base dir on the remote storage Apr 28, 2024
@Ox0400 Ox0400 changed the title deploy a pipeline from other path(subdir), the source code missing base dir on the remote storage source code missing base dir on the remote storage when deploy at other path(subdir) Apr 28, 2024
@Ox0400
Copy link
Author

Ox0400 commented Apr 28, 2024

I have put PR at: #13141

Ox0400 added a commit to Ox0400/prefect that referenced this issue Apr 29, 2024
Ox0400 added a commit to Ox0400/prefect that referenced this issue May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature needs:triage Needs feedback from the Prefect product team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant