Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

d2g: docker-worker artifacts without an expires property are translated to generic-worker artifact with expires 0001-01-01T00:00:00.000Z #6858

Open
petemoore opened this issue Feb 26, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@petemoore
Copy link
Member

petemoore commented Feb 26, 2024

Describe the bug

There is a long standing issue with jsonschema2go/timestamps whereby an optional timestamp that is not provided is reported as 0001-01-01T00:00:00.000Z. This is the zero value of the go type that backs the field. jsonschema2go generates structs to represent json objects, rather than struct pointers, and in turn we have the issue that we sometimes cannot distinguish between an empty value, and the zero value of a type.

Adapting jsonschema2go to use pointer types is probably undesirable, as this would be a major change and probably break a ton of things that depend on its current behavior. One thing we did before was explicitly replace empty values, see e.g. here.

This is pretty dirty, but it might be the simplest/easiest way to achieve the same in d2g, while retaining a very small impact footprint.

Example output:

[taskcluster 2024-02-26T16:55:37.489Z] === Task Starting ===
[taskcluster:warn 2024-02-26T16:55:37.493Z] This task was designed to run under Docker Worker. Docker Worker is no longer maintained.
[taskcluster:warn 2024-02-26T16:55:37.493Z] In order to execute this task, it is being converted to a Generic Worker task, using the D2G
[taskcluster:warn 2024-02-26T16:55:37.493Z] utility (Docker Worker 2 Generic Worker):
[taskcluster:warn 2024-02-26T16:55:37.493Z]     https://github.com/taskcluster/taskcluster/tree/main/clients/client-shell#translating-docker-worker-task-definitionpayload-to-generic-worker-task-definitionpayload
[taskcluster:warn 2024-02-26T16:55:37.493Z] 
[taskcluster:warn 2024-02-26T16:55:37.493Z] We recommend that you convert all your Docker Worker tasks to Generic Worker tasks, to ensure
[taskcluster:warn 2024-02-26T16:55:37.493Z] continued support. For this task, see the converted payload below. If you have many tasks that
[taskcluster:warn 2024-02-26T16:55:37.493Z] require conversion, consider using the d2g tool (above) directly. It simply takes a Docker
[taskcluster:warn 2024-02-26T16:55:37.493Z] Worker task payload as input, and outputs a Generic Worker task payload. It can also convert
[taskcluster:warn 2024-02-26T16:55:37.493Z] Docker Worker scopes to equivalent Generic Worker scopes.
[taskcluster:warn 2024-02-26T16:55:37.493Z] 
[taskcluster:warn 2024-02-26T16:55:37.493Z] Converted task definition (conversion performed by d2g):
[taskcluster:warn 2024-02-26T16:55:37.493Z] ---
[taskcluster:warn 2024-02-26T16:55:37.493Z]   created: "2024-02-26T16:38:17.896Z"
[taskcluster:warn 2024-02-26T16:55:37.493Z]   deadline: "2024-02-26T20:38:17.896Z"
[taskcluster:warn 2024-02-26T16:55:37.493Z]   dependencies:
[taskcluster:warn 2024-02-26T16:55:37.493Z]   - WLjR35xnQNKoICirpYPZqw
[taskcluster:warn 2024-02-26T16:55:37.493Z]   expires: "2024-03-04T16:38:17.896Z"
[taskcluster:warn 2024-02-26T16:55:37.493Z]   extra: {}
[taskcluster:warn 2024-02-26T16:55:37.493Z]   metadata:
[taskcluster:warn 2024-02-26T16:55:37.493Z]     description: Bugmon worker
[taskcluster:warn 2024-02-26T16:55:37.493Z]     name: ProcessorTask (1881076)
[taskcluster:warn 2024-02-26T16:55:37.493Z]     owner: jkratzer@mozilla.com
[taskcluster:warn 2024-02-26T16:55:37.493Z]     source: https://github.com/MozillaSecurity/bugmon
[taskcluster:warn 2024-02-26T16:55:37.493Z]   payload:
[taskcluster:warn 2024-02-26T16:55:37.493Z]     artifacts:
[taskcluster:warn 2024-02-26T16:55:37.493Z]     - expires: "0001-01-01T00:00:00.000Z"
[taskcluster:warn 2024-02-26T16:55:37.493Z]       name: project/fuzzing/bugmon
[taskcluster:warn 2024-02-26T16:55:37.493Z]       path: artifact0
[taskcluster:warn 2024-02-26T16:55:37.493Z]       type: directory
[taskcluster:warn 2024-02-26T16:55:37.493Z]     command:
[taskcluster:warn 2024-02-26T16:55:37.493Z]     - - bash
[taskcluster:warn 2024-02-26T16:55:37.493Z]       - -cx
[taskcluster:warn 2024-02-26T16:55:37.493Z]       - "timeout 14400 podman run -t --name taskcontainer --privileged --security-opt=seccomp=unconfined
[taskcluster:warn 2024-02-26T16:55:37.493Z]         --device=/dev/shm --device=/dev/snd --add-host=taskcluster:127.0.0.1 --net=host
[taskcluster:warn 2024-02-26T16:55:37.493Z]         -e BUG_ACTION -e MONITOR_ARTIFACT -e PROCESSOR_ARTIFACT -e RUN_ID -e TASKCLUSTER_PROXY_URL
[taskcluster:warn 2024-02-26T16:55:37.493Z]         -e TASKCLUSTER_ROOT_URL -e TASKCLUSTER_WORKER_LOCATION -e TASK_ID -e TRACE_ARTIFACT
[taskcluster:warn 2024-02-26T16:55:37.493Z]         docker-archive:dockerimage \nexit_code=$?\npodman cp taskcontainer:/bugmon-artifacts/
[taskcluster:warn 2024-02-26T16:55:37.493Z]         artifact0\npodman rm taskcontainer\nexit \"${exit_code}\""
[taskcluster:warn 2024-02-26T16:55:37.493Z]     env:
[taskcluster:warn 2024-02-26T16:55:37.493Z]       BUG_ACTION: process
[taskcluster:warn 2024-02-26T16:55:37.493Z]       MONITOR_ARTIFACT: monitor-1881076-WLjR35xnQNKoICirpYPZqw.json
[taskcluster:warn 2024-02-26T16:55:37.493Z]       PROCESSOR_ARTIFACT: processor-result-1881076-WLjR35xnQNKoICirpYPZqw.json
[taskcluster:warn 2024-02-26T16:55:37.493Z]       TRACE_ARTIFACT: processor-rr-trace-1881076-WLjR35xnQNKoICirpYPZqw.tar.gz
[taskcluster:warn 2024-02-26T16:55:37.493Z]     features:
[taskcluster:warn 2024-02-26T16:55:37.493Z]       backingLog: true
[taskcluster:warn 2024-02-26T16:55:37.493Z]       liveLog: true
[taskcluster:warn 2024-02-26T16:55:37.493Z]       loopbackAudio: true
[taskcluster:warn 2024-02-26T16:55:37.493Z]       taskclusterProxy: true
[taskcluster:warn 2024-02-26T16:55:37.493Z]     logs:
[taskcluster:warn 2024-02-26T16:55:37.493Z]       backing: public/logs/live_backing.log
[taskcluster:warn 2024-02-26T16:55:37.493Z]       live: public/logs/live.log
[taskcluster:warn 2024-02-26T16:55:37.493Z]     maxRunTime: 15300
[taskcluster:warn 2024-02-26T16:55:37.493Z]     mounts:
[taskcluster:warn 2024-02-26T16:55:37.493Z]     - content:
[taskcluster:warn 2024-02-26T16:55:37.493Z]         artifact: public/bugmon.tar.zst
[taskcluster:warn 2024-02-26T16:55:37.493Z]         namespace: project.fuzzing.orion.bugmon.master
[taskcluster:warn 2024-02-26T16:55:37.493Z]       file: dockerimage
[taskcluster:warn 2024-02-26T16:55:37.493Z]       format: zst
[taskcluster:warn 2024-02-26T16:55:37.493Z]     onExitStatus:
[taskcluster:warn 2024-02-26T16:55:37.493Z]       retry:
[taskcluster:warn 2024-02-26T16:55:37.493Z]       - 125
[taskcluster:warn 2024-02-26T16:55:37.493Z]       - 128
[taskcluster:warn 2024-02-26T16:55:37.493Z]   priority: high
[taskcluster:warn 2024-02-26T16:55:37.493Z]   projectId: none
[taskcluster:warn 2024-02-26T16:55:37.493Z]   provisionerId: proj-fuzzing
[taskcluster:warn 2024-02-26T16:55:37.493Z]   requires: all-completed
[taskcluster:warn 2024-02-26T16:55:37.493Z]   retries: 5
[taskcluster:warn 2024-02-26T16:55:37.493Z]   routes:
[taskcluster:warn 2024-02-26T16:55:37.493Z]   - notify.email.jkratzer@mozilla.com.on-failed
[taskcluster:warn 2024-02-26T16:55:37.493Z]   schedulerId: '-'
[taskcluster:warn 2024-02-26T16:55:37.493Z]   scopes:
[taskcluster:warn 2024-02-26T16:55:37.493Z]   - generic-worker:capability:device:hostSharedMemory
[taskcluster:warn 2024-02-26T16:55:37.493Z]   - generic-worker:loopback-audio:*
[taskcluster:warn 2024-02-26T16:55:37.493Z]   - generic-worker:capability:disableSeccomp
[taskcluster:warn 2024-02-26T16:55:37.493Z]   - generic-worker:capability:privileged
[taskcluster:warn 2024-02-26T16:55:37.493Z]   - queue:get-artifact:project/fuzzing/bugmon/monitor-1881076-WLjR35xnQNKoICirpYPZqw.json
[taskcluster:warn 2024-02-26T16:55:37.493Z]   - queue:scheduler-id:-
[taskcluster:warn 2024-02-26T16:55:37.493Z]   tags: {}
[taskcluster:warn 2024-02-26T16:55:37.493Z]   taskGroupId: WLjR35xnQNKoICirpYPZqw
[taskcluster:warn 2024-02-26T16:55:37.493Z]   taskQueueId: proj-fuzzing/bugmon-pernosco
[taskcluster:warn 2024-02-26T16:55:37.493Z]   workerType: bugmon-pernosco

The undesirable line above is this one:

[taskcluster:warn 2024-02-26T16:55:37.493Z]     - expires: "0001-01-01T00:00:00.000Z"
@petemoore petemoore added the bug Something isn't working label Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Todo
Development

No branches or pull requests

1 participant