Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add performance data source #10

Open
EnricoMi opened this issue Jun 15, 2020 · 0 comments · May be fixed by #28
Open

Add performance data source #10

EnricoMi opened this issue Jun 15, 2020 · 0 comments · May be fixed by #28
Labels
enhancement New feature or request

Comments

@EnricoMi
Copy link
Collaborator

EnricoMi commented Jun 15, 2020

Add a data source that does not read the actual data but provides performance metrics. Each partition sends a query to the Dgraph cluster and retrieved besides the data also these metrics:

  "extensions": {
    "server_latency": {
      "parsing_ns": 78501,
      "processing_ns": 881611,
      "encoding_ns": 110785,
      "total_ns": 1145597
    },
    "txn": {
      "start_ts": 10007
    },
    "metrics": {
      "num_uids": {
        "dgraph.graphql.schema": 10,
        "dgraph.type": 10,
        "director": 10,
        "name": 10,
        "release_date": 10,
        "revenue": 10,
        "running_time": 10,
        "starring": 10,
        "uid": 16
      }
    }
  }

The performance data source can encode these information (together with information from TaskContext and the individual partitions) rather than the actual data result into the DataFrame. This provides benchmarking tools to measure per-partition timings and cardinality information and write them via Spark to disk.

@EnricoMi EnricoMi added the enhancement New feature or request label Jun 15, 2020
@EnricoMi EnricoMi linked a pull request Jul 2, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant