Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory and speed optimizations #653

Merged
merged 6 commits into from Aug 30, 2022
Merged

Conversation

xiam
Copy link
Member

@xiam xiam commented Mar 14, 2022

This PR contains different optimizations around memory usage.

This is the result of profiling the PostgreSQL test suite on master right now:

(pprof) top10
Showing nodes accounting for 2017.31MB, 68.05% of 2964.67MB total
Dropped 391 nodes (cum <= 14.82MB)
Showing top 10 nodes out of 151
      flat  flat%   sum%        cum   cum%
  516.63MB 17.43% 17.43%   547.13MB 18.46%  github.com/upper/db/v4/internal/sqlbuilder.expandQuery
  426.01MB 14.37% 31.80%  1085.03MB 36.60%  github.com/upper/db/v4/internal/cache/hashstructure.(*walker).visit
  206.50MB  6.97% 38.76%   206.50MB  6.97%  encoding/binary.Write
     195MB  6.58% 45.34%   392.51MB 13.24%  github.com/upper/db/v4/internal/cache/hashstructure.hashUpdateOrdered
  151.50MB  5.11% 50.45%   151.50MB  5.11%  reflect.(*structType).Field
  139.01MB  4.69% 55.14%   139.01MB  4.69%  reflect.packEface
  123.94MB  4.18% 59.32%   130.44MB  4.40%  github.com/upper/db/v4/internal/sqladapter.ReplaceWithDollarSign
   91.21MB  3.08% 62.39%    91.21MB  3.08%  github.com/jackc/chunkreader/v2.(*ChunkReader).newBuf (inline)
      85MB  2.87% 65.26%       85MB  2.87%  github.com/upper/db/v4/internal/sqladapter/exql.ColumnWithName
   82.51MB  2.78% 68.05%    94.03MB  3.17%  github.com/jackc/pgx/v4/stdlib.(*Rows).Next

Versus the result of profiling the same suite after applying optimizations:

(pprof) top10
Showing nodes accounting for 634.12MB, 45.92% of 1380.82MB total
Dropped 313 nodes (cum <= 6.90MB)
Showing top 10 nodes out of 176
      flat  flat%   sum%        cum   cum%
  120.44MB  8.72%  8.72%   120.44MB  8.72%  github.com/jackc/chunkreader/v2.(*ChunkReader).newBuf (inline)
   79.51MB  5.76% 14.48%    92.54MB  6.70%  github.com/jackc/pgx/v4/stdlib.(*Rows).Next
   67.29MB  4.87% 19.35%    99.79MB  7.23%  github.com/upper/db/v4/internal/sqlbuilder.expandQuery
   63.01MB  4.56% 23.92%    74.51MB  5.40%  encoding/json.Unmarshal
   62.08MB  4.50% 28.41%  1311.78MB 95.00%  github.com/upper/db/v4/adapter/postgresql.testPostgreSQLTypes
   58.11MB  4.21% 32.62%   185.11MB 13.41%  github.com/jackc/pgconn.(*PgConn).Prepare
   56.03MB  4.06% 36.68%    89.03MB  6.45%  github.com/upper/db/v4/internal/sqlbuilder.toColumnsValuesAndArguments
   44.63MB  3.23% 39.91%   255.11MB 18.48%  github.com/upper/db/v4/adapter/postgresql.(*database).CompileStatement
   42.53MB  3.08% 42.99%    48.03MB  3.48%  github.com/upper/db/v4/internal/sqlbuilder.Map
   40.51MB  2.93% 45.92%    40.51MB  2.93%  github.com/jackc/pgx/v4/stdlib.(*Rows).Columns```

@xiam xiam marked this pull request as draft March 14, 2022 06:14
@xiam xiam force-pushed the refactor-memory-and-speed-optimizations branch from 3dc4106 to 3f1dff4 Compare April 3, 2022 17:32
@xiam xiam marked this pull request as ready for review August 15, 2022 03:05
@xiam xiam force-pushed the refactor-memory-and-speed-optimizations branch from 7f8c184 to ba90d64 Compare August 15, 2022 03:36
@xiam xiam changed the title Memory and speed optimizations around cache Memory and speed optimizations Aug 15, 2022
@xiam xiam merged commit 8a3fe0c into master Aug 30, 2022
@xiam xiam deleted the refactor-memory-and-speed-optimizations branch August 30, 2022 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant