[DOC-325] [2.21.1] Added ycql-stat-statements and yb_local_tablets #22256
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary:
The design of our newer modals all use confirmation textboxes instead of confirmation checkboxes.
This diff aligns the HA promotion modal to do the same.
Test Plan:
Verify form submit is disabled when `PROMOTE` is not entered.
Verify form submit is not disabled when `PROMOTE` is entered.
{F173446}
Reviewers: rmadhavan
Reviewed By: rmadhavan
Subscribers: yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D34502
Summary: `create_bitmap_subplan` takes several lists as input, and each list should be set to a real value or NIL. D32651 / 773869c added another list to this function for pushdown quals, but did not set the value in all cases. Callers of the function would pass in an unitialized value for `indexpushdownquals`, and expect it to be a valid list. Instead, it might remain unitialized and potentially cause a crash. Fix the issue by handling the one remaining branch where indexpushdownquals is not set. Jira: DB-11003 Test Plan: ``` ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressGin' Reviewers: amartsinchyk, tnayak Reviewed By: tnayak Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D34367
…nsaction + Changes subtxn id semantics at pg/pg_client_service
Summary:
As we allow buffering of operations in YSQL, multiple batches belonging to the same subtxn id are launched in parallel in an async manner. There is only one outstanding batch executing `PgClientSession::SetupSession`, while there can be many outstanding batches executing `YBTransaction::Impl::Prepare` as part of callback from `LookupByKeyRpc`.
In the existing implementation, it leads to data race issues with `YBSubTransaction`. A previously launched batch is trying to access `highest_subtransaction_id_` during `Prepare` to populate in-flight ops metadata, while a subsequent batch is trying to set the same field `highest_subtransaction_id_`. Though the writer thread tries to overwrite `highest_subtransaction_id_` with the same old value, it leads to a data race. Also, for RC transactions, we seem to hit a similar race while executing `RollbackToSubTransaction`. Seems like the pg backend issues a rollback on error before all inflight operations complete.
```
Status PgApiImpl::RollbackToSubTransaction(SubTransactionId id) {
pg_session_->DropBufferedOperations();
return pg_session_->RollbackToSubTransaction(id);
}
```
**Change 1**
To address the data race, we now set subtxn metadata for the batch (batch of ops) by setting it during `Batcher::FlushAsync`. Batcher then launches `YBTransaction::Impl::Prepare` for the underlying transaction, which sets only the transaction metadata. If the batch fails, the subtxn metadata is copied to the retry batcher.
**Change 2**
This diff also addresses an anomaly with `active_sub_transaction_id_` passed from `pg_session`. Postgres assigns subtransaction id(s) starting from 1. But in the existing implementation, we see that `active_sub_transaction_id_` starts from 0 and then bumps up to 2 on savepoint creation (value as seen in the requests at `pg_client_session.cc`). In `client/transaction.cc`, we leave the subtxn metadata unpopulated if no savepoint has been created yet. The downstream code assumes that the subtxn belonged to id 1 since the subtxn metadata was unpopulated. To avoid this confusion, we change the default value of `active_sub_transaction_id_` and populate the subtxn metadata pb only when subtxn is not in its default state.
Enabling test `PgTableSizeTest.PartitionedTableSize` to run in tsan mode, as the most of the race issues with pggate are resolved now.
Note: The earlier version of the [[ https://phorge.dev.yugabyte.com/D23412 | fix ]] lead to a data loss [[ yugabyte#16988 | issue ]] as the subtxn metadata was not being populated for retry batchers (which are quite probable in case of tablet splits). Hence, these ops were wrongly being treated as belonging to `kMinSubtransactionId` by downstream code, which might have lead to the issue. Added a test that validates the reasoning.
**Additional note**
We discussed offline that this change shouldn't have upgrade/downgrade consequences. Since pg process is a child process of the tserver process, changes at pg_client_session/service and pg_txn_manager would have taken effect in combination. So the subtxn metadata in the actual rpc to docdb would remain unchanged.
Jira: DB-5343, DB-10441
Test Plan:
Jenkins
```
./yb_build.sh tsan --gtest_filter PgTableSizeTest.PartitionedTableSize -n 100
./yb_build.sh --gtest_filter PgTabletSplitTest.SplitAmidstRunningTransaction
./yb_builds tsan --cxx-test='TEST_F(GeoPartitionedReadCommiittedTest, TestPromotionAmidstConflicts) {'
```
Reviewers: pjain, esheng, rsami, rthallam
Reviewed By: pjain
Subscribers: yql, ybase, bogdan
Differential Revision: https://phorge.dev.yugabyte.com/D25071
…e#22225) * Add docs for configure_read_replica command. * Updated docs for backup, restore, configure_read_replica, finalize_upgrade commands. * Fixing links. * Apply suggestions from code review --------- Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com>
Summary: Do not display TEST flags in the human readable `/varz` endpoint unless the user overrides them. TEST flags have tag `kHidden` and their name starts with `TEST_`. This is ensured by the `DEFINE_test_flag` macro. Fixes yugabyte#14755 Jira: DB-4087 Test Plan: MasterPathHandlersItest.TestTestFlag Reviewers: esheng Reviewed By: esheng Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D34701
* get started guide * misc changes * misc edits * Edits * format * Add suggested links * minor edit * reorganization * Triggering rebuild * Triggering preview rebuild modified the title * review comments * comments --------- Co-authored-by: Premkumar <contactprem+premkumr@gmail.com> Co-authored-by: Premkumar <pthangamani@yugabyte.com>
…pped successfully Summary: To reproduce the bug: ``` yugabyte=# create table foo(id int); CREATE TABLE yugabyte=# alter table foo drop column id; ALTER TABLE ``` Note that the ALTER TABLE statement completed successfully. Look at yb-master.INFO, saw: ``` I0502 19:26:37.713428 27752 ysql_ddl_handler.cc:396] Sending Alter Table request as part of rollback for table foo ``` The log says `rollback` which suggests the DDL operation failed. But in this case `roll forward` is more appropriate. We first mark the column id to be dropped in DocDB metadata for table `foo`, then drop the column from PG catalog metadata for table `foo`. Once the PG side has successfully committed, we then roll forward the DocDB metadata change by dropping the column. This completes the entire DDL operation. I made a fix to log `roll forward` if the PG side commits successfully, and `rollback` if the PG side aborts. Also made some small test code cleanup for DDL atomicity because --ysql_yb_ddl_rollback_enabled is true by default now. Jira: DB-11161 Test Plan: (1) Manually do the test described in the summary, saw the new log: ``` I0502 20:12:33.628252 4412 ysql_ddl_handler.cc:399] Sending Alter Table request as part of roll forward for table foo ``` (2) ./yb_build.sh release --java-test 'org.yb.pgsql.TestPgUniqueConstraint' (3) YB_ENABLE_YSQL_CONN_MGR_IN_TESTS=true ./yb_build.sh release --java-test 'org.yb.pgsql.TestPgUniqueConstraint' Reviewers: fizaa Reviewed By: fizaa Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D34704
…gn 2) (yugabyte#21308) * Enable Drift & move kapa.ai to header right after search * Change icon and text * Fix button issue while page is loading * Enale/Disable Drift option and move drift in head * Update search banner message * Update search banner message
Summary: The diff adds the YSQL syntax to support DB cloning: creating a database as a lightweight copy of another database. User can now create a database `db2` as a clone of database `db1` as of a point in time `t1`, by writing the following sql query: `CREATE DATABASE db2 TEMPLATE db1 AS OF t1;` where t1 is a linux timestamp in microseconds. Using the template syntax to support DB cloning is in line with postgres [[ https://www.postgresql.org/docs/current/manage-ag-templatedbs.html#MANAGE-AG-TEMPLATEDBS | intention ]] to support copying databases. Before this diff, it was not supported to use the TEMPLATE keyword with any database other than `template0` and `template1`. Now we removed this limitation. Yugabyte can also perform the cloning without the strict condition inherited from postgres: `no other sessions can be connected to the source database while it is being copied`. So removed this limitation in this diff and we can copy the database even if there are other connections to the source database. **Upgrade/Rollback safety:** The clone feature is guarded by the preview flag `enable_db_clone `. Jira: DB-8620 Test Plan: ybd --cxx-test integration-tests_minicluster-snapshot-test --gtest_filter PgCloneTest.CloneYsqlSyntax Removed `CREATE DATABASE test TEMPLATE = some_template;` from `CREATE DATABASE` unsupported options in regress tests as now we support copying databases from templates other than `template0` or `template1`. The same statement cannot be added to the regress test as supported because we require having a snapshot schedule on the template database as a requirement for copying the database. However, the unit test `PgCloneTest.CloneYsqlSyntax` is covering this case. Reviewers: asrivastava, yguan Reviewed By: asrivastava, yguan Subscribers: pjain, yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D33408
…ection Manager Summary: This diff adds the support for login limit connections per user in ysql connection manager. The only known limitation of this implementation is that the client has to wait for few seconds (by default 10seconds) for connection manager stats to get updated which is used to calculate current number of connections made by a particular user. Track Here: [[ yugabyte#21645 | GH yugabyte#21645 ]] One of the existing java unit tests (`org.yb.pgsql.TestPgAuthorization#testAttributes`) which tests connection limit per user is not fixed as part of this diff, as it still fails while dropping the user which is not supported in ysql connection manager at the time of creating this diff. (Tracked by [[ yugabyte#21862 | GH yugabyte#21862 ]] ). Added a TODO comment to fix conn per limit user part of the test. Jira: DB-7747 Test Plan: Added java unit test: ```./yb_build.sh --enable-ysql-conn-mgr-test --java-test org.yb.ysqlconnmgr.TestUserLoginLimit``` Reviewers: rbarigidad Reviewed By: rbarigidad Subscribers: rbarigidad, nkumar, yql Differential Revision: https://phorge.dev.yugabyte.com/D33469
Summary: The ssh port is moved out off region metadata. This diff implements the fix to show the correct Port no
Test Plan:
Tested manually.
{F175066}
Reviewers: lsangappa
Reviewed By: lsangappa
Subscribers: yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D34691
* yba prepare section * misc edits * more edits * overview page * add more requirements * more requirements * more updates * misc updates * more updates * misc edits * minor fixes * Update with latest changes * Misc edits * prepare landing page * Update docs/content/preview/yugabyte-platform/prepare/_index.md * menu order * ear changes * links and redirects * review comments * redirects and links * links * header links * tidyups * minor edit * review comments * review comments * minor edit * review comments * format * review comments * review comments * typo * review comments * review, format * more fixes * image * comments * format, edit * minor edits * intro format * fix for Replication migration * review comments * minor edit * fix links and delet unused files * minor edits * minor edit
Summary:
[PLAT-13089] SystemdUpgrade is now retriable
Adjusted the state check in verify params to allow SystemdUpgrade.
to go through on a retry.
Test Plan:
{F175509}
{F175508}phabricator
localtest
Reviewers: #yba-api-review!, nsingh
Reviewed By: nsingh
Subscribers: nsingh, yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D34710
…ender Summary: We have debug logs in place in walsender to help identify issues in case of errors. Some of these logs are printed for every row being streamed. As a result, they can have a substantial overhead. This revision updates them with the following strategy: 1. Batch level: DEBUG1. These logs are printed once per batch of changes received from the VWAL. The current batch size is 500. So these are summary logs 2. Row level: DEBUG2. These logs are printed for every row being streamed. Also did misc. improvements in the logs. Jira: DB-11152 Test Plan: Jenkins: test regex: .*ReplicationSlot.* Existing tests Reviewers: asrinivasan Reviewed By: asrinivasan Subscribers: ybase, ycdcxcluster, yql Differential Revision: https://phorge.dev.yugabyte.com/D34682
…rics Summary: Currently the CDCSDK metrics fail the DCHECK on attributes in metrics-writer class method `AddAggregatedEntry()`. The DCHECK was introduced in the diff https://phorge.dev.yugabyte.com/D33396 and causes tserver crash, as it compares and expects all the attributes to be the same for a given metric entity type, i.e. it expects that all the attributes of entity type cdcsdk should have the same attributes. Currently the CDCSDK metrics store `table_name` and `table_id` in the attributes. However since the CDCSDK metrics are aggregated at stream level, they can have multiple tables associated with them. This causes the DCHECK to fail. Since we do not need `table_name` and `table_id` attributes for aggregation of CDCSDK metrics, the fix for this issue is removing the table level attributes from the attribute map for CDCSDK metrics. This DCHECK failure was not caught in any existing unit test, hence modifying an existing test. Also this was not caught in any QA stress tests as these are run in release mode. Jira: DB-11070 Test Plan: ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCDCSDKMetricsTwoTablesSingleStream PG connector test PostgresConnectorIT#shouldProduceEventsWithInitialSnapshot Reviewers: skumar, asrinivasan, vkushwaha, stiwary, yyan Reviewed By: yyan Subscribers: mlillibridge, ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34528
Summary: The ybgin and lsm access methods repeat a lot of logic during column binding. This diff deduplicates such logic. This diff also adds the `yb_is_supported` method to the IndexAMHandler interface to simplify areas of the code that check whether an index is supported by YB. This diff also adds the `yb_am_bind_schema` method the same interface. This method allows YB AM's to customize underlying DocDB schema creation and also pass along any extra metadata. Right now, the two YB AM's, yblsm and ybgin, implement the same logic for this method. These changes are useful for upcoming changes where new index types are added by users or extensions, more specifically for the upcoming addition of a vector-based AM. Jira: DB-11118 Test Plan: Jenkins Reviewers: jason Reviewed By: jason Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D34676
… linux versions Summary: Fixed: useAsDefault not set when new bundle is added Test Plan: tested manually Reviewers: lsangappa Reviewed By: lsangappa Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D34684
…ster Summary: Remove SystemdUpgrade from IN_TRANSIT list because we marked SystemdUpgrade as retryable Test Plan: run itest Reviewers: svarshney Reviewed By: svarshney Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D34693
…call races. Summary: The existing `GetClusterConfig()` function can segfault if it is called while the catalog manager is reloading the system catalog because the `cluster_config_` field is reset during catalog loading. However it does already return a `Status` object, there is just no codepath that returns a not-OK status. This diff refactors the function to return a `Result` instead, and returns a not-OK status if the `cluster_config_` field is null. Since all clients are already dealing with the returned `Status` object in the old signature, this was a minimal change that makes calls safe, regardless of whether the caller holds the leader lease or not. Jira: DB-10650 Test Plan: Existing tests to be sure the refactor is safe. I didn't add a test that the race is fixed as the race is difficult to expose deterministically. Reviewers: asrivastava, mhaddad, xCluster, hsunder Reviewed By: asrivastava, hsunder Subscribers: ybase, esheng, slingam Differential Revision: https://phorge.dev.yugabyte.com/D34609
* added new TA * edits to change the case * Apply suggestions from code review Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com> --------- Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com>
…ascade
Summary:
According to the test comments, the intention is to check that after a sequence that is used as a default value for a column is dropped, the dependency between the "table column type" and the sequence does not exist. However, such a dependency is never created and the logic for retrieving the dependency is incorrect:
```
rs = statement.executeQuery("SELECT objid FROM pg_depend WHERE " +
"classid=" + PG_TYPE_OID + " AND refobjid=" + oidSequence);
```
This code retrieves the dependency between the sequence and it's associated type.
This diff changes the test to instead check for the dependency between the column's default (metadata stored in `pg_attrdef`) and the sequence. Specifically, it changes the above query to:
```
rs = statement.executeQuery("SELECT objid FROM pg_depend WHERE " +
"classid=" + PG_ATTRDEF_OID + " AND refobjid=" + oidSequence);
```
Test Plan:
Jenkins: test regex: .*TestPgDepend.*
./yb_build.sh --java-test 'org.yb.pgsql.TestPgDepend#testSequenceDeletionWithCascade'
Reviewers: myang
Reviewed By: myang
Subscribers: yql
Differential Revision: https://phorge.dev.yugabyte.com/D34708
Summary: Hide autoflags from users while listing gflags in UI. Test Plan: validated in UI that autoflags are not available in UI after changes. Reviewers: asharma, #yba-api-review, dkumar Reviewed By: asharma, #yba-api-review, dkumar Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D34719
…TLIER Summary: Fix layout for ASH view graph in case of OVERALL and OUTLIER The buttons were coming out of the screen Test Plan: Fixed layout for ASH view graph in case of OVERALL and OUTLIER Reviewers: cdavid, amalyshev Reviewed By: cdavid Differential Revision: https://phorge.dev.yugabyte.com/D34635
Summary: Support grouping by event type for ASH graphs Test Plan: unit tested + tested manually. Reviewers: rmadhavan Reviewed By: rmadhavan Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D34740
…te#19672) * Track UTMs to monitor traffic driving sign-ups for YBM * Remove unused code and improvement related to the cookie set/get
…e metics page
Summary:
Noticed metrics heading is same for 2 metrics in 2 sections of the metics page
1. YSQL Op Latency (Avg) is being repeated twice and hence fixing the defect
2. update gflag keyword regex
Test Plan:
Please refer to the screenshot
{F174656}
Reviewers: amalyshev, jmak
Reviewed By: jmak
Subscribers: yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D34645
Summary:
Added unit tests for verifying LDAP and OIDC values
1. Tried with valid values
2. Tried with wrong attr
3. Tried with missing "=" sign between attr key and attr values
4. Modified first instance of LDAP and JWT with slight changes to verify if thats valid
5. Tried missing quotes around attr value and verified if it throws an error
Test Plan:
Please refer to screenshot
{F175176}
Reviewers: jmak
Reviewed By: jmak
Subscribers: yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D34706
Summary: Add ability to groupBy Event type for ASH graphs Test Plan: Tested locally Reviewers: amalyshev, cdavid Reviewed By: amalyshev Differential Revision: https://phorge.dev.yugabyte.com/D34743
Summary: According to the postgres protocol (https://www.postgresql.org/docs/current/protocol-flow.html), the server will always respond with a ReadyForQuery message when it finishes processing a request to let the client know that it's ready to process a new query. We set the metadata after the first input message after a ReadyForQuery message was sent. We unset this metadata when we are just about the send a ReadyForQuery message to the client. This way we don't have to worry about all the individual cases, and whether this is a simple query or an extended query. This means that for every incoming request from the client, we set the root request id once. If there is a multi-statement query request, all the queries will have the same root request id because all of them came in a single request from the client. Note that the query id will not be same in this case, it will keep changing as the server keeps processing each individual query. Only in case the client sends a termination request, or the connection is terminated unexpectedly, we set the metadata but never unset it. But it should be fine, since the backend process is killed just after that, that means we cannot fetch samples for this backend. ``` bin/ysqlsh -c "create table if not exists a(id int); insert into a select i from generate_series(1, 1000) as i;" yugabyte=# SELECT yugabyte-# SUBSTRING(query, 1, 50) AS query, yugabyte-# root_request_id, yugabyte-# wait_event_component, yugabyte-# wait_event, yugabyte-# wait_event_type, yugabyte-# COUNT(*) yugabyte-# FROM yugabyte-# yb_active_session_history yugabyte-# JOIN yugabyte-# pg_stat_statements yugabyte-# ON yugabyte-# query_id = queryid yugabyte-# WHERE yugabyte-# sample_time >= current_timestamp - interval '20 minutes' yugabyte-# GROUP BY yugabyte-# query, yugabyte-# root_request_id, yugabyte-# wait_event_component, yugabyte-# wait_event, yugabyte-# wait_event_type yugabyte-# ORDER BY yugabyte-# root_request_id, yugabyte-# query, yugabyte-# wait_event_component, yugabyte-# wait_event_type; query | root_request_id | wait_event_component | wait_event | wait_event_type | count ----------------------------------------------------+--------------------------------------+----------------------+--------------------------+-----------------+------- create table if not exists a(id int) | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | TServer | OnCpu_Active | Cpu | 2 create table if not exists a(id int) | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | TServer | YBClient_WaitingOnDocDB | Network | 13 create table if not exists a(id int) | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | YSQL | QueryProcessing | Cpu | 11 create table if not exists a(id int) | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | YSQL | CatalogRead | Network | 13 create table if not exists a(id int) | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | YSQL | StorageFlush | Network | 1 insert into a select i from generate_series($1, $2 | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | TServer | Raft_ApplyingEdits | Cpu | 2 insert into a select i from generate_series($1, $2 | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | TServer | OnCpu_Passive | Cpu | 2 insert into a select i from generate_series($1, $2 | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | TServer | YBClient_LookingUpTablet | Network | 1 insert into a select i from generate_series($1, $2 | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | TServer | YBClient_WaitingOnDocDB | Network | 2 insert into a select i from generate_series($1, $2 | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | YSQL | QueryProcessing | Cpu | 2 insert into a select i from generate_series($1, $2 | 0e5eed5a-a324-0ffb-d53d-58b6d01c48a7 | YSQL | StorageFlush | Network | 3 ``` Jira: DB-11338 Test Plan: Jenkins Reviewers: jason Reviewed By: jason Subscribers: hbhanawat, amitanand, yql Differential Revision: https://phorge.dev.yugabyte.com/D35126
…L Connection Manager enabled Summary: When attempting to use startup parameters with connection manager, any parameter values that have spaces in them, for example `transaction_isolation = read committed`, would get parsed incorrectly. Odyssey's code uses any space found in the "value" as the cue to end parsing the value for a certain startup parameter; this patch resolves this by instead searching for the next option as per the syntax of passing multiple pg options. This patch also modifies TestSessionParameters.java to allow testing of startup parameters with spaces. Jira: DB-11168 Test Plan: Jenkins: test regex: .*TestSessionParameters.* ./yb_build.sh --java-test org.yb.ysqlconnmgr.TestSessionParameters Reviewers: mkumar, nkumar Reviewed By: mkumar Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D34814
abhinab-yb
approved these changes
May 20, 2024
docs/content/preview/explore/query-1-performance/ycql-stat-statements.md
Outdated
Show resolved
Hide resolved
hbhanawat
reviewed
May 20, 2024
docs/content/preview/explore/query-1-performance/ycql-stat-statements.md
Outdated
Show resolved
Hide resolved
docs/content/preview/explore/query-1-performance/ycql-stat-statements.md
Outdated
Show resolved
Hide resolved
|
|
||
| Databases can be resource-intensive, consuming a lot of memory CPU, IO, and network resources. Optimizing your CQL can be very helpful in minimizing resource utilization. The `ycql_stat_statements` module helps you track planning and execution statistics for all the YCQL statements executed by a server. | ||
|
|
||
| This view provides YCQL statement metrics (similar to pg_stat_statements) that are also present on `<yb-tserver-ip>:12000/statements`, accessible via YSQL. The view can be joined with YCQL wait events in the [yb_active_session_history](../../observability/active-session-history/#yb-active-session-history) view using the query ID. |
There was a problem hiding this comment.
This view provides YCQL statement metrics (similar to pg_stat_statements) that are also present on <yb-tserver-ip>:12000/statements, accessible via YSQL
Please remove ", accessible via YSQL. It is not making sense. This is a YCQL endpoint only.
There was a problem hiding this comment.
Oh I added it based on this response : https://docs.google.com/document/d/1VRiPYWTclpxBnyxxix9yD87WIZQrsALviDY5qZXha9Y/edit?disco=AAABMAeLnqA
@hbhanawat
…es (yugabyte#22367) * [DOC-310] Explained the meaning of upgrade delay between nodes * Update docs/content/preview/yugabyte-platform/manage-deployments/upgrade-software-install.md --------- Co-authored-by: Aishwarya Chakravarthy <ashchakravarthy@gmail.com>
… and pre-aggregate data inside the DB. Summary: Right now we load all the raw data to build TS storage based graph in memory and aggregate in TS code. This makes PG storage to perform aggregation and we only load results in memory. We still perform topK filtering and average across nodes line calculation in TS. Test Plan: Unit tested Reviewers: rmadhavan Reviewed By: rmadhavan Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D34873
* removed port * added comment
* Update docs * Update yb-tserver.md * Update yb-master.md
…tion info Summary: Adding `GetXClusterOutboundReplicationGroups` and `GetXClusterOutboundReplicationGroupInfo` RPCs and corresponding `list_xcluster_outbound_replication_groups` and `get_xcluster_outbound_replication_group_info` yb-admin commands. **Upgrade/Rollback safety:** Only adding new APIs. All changed are guarded under `enable_xcluster_api_v2` Fixes yugabyte#22278 Jira: DB-11194 Test Plan: XClusterDBScopedTest.TestYbAdmin XClusterOutboundReplicationGroupTest.TestListAPIs Reviewers: jhe, xCluster, mlillibridge Reviewed By: mlillibridge Subscribers: cwang, mlillibridge, ybase Differential Revision: https://phorge.dev.yugabyte.com/D35011
Summary: NamespaceReplication is not in use and the tests are flaky. Disabling them for now. List of tests: SetupNamespaceReplicationWithBootstrap SetupNamespaceReplicationWithBootstrapFailToSendSnapshot SetupNamespaceReplicationWithBootstrapFailCreateProducerSnapshot SetupNamespaceReplicationWithBootstrapFailRestoreSnapshot SetupNamespaceReplicationWithBootstrapRequestFailures Fixes yugabyte#22457 Jira: DB-11376 Test Plan: Jenkins Reviewers: jhe, slingam, xCluster, yyan Reviewed By: yyan Subscribers: ybase Differential Revision: https://phorge.dev.yugabyte.com/D35205
…nfiguration section for K8s universes. (yugabyte#22285) * first cut * added k8s operator page * added image * added upgrade instructions * updated instance configuration for 2024.1 * added screenshot * minor edits --------- Co-authored-by: Dwight Hodge <ghodge@yugabyte.com>
Summary:
Allow users to create DR with db scoped replication on YBA.
Users can enable this feature by enabling the `yb.xcluster.db_scoped.enabled` runtime flag. If the `dbScoped` parameter is passed into the the `Create` endpoint and the runtime flag is enabled, a db scoped DR config will be created.
Currently only supports the most basic creation. That is, source and target universe do not have TLS enabled, and bootstrapping is not needed.
Example of create API:
```
curl --location '<platform-url>/api/v1/customers/f33e3c9b-75ab-4c30-80ad-cba85646ea39/dr_configs' \
--header 'X-AUTH-YW-API-TOKEN: <api-token' \
--header 'Content-Type: application/json' \
--data '{
"name": "dr-config-test",
"sourceUniverseUUID": "53297075-ec69-4235-aeae-eceb7f3e7208",
"targetUniverseUUID": "bdb574d2-ae9e-4834-ad93-2f0b5e63229a",
"dbs": [
"000033c3000030008000000000000000"
],
"dryRun": false,
"bootstrapParams": {
"backupRequestParams": {
"storageConfigUUID": "<storage_config_uuid>"
}
},
"dbScoped": true
}'
```
A new table called xcluster_namespace_config is added to keep track of the dbs/namespaces that are stored per xcluster config, instead of the old `xcluster_table_config` table. Schema below:
```
Table "public.xcluster_namespace_config"
Column | Type | Collation | Nullable | Default
---------------------+-----------------------+-----------+----------+---------
config_uuid | uuid | | not null |
source_namespace_id | character varying(64) | | not null |
Indexes:
"pk_xcluster_namespace_config" PRIMARY KEY, btree (config_uuid, source_namespace_id)
Foreign-key constraints:
"fk_xcluster_namespace_config_config_uuid" FOREIGN KEY (config_uuid) REFERENCES xcluster_config(uuid) ON UPDATE CASCADE ON DELETE CASCADE
```
Test Plan:
Added UTs
Create a source and target universe with db version >= `2.23.0.0-b247`.
Add some tables to the `yugabyte` db on both source and target universe without any data in it.
```
CREATE TABLE house(
ID INT PRIMARY KEY NOT NULL,
LOCATION TEXT NOT NULL
);
CREATE TABLE ROOM(
ID INT PRIMARY KEY NOT NULL,
size TEXT NOT NULL
);
create index on house(id);
```
Make a call to the create Dr Config endpoint with `dbScoped` boolean set as true.
Make sure that the task succeeds.
On the source universe master UI xCluster page, we should see (`<master_url>:7000/xcluster`):
{F179208}
On the target universe's master UI xCluster page, we should see (`<master_url>:7000/xcluster`):
{F179209}
Reviewers: #yba-api-review, sanketh, hzare, jmak
Reviewed By: #yba-api-review, hzare
Subscribers: dkumar, sanketh, yugaware
Differential Revision: https://phorge.dev.yugabyte.com/D34802
Summary: This diff adds a tserver flag `ysql_yb_default_replica_identity` which decides the default replica identity for a table at the time of creation. This flag has a default value of "CHANGE". The flag is case sensitive and can take one of these four values "FULL", "DEFAULT", "NOTHING", and "CHANGE. If the flag is assigned any value other than these four, it treats it as an invalid input and the table creation fails. Jira: DB-11234 Test Plan: ./yb_build.sh --java-test 'org.yb.pgsql.TestPgAlterTable#testReplicaIdentity' ./yb_build.sh --cxx-test pg_mini-test --gtest_filter PgMiniTest.AlterTableWithReplicaIdentity Reviewers: stiwary, asrinivasan, skumar Reviewed By: stiwary Subscribers: yql, ybase, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D34942
Summary: On a customer env, we noticed that when an incremental backup time expires, YBA generates a new time, but unfortunately, it was giving old time as during that check, it wasn't expired, we suspect this could be due to clock skewness so we did following 2 improvements: - Use the same date variable while comparing and getting new incremental task times. - Always add incremental frequency time, if old values exists Test Plan: Tested manually to see that the incremental backup schedule works as expected. Reviewers: vkumar, vpatibandla Reviewed By: vkumar Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D35193
Summary: Modified several upgrade tests to fit into new test schema. Test Plan: sbt test Reviewers: nsingh, cwang Reviewed By: cwang Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D34827
… records are added to response Summary: At the end of PopulateCDCSDKWriteRecord(), if we have not added any DML records (scenario described below) to the GetChanges response, we want to remove the BEGIN record that was added at the start of this method. To remove this BEGIN record, we were working under the assumption that the last added record in the response would be the BEGIN record and hence simply removing the last record without verifying its op. But, during the 1st GetChanges call, right after the BEGIN record, we add a DDL record. Therefore, the assumption becomes incorrect. Hence, the logic has been fixed now to find out the last added BEGIN record by iterating over the response and then removing it. One scenario where we do not add a DML record to the response: When a user has updated a table row that does not exist. Jira: DB-10540 Test Plan: Jenkins ./yb_build.sh --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestUpdateOnNonExistingEntry ./yb_build.sh --cxx-test cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestVWALConsumptionWhileUpdatingNonExistingRow Reviewers: asrinivasan, stiwary, skumar, vkushwaha Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D35188
…tel config Summary: Use exporter UUID instead of export config name in the otel config. Test Plan: Manually tested and checked the nodes otel collector config file. Reviewers: amalyshev Reviewed By: amalyshev Subscribers: yugaware Differential Revision: https://phorge.dev.yugabyte.com/D35196
Summary:
Currently we show the local calls section in /rpcz by default. But this only makes sense for the tserver end point, not the cql end point.
With this change we only fetch the local-calls only for RpcServers which have ShouldExportLocalCalls as true.
This is set to true only for TServer.
Jira: DB-11367
Test Plan:
Test on a local cluster
```
dev-server-amitanand2:~/code/yugabyte-db [:1f37b8438c|✚ 5]
04:19 $ curl 127.0.0.1:9000/rpcz 2>/dev/null | jq '.local_calls'
{
"remote_ip": "local calls",
"state": "OPEN"
}
dev-server-amitanand2:~/code/yugabyte-db [:1f37b8438c|✚ 5]
04:19 $ curl 127.0.0.1:12000/rpcz 2>/dev/null | jq '.local_calls'
null
dev-server-amitanand2:~/code/yugabyte-db [:1f37b8438c|✚ 5]
04:19 $ curl 127.0.0.1:7000/rpcz 2>/dev/null | jq '.local_calls'
null
dev-server-amitanand2:~/code/yugabyte-db [:1f37b8438c|✚ 5]
04:19 $ curl 127.0.0.1:13000/rpcz 2>/dev/null | jq '.local_calls'
null
```
Reviewers: asaha
Reviewed By: asaha
Subscribers: esheng, ybase
Differential Revision: https://phorge.dev.yugabyte.com/D35220
Summary: This change implements the YSQL side of vector index creation. This diff adds support for index creation statements with a dummy ANN method called `ybdummyann` for now in the form ``` create extension vector; CREATE TABLE items (id bigserial PRIMARY KEY, embedding vector(3)); CREATE INDEX ON items USING ybdummyann (embedding vector_l2_ops); ``` This creates an inverted index in DocDB with a schema that looks like `BaseYBCTID | embedding |` With only `BaseYBCTID` as the key. We can do an index ANN scan based on certain query vector such as ` SELECT * FROM items ORDER BY embedding <-> '[1.0, 0.4, 0.35]' LIMIT 5; ` or an index only scan such as ` SELECT embedding FROM items ORDER BY embedding <-> '[1.0, 0.4, 0.35]' LIMIT 5; ` Note that the results from a `ybdummyann` index won't actually be sorted by their distance from the given query vector as the DocDB side of vector indexing has not been implemented. This is made clear by the following client warning when such an index is created. In the future, when we fully have end-to-end support of vector indexing we will add index AM's such as `hnsw` and `ivfflat` meant for external usage. ``` WARNING: ybdummyann is meant for internal-testing only. It does not yield ordered results. ``` When a vector index is created, a message of type `PgVectorIdxOptionsPB` found in `common.proto` is populated into `IndexInfo`. A log message has been inserted into `tablet.cc` to show how this can be accessed. Vector index scans populate a field of type `PgVectorReadOptionsPB` in the `PgsqlReadRequestPB`. The relcache preloader is adjusted to not load index relations whose user-defined AM handler procs might not be loaded yet. A new access method handler called `ybdummyannhandler` is created by this diff. Any future vector index AM/AM handler will share functionality very similar to `ybdummyann`. For this reason, this common functionality is all placed in `src/ybvector/ybvector*`. The main remaining TODOs after this change are: - Build out DocDB side. - Add capabilities to mergesort rows from tablets based on their distance from the query vector. - Add an extra key column to denote future sharding information of each row. - Allow included values. - Allow a mix of vector and non-vector key attributes. **Upgrade/Rollback safety:** This adds vector index protobuf fields that should not be used by anybody production customer right now. Jira: DB-11118 Test Plan: ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressThirdPartyExtensionsPgvector' Reviewers: timur, jason, mbautin, sergei Reviewed By: timur, jason Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D34200
Summary: Adding PG_CRON_LEADER Stateful service to ensure there is only one cron leader running in the universe at any time. Yb-master will create the Stateful service when the `cron.job` table is created. It never gets dropped once created. The same Stateful service will be used if cron extension is dropped and recreated, even on another db. The Stateful service is activated after its underlying raft peer becomes a leader. It will be deactivated when the peer loses the leadership. There can be an overlap between the deactivation on one node and activation on another node. To protect against it Stateful services check their term around critical sections. The cron leader Stateful service sets a 60s lease (FLAGS_ pg_cron_leader_lease_sec). This is refreshed every 10s (cron_leadership_refresh_sec) after checking if the term is valid. It sets the shared memory `cron_leader_lease_` to indicate for how long the lease will be valid for. This ensures pg cron runs only when both Tserver and pg are healthy and it safely stops if the Stateful service or raft gets stuck (like during high Cpu). When the Stateful service gets activated in a new node it first waits out the lease period of 60s before setting its local shared memory. Pg_cron launcher(a pg backend) makes sure the specified lease time in the shared memory has not expired inorder to act as the leader. - We reset the catalog cache version in the cron background worker to handle cases when the job is scheduled on a different node. - Converted `enable_pg_cron` to a preview flag. - Using a generic (id int64, data jsonb) schema for the PG_CRON_LEADER Stateful service tablet. This is not currently used. - Added gFlag `ysql_cron_database_name` which will update the `cron.database_name` guc. This is NON_RUNTIME since the change requires restart of pg_cron to kill the inflight jobs. - Cherry picking commit 19f8ebf9349b6a3642e81a4d19dd0ea967d3f357 from pg_cron. **Upgrade/Downgrade safety** New service is only enabled if the flag `enable_pg_cron` is enabled. This flag Fixes yugabyte#22360 Jira: DB-11263 Test Plan: PgCronTest.GracefulLeaderMove PgCronTest.LeaderCrash PgCronTest.TaskOnDifferentDB PgCronTest.ChangeCronDB Reviewers: tnayak, fizaa Reviewed By: tnayak Subscribers: jason, yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D35009
svarnau
force-pushed
the
master
branch
2 times, most recently
from
fe7d36e
to
d212e43
Compare
|
Created another one - #22564 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.