What's New in v2.1.0-beta.20180827

August 27, 2018

We have now transitioned into the CockroachDB 2.1 Beta phase and will be releasing weekly until the GA release. This week's release includes PostgreSQL compatibility enhancements, general usability improvements, performance improvements, and bug fixes. In addition, we want to highlight a few major benefits:

  • Automatic performance optimizations - Range leases are now automatically rebalanced throughout the cluster to even out the amount of QPS being handled by each server.
  • Better controls for geo-distributed clusters - We’ve added more sophisticated support for controlling the network interfaces to use in certain situations, so nodes can prefer local, private IPs for inter-DC communication, and only use public IPs when making hops that must go over the open internet. See the --locality-advertise-addr flag of the cockroach start command for more details.

Get future release notes emailed to you:

Downloads

Backward-incompatible changes

  • Support for PostgreSQL's TIMETZ data type has been removed due to incomplete/incorrect implementation. This feature was available only in previous 2.1 alpha releases. Before upgrading to this release, tables with the TIMETZ type must be dropped entirely; it is not possible to convert the data or drop a single TIMETZ column. #28095
  • Support for the BIT data type has been removed due to incorrect implementation and incompatibility with some client apps. Tables with the BIT type will continue to work but will see their type automatically changed to INT in the output of SHOW TABLES, information_schema, etc. This is backward-compatible insofar that the previous BIT type in CockroachDB was actually a simple integer. A PostgreSQL-compatible replacement will likely be added at a later time. #28814

General changes

  • CockroachDB now supports a separate CA (ca-ui.crt) and certificate (ui.crt) for the Web UI. #27916
  • The ability to set lease placement preferences in replication zones is now fully supported. Existing lease placement preferences will continue to function as in v2.0. #28261
  • The new /_admin/v1/enqueue_range admin server endpoint runs a specified range through a specified internal queue on one or all nodes. The skip_should_queue parameter can also be specified to tell the system to blindly run without first checking whether it needs to be run. This endpoint is intended primarily for debugging purposes. #26554
  • If enabled, anonymous diagnostics reporting now includes hardware and OS information as well as basic stats about the size of IMPORT jobs. #28676 #28726

Enterprise edition changes

  • This release includes several changes to the Change Data Capture feature:
    • CHANGEFEEDs now support interleaved tables #27991
    • CREATE CHANGEFEED now requires an enterprise license when used with Kafka. #27962
    • CHANGEFEEDs now produce an error when column families are added (instead of returning incorrect results) and when targeting system tables (instead of operating with undefined behavior). #27962
    • CREATE CHANGEFEED is now restricted to superusers. #27962
    • CHANGEFEED job descriptions now substitute values for SQL placeholders. #28220
    • CHANGEFEEDs can now only target lists of physical tables. #27996
    • CHANGEFEEDs now produce an error when a watched table is truncated, dropped, or renamed. #28204
    • CHANGEFEED Kafka tunings have been adjusted for faster flushes, improving throughput. #28586
    • CHANGEFEEDs now checkpoint progress more granularly. #28319
    • CHANGEFEEDs now export metrics for production monitoring. #28162
    • The CHANGEFEED timestamp option has been split into updated and resolved. #28733
    • CHANGEFEEDs are now executed using our distributed SQL framework. #28555
  • This release includes the following changes to the Encryption At Rest feature:
    • The status of encryption is now written to debug logs. #27880
    • Data keys are now rotated while nodes are running. #28148
    • The new cockroach debug encryption-status command displays encryption key information. #28582

SQL language changes

  • Added foreign key support to IMPORT ... MYSQLDUMP. #27861
  • The output of SHOW GRANTS is now fully sorted. #27884
  • Reads from Google Cloud Storage for IMPORT or BACKUP jobs are now more resilient to quota limits. #27862
  • The ORDER BY INDEX notation now implies an ordering by the implicit primary key columns appended to an index. #27812
  • Added the server_encoding session variable and protocol status parameter, for compatibility with PostgreSQL. It is set to UTF8 and cannot be changed. #27943
  • Extended support of the extra_float_digits session variable, for compatibility with PostgreSQL. #27952
  • Improved the handling of SET, RESET and SHOW, for better compatibility with PostgreSQL. #27947
  • Exposed the integer_datetimes session variable in SHOW and pg_settings, for compatibility with PostgreSQL. #27947
  • The default values of the client_min_messages and extra_float_digits session variables now match PostgreSQL. #27947
  • Corrected the oids and formatting of some columns in the pg_catalog.pg_index table. #27961
  • The distribution of queries that use the repeat() built-in function are now permitted. #28039
  • Statement statistics are now grouped separately for queries using the cost-based optimizer and heuristic planner. #27806
  • CockroachDB now supports empty tuples with the syntax (), 1-valued tuples with the syntax (x,) in addition to row(x), and the ability to use IN with an empty tuple as right operand. This is a CockroachDB extension. #28143
  • CockroachDB now supports constructing array values using parentheses, for example ARRAY(1,2,3,4). This is a CockroachDB extension; the standard PostgreSQL syntax ARRAY[1,2,3,4] remains supported. #28238
  • CockroachDB now supports converting arrays and tuples to strings, for compatibility with PostgreSQL. #28183
  • ANY/ALL/SOME comparisons are now more permissive about the types of their input expressions, and comparisons with empty tuples are now allowed. #28226
  • Improved the handling of decimal 0s. Specifically, -0 is coerced to 0 and values like 0.00 retain the digits after the decimal point. #27978
  • Arrays of arrays are no longer allowed, even as intermediate results. #28116
  • IMPORT ... PGDUMP now supports CockroachDB dump files. #28359
  • The decimal variants of the ceil() and ceiling() functions now return 0 where they would have returned -0 previously. #28366
  • Improved support for S3-compatible endpoints in BACKUP, RESTORE, and IMPORT. The AWS_REGION parameter is no longer required. Services like Digital Ocean Spaces and Minio now work correctly. #28394
  • CockroachDB now supports an optional FILTER clause with aggregates when used as window functions. #28357
  • Normalized the case of table names imported via IMPORT ... MYSQLDUMP. #28397
  • All queries now run through the DistSQL execution engine. #27863
  • It is now an error to specify both FORCE_INDEX and NO_INDEX_JOIN hints at the same time. #28411
  • Added numeric_precision_radix to the information_schema.columns table. #28467
  • Added the schemachanger.lease.duration and schemachanger.lease.renew_fraction cluster settings to control the schema change lease. #28342
  • Added the string_agg() aggregation function, which concats a collection of strings into a single string and separates them with a specified delimiter. #28392
  • CockroachDB now fully supports the RANGE mode for specification of window function frames. #27022
  • CockroachDB now supports the GROUPS mode for specification of window function frames. #28244
  • CockroachDB now supports the ARRAY() operator and comparisons with sub-queries on the right side of the comparison, when they appear themselves in sub-queries. #28618
  • CockroachDB now supports two experimental compatibility modes with how PostgreSQL handles SERIAL and sequences, to ease reuse of 3rd party frameworks or apps developed for PostgreSQL. These modes can be enabled with the experimental_serial_normalization session variable (per client) and sql.defaults.serial_normalization cluster setting (cluster-wide). The first mode, virtual_sequence, enables compatibility with many applications using SERIAL with maximum performance and scalability. The second mode, sql_sequence, enables maximum PostgreSQL compatibility but uses regular SQL sequences and is thus subject to performance constraints. #28575
  • The output of SHOW COLUMNS now indicates which columns are hidden. #28750
  • SHOW CREATE now reports the FLOAT column types as FLOAT4 and FLOAT8 (the default) instead of REAL and FLOAT. #28776

Command-line changes

  • This release includes the following changes to the cockroach start command:
    • The new --listen-addr flag recognizes both a hostname/address and port and replaces the --host and --port flags, which are now deprecated for cockroach start but remain valid for other client commands. The port portion of --listen-addr can be either a service name or numeric value; when specified as 0, a port number is automatically allocated. #27800 #28373 #28502
    • The new --advertise-addr flag recognizes both a hostname/address and port and replaces the --advertise-host and --advertise-port flags, which are now deprecated. The port portion of --advertise-addr can be either a service name or numeric value; when specified as 0, a port number is automatically allocated. #27800 #28373 #28502
    • The new --http-addr flag recognizes both a hostname/address and port and replaces the --http-host flag, which is now deprecated. The port portion of --http-addr can be either a service name or numeric value; when specified as 0, a port number is automatically allocated. #28373 #28502
    • The new --locality-advertise-addr flag can be used advertise a hostname/address and port to other CockroachDB nodes for specific localities. This is useful in deployments with "local" or "private" interfaces that are only accessible by a subset of the nodes and "global" or "public" interfaces that are slower or more expensive but accessible by any node. In such cases, --locality-advertise-addr can be used to route traffic over the local interface whenever possible. #28531
    • The command now reports the URL of the web UI with the prefix "webui:", not admin:. #28038
    • The command now reports a warning if more than 75% of available RAM is reserved by --cache and --max-sql-memory. #28199
    • The command now suggests which command-line flags to use to access the newly started node in client commands (e.g., cockroach quit, etc.). #28198 cockroach start
  • This release includes the following changes to cockroach client commands:
    • Client commands now better attempt to inform the user about why a connection is failing. #28200
    • Client commands that print out SQL results now issue a warning if more than 10000 result rows are buffered in the table formatter. #28490
    • Client commands that use a SQL connection (e.g., cockroach sql, cockroach node, cockroach user) now produce an error if a connection could not be established within 5 seconds instead of waiting forever. #28326
    • The cockroach sql command and other client commands that display SQL results now use the new table result formatter by default, replacing the previous formatter called pretty. This provides more compact and more reusable results. #28465
    • The cockroach sql command and other client commands that display SQL results containing byte arrays now print them as if they were converted by a SQL cast to the STRING type. #28494
    • The --host flag and COCKROACH_HOST environment variable for client commands now recognize both a hostname/address and port number. The --port flag is still recognized but no longer documented; --host is now preferred. The COCKROACH_PORT environment variable is now deprecated in favor of COCKROACH_HOST. Also, the syntax to specify IPv6 addresses has been changed to use square brackets, for example, --host=[::1] instead of just --host=::1; the previous syntax is still recognized for backward compatibility but is deprecated. #28373
  • The new timeseries.storage.10s_resolution_ttl and timeseries.storage.30m_resolution_ttl cluster settings control how long time series data is retained on the cluster. They work with the recently added "roll-ups" to allow longer retention of time series data while consuming considerably less disk space. #28169
  • The cockroach demo command now supports starting with one of various datasets loaded. #28383
  • The file generated by running cockroach debug zip now contains the contents of the system.rangelog table, which is a record of range splits and rebalances in the cluster. The problem ranges report is now included as well. #28396 #28253
  • The cockroach node status command now works on unavailable/broken clusters. #28249
  • CockroachDB now reports a non-zero exit status if an attempt is made to use a non-existent command. #28492
  • CockroachDB now attempts to inform the operator if the names and IP addresses listed in the configured certificates do not match the server configuration. #28502
  • Added a locality filter for the cockroach gen haproxy command #28649

Web UI changes

  • Added disk read and write time charts to the Hardware dashboard. #27977 #28594
  • The Hardware dashboard now shows system and user CPU summed instead of separately, and normalized by number of CPUs. #28596
  • Added a link to the Statements page from the sidebar. #27928
  • The Statements page now reveals whether a SQL query used the new cost-based optimizer. #28094
  • Added the number of CPUs and percentages of memory and disk usage to the Node List. #28189
  • Removed "distsql reads" time series from the SQL dashboard, since execution engines are being merged. #28350
  • The Problem Ranges report now shows the number of replicas that have an excessively large log. #28034
  • The Stores report now shows encryption statistics. #26890
  • Login is now required by default on secure clusters. #28416
  • Enlarged the clickable area on dropdown components to include entirety of the surrounding container. #28331
  • The Jobs page now supports indefinitely-running job types that have a "highwater timestamp", instead of the "fraction completed" used by jobs with a finite task. #28535
  • Improved the alert text that is displayed when the Web UI connection is lost. #28838

Bug fixes

  • Fixed a bug where the Statements page in the Web UI blanked out after reloading itself. #28108
  • CockroachDB no longer erroneously allows generator functions, aggregates, and window functions in the ON clause of joins. #28839
  • Fixed an index-id does not exist error that could happen on ADD COLUMN or DROP COLUMN. #28803
  • Fixed row counts in the output of IMPORT. #28469
  • Fixed various problems related to the rollback of schema changes.#28014 #28050
  • Prevented a node from freezing after DROP DATABASE when the command aborts, and fixed the rare use of an older descriptor after DROP INDEX. #28381
  • Fixed the handling of regular aggregations combined with window functions and columns "as-is". #27897
  • Fixed a panic caused by key-value tracing a plan that uses an index joiner. #27942
  • The bytea_output session variable is now properly effective for distributed queries. #27951
  • Limited the size of "batch groups" when committing a batch to RocksDB to avoid rare scenarios in which multi-gigabyte batch groups are created, which can cause a server to run out of memory when replaying the RocksDB log at startup. #27895
  • Fixed the round-tripping of cast expression formatting in the presence of collated strings. #27941
  • Prevented spurious query errors when planning some complex correlated SRFs through the distributed execution engine. #27995
  • Fixed the handling of frame boundary offsets in WINDOW clauses. #27933
  • Fixed the formatting of time datatypes in some circumstances. #28040
  • Fixed the behavior of crdb_internal.cluster_id() in distributed queries. #28042
  • Fixed incorrect NULL handling in the distributed implementations of INTERSECT and EXCEPT. #28097
  • Corrected erroneous failures of privileged built-ins in queries run through the distributed execution engine. #28107
  • Ensured that the TIMESTAMP data type never retains a timezone and renders consistently across a distributed computation flow. #28112
  • Corrected casts and binary operators between TIMESTAMPTZ and TIMESTAMP in some cases. #28128
  • Prevented some sequence built-ins from incorrectly running in distributed flows. #28114
  • Corrected the round-trip formatting of negative floats and decimals in the context of other expressions when executing in a distributed flow. #28129
  • Fixed bug that could skip the row following a deleted row during BACKUP. #28172
  • The cockroach user set --password command can now change the password of existing users. #28197
  • CockroachDB now supports a wider range of tuple and array values in query results. #28151
  • This release includes the following fixes to the cockroach sql and cockroach demo command:
    • The commands are now properly able to customize the prompt with ~/.editrc on Linux. #28233
    • The commands once again support copy-pasting special unicode character from other documents. #28233
    • The commands once again properly handle copy-pasting a mixture of client-side commands (e.g., \set) and SQL statements. #28235
    • The commands now properly print a warning when a ? character is mistakenly used to receive contextual help in a non-interactive session, instead of crashing. #28324
    • The commands now work properly even when the TERM environment variable is not set. #28613
  • Generator built-ins now correctly return no rows instead of NULL when given NULL arguments. #28252
  • Fixed out-of-memory errors caused by very large raft logs. #28293 #28511
  • Certain queries that use empty arrays constructed from subqueries no longer spuriously fail when executed via the distributed execution engine. #28391
  • SHOW JOBS now uses placeholder values for BACKUP and RESTORE job descriptions. #28321
  • CockroachDB now handles negative FLOAT zeros properly in more cases. #28569
  • CockroachDB now correctly handles computation of array_agg() when used as a window function. #28291
  • Decommissioning multiple nodes is now possible without posing a risk to cluster health. Recommissioning a node no longer requires a restart of the target node to take effect. #28707
  • Fixed a rare scenario where the value written for one system key was seen when another system key was read, leading to the violation of internal invariants. #28794
  • Hidden columns are now listed in information_schema and pg_catalog tables, for better compatibility with PostgreSQL. #28750
  • Casting arrays now correctly preserves NULL values. #28860
  • IMPORT no longer silently converts \r\n characters in CSV files into \n. #28181
  • Fixed initial poor latencies introduced in a recent release. #28599

Performance improvements

  • CockroachDB now periodically refreshes table leases to avoid initial latency on tables that have not been accessed recently. #28725
  • Improved the fixed cost of running distributed sql queries. #27899
  • Prevent large buffer allocation for DML statements with RETURNING clauses. #27944
  • Improved low-level iteration performance in the presence of range tombstones. #27904
  • Data ingested with RESTORE and IMPORT is now eligible for a performance optimization used in incremental BACKUP and CHANGEFEEDs. #27966
  • Reduced lock contention in RemoteClockMonitor. #28000
  • Reduced lock contention in the Replica write path. #24990
  • Reduced lock contention in the Gossip server. #28001 #28127
  • Reduced lock contention and avoided allocations in raftEntryCache. #27997
  • Fixed a batch commit performance regression that reduced write performance by 20%. #28163
  • Greatly improved the performance of catching up followers that are behind when Raft logs are large. #28511
  • Slightly improved the performance of the nextval() sequence function. #28576
  • Reduced the cost of Raft log truncations and increased single-range throughput. #28126
  • Subqueries are now run through the distributed execution engine. #28580
  • Range leases are now automatically rebalanced throughout the cluster to even out the amount of QPS being handled by each node. #28340
  • Greatly improved the performance of deleting from interleaved tables that have ON DELETE CASCADE clauses. #28330

Doc updates

Contributors

This release includes 493 merged PRs by 39 authors. We would like to thank the following contributors from the CockroachDB community:

  • Constantine Peresypkin
  • Garvit Juniwal
  • Joseph Lowinske (first-time contributor, CockroachDB team member)
  • Song Hao
  • Takuya Kuwahara
  • Tim O'Brien (first-time contributor, CockroachDB team member)
  • neeral

Yes No

Yes No

Product

Resources

Learn

Support Channels

Company

Get developer news