Happy Thanksgiving! This release has a major architectural change.

Before:

S3 Client → S3 API → Filer → Volume Server

After (Direct Access):

S3 Client → S3 API → Volume Server
                ↓
              Filer (metadata only)

This change is benchmarked in test cases from user @yahooguntu

• reduces write latency by 30%
• reduce read latency by 5%

• S3

  • S3: prevent deleting buckets with object locking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7434
  • do delete expired entries on s3 list request by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7426
  • S3: Perf related by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7463
  • S3: Enforce bucket policy by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7471
  • S3: add context aware action resolution by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7479
  • Refactor data structure by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7472
  • S3: Lazy Versioning Check, Conditional SSE Entry Fetch, HEAD Request Optimization by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7480
  • S3: Directly read write volume servers by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7481
  • S3: Add tests for PyArrow with native S3 filesystem by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7508
  • Account Info by @digitalinfobr in https://github.com/seaweedfs/seaweedfs/pull/7507
  • S3: JWT generation for volume server authentication by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7514
  • S3: S3 Object Retention API to include XML namespace support by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7517
  • S3: adds FilerClient to use cached volume id by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7518
  • test read write by s3fs and PyArrow native file system for s3 by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7520
  • S3: list owned buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7519
  • S3: set identity to request context, and remove obsolete code by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7523
  • S3: Add Vary header for non-wildcard AllowOrigin by @llc1123 in https://github.com/seaweedfs/seaweedfs/pull/7547
  • S3: Fix encrypted file copy with multiple chunks (#7530) by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7546
  • fix copying for paused versioning buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7548
  • S3: Auto create bucket by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7549
  • Support multiple filers for S3 and IAM servers with automatic failover by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7550
  • Filer, S3: Feature/add concurrent file upload limit by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7554
  • fix(s3api): fix AWS Signature V2 format and validation by @qzhello in https://github.com/seaweedfs/seaweedfs/pull/7488
  • S3: pass HTTP 429 from volume servers to S3 clients by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7556
  • Metrics: Add Prometheus metrics for concurrent upload tracking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7555
  • s3api: Fix response-content-disposition query parameter not being honored by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7559

• Misc

  • Nit: use time.Durations instead of constants in seconds. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7438
  • Fix date string parsing bug for the SQL Engine. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7446
  • chore: fix the diagram in RDMA sidecar readme by @dimaqq in https://github.com/seaweedfs/seaweedfs/pull/7503
  • add build info metrics by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7525
  • hide millseconds in up time by @trim21 in https://github.com/seaweedfs/seaweedfs/pull/7553

• Docker, Helm Charts

  • Don't make nginx the default ingress controller by @facorazza in https://github.com/seaweedfs/seaweedfs/pull/7436
  • docker: fix /data ownership and permission by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7451
  • Add explicit IP and binding parameters in Docker Compose by @gophlet in https://github.com/seaweedfs/seaweedfs/pull/7533

• Master

  • master: fix negative active volumes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7440

• Shell

  • Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7450
  • Rework parameters passing for functions within volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7448
  • Rework parameters passing for functions within ec.rebuild by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7445
  • backup: handle volume not found when backing up by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7465
  • Split logic for volume.check.disk into writable and read-only volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7476
  • Parallelize ec.rebuild operations per affected volume. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7466
  • volume.check.disk: add support for uni- or bi-directional sync between volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7484
  • feat(volume.fix): show all replica locations for misplaced volumes by @CodeRusher in https://github.com/seaweedfs/seaweedfs/pull/7560

• Volume Server

  • Volume Server: avoid aggressive volume assignment by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7501
  • fix: pass proxied query param by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7477
  • Bootstrap logic to fix read-only volumes with volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7531

• Filer

  • de/compress the fs meta file if filename ends with gz/gzip by @shaofeng66 in https://github.com/seaweedfs/seaweedfs/pull/7500
  • filer store: add foundationdb by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7178
  • Add error list each entry func by @tam-i13 in https://github.com/seaweedfs/seaweedfs/pull/7485
  • fix(filer-ui): support folder creation with JWT token in URL by @littlemilkwu in https://github.com/seaweedfs/seaweedfs/pull/7271
  • fix(tikv): replace DeleteRange with transaction-based batch deletes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7557
  • fix(tikv): improve context propagation and refactor batch delete logic by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7558

• Java, Hadoop Compatible File System

  • HDFS: Java client replication configuration by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7526

• @dimaqq made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7503
• @shaofeng66 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7500
• @digitalinfobr made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7507
• @gophlet made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7533
• @llc1123 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7547
• @tam-i13 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7485
• @littlemilkwu made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7271

Full Changelog: https://github.com/seaweedfs/seaweedfs/compare/4.00...4.01

Happy Thanksgiving! This release has a major architectural change.

Before:

S3 Client → S3 API → Filer → Volume Server

After (Direct Access):

S3 Client → S3 API → Volume Server
                ↓
              Filer (metadata only)

This change is benchmarked in test cases from user @yahooguntu

• reduces write latency by 30%
• reduce read latency by 5%

• S3

  • S3: prevent deleting buckets with object locking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7434
  • do delete expired entries on s3 list request by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7426
  • S3: Perf related by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7463
  • S3: Enforce bucket policy by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7471
  • S3: add context aware action resolution by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7479
  • Refactor data structure by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7472
  • S3: Lazy Versioning Check, Conditional SSE Entry Fetch, HEAD Request Optimization by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7480
  • S3: Directly read write volume servers by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7481
  • S3: Add tests for PyArrow with native S3 filesystem by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7508
  • Account Info by @digitalinfobr in https://github.com/seaweedfs/seaweedfs/pull/7507
  • S3: JWT generation for volume server authentication by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7514
  • S3: S3 Object Retention API to include XML namespace support by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7517
  • S3: adds FilerClient to use cached volume id by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7518
  • test read write by s3fs and PyArrow native file system for s3 by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7520
  • S3: list owned buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7519
  • S3: set identity to request context, and remove obsolete code by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7523
  • S3: Add Vary header for non-wildcard AllowOrigin by @llc1123 in https://github.com/seaweedfs/seaweedfs/pull/7547
  • S3: Fix encrypted file copy with multiple chunks (#7530) by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7546
  • fix copying for paused versioning buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7548
  • S3: Auto create bucket by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7549
  • Support multiple filers for S3 and IAM servers with automatic failover by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7550
  • Filer, S3: Feature/add concurrent file upload limit by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7554
  • fix(s3api): fix AWS Signature V2 format and validation by @qzhello in https://github.com/seaweedfs/seaweedfs/pull/7488
  • S3: pass HTTP 429 from volume servers to S3 clients by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7556
  • Metrics: Add Prometheus metrics for concurrent upload tracking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7555
  • s3api: Fix response-content-disposition query parameter not being honored by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7559

• Misc

  • Nit: use time.Durations instead of constants in seconds. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7438
  • Fix date string parsing bug for the SQL Engine. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7446
  • chore: fix the diagram in RDMA sidecar readme by @dimaqq in https://github.com/seaweedfs/seaweedfs/pull/7503
  • add build info metrics by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7525
  • hide millseconds in up time by @trim21 in https://github.com/seaweedfs/seaweedfs/pull/7553

• Docker, Helm Charts

  • Don't make nginx the default ingress controller by @facorazza in https://github.com/seaweedfs/seaweedfs/pull/7436
  • docker: fix /data ownership and permission by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7451
  • Add explicit IP and binding parameters in Docker Compose by @gophlet in https://github.com/seaweedfs/seaweedfs/pull/7533

• Master

  • master: fix negative active volumes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7440

• Shell

  • Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7450
  • Rework parameters passing for functions within volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7448
  • Rework parameters passing for functions within ec.rebuild by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7445
  • backup: handle volume not found when backing up by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7465
  • Split logic for volume.check.disk into writable and read-only volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7476
  • Parallelize ec.rebuild operations per affected volume. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7466
  • volume.check.disk: add support for uni- or bi-directional sync between volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7484
  • feat(volume.fix): show all replica locations for misplaced volumes by @CodeRusher in https://github.com/seaweedfs/seaweedfs/pull/7560

• Volume Server

  • Volume Server: avoid aggressive volume assignment by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7501
  • fix: pass proxied query param by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7477
  • Bootstrap logic to fix read-only volumes with volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7531

• Filer

  • de/compress the fs meta file if filename ends with gz/gzip by @shaofeng66 in https://github.com/seaweedfs/seaweedfs/pull/7500
  • filer store: add foundationdb by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7178
  • Add error list each entry func by @tam-i13 in https://github.com/seaweedfs/seaweedfs/pull/7485
  • fix(filer-ui): support folder creation with JWT token in URL by @littlemilkwu in https://github.com/seaweedfs/seaweedfs/pull/7271
  • fix(tikv): replace DeleteRange with transaction-based batch deletes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7557
  • fix(tikv): improve context propagation and refactor batch delete logic by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7558

• Java, Hadoop Compatible File System

  • HDFS: Java client replication configuration by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7526

• @dimaqq made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7503
• @shaofeng66 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7500
• @digitalinfobr made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7507
• @gophlet made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7533
• @llc1123 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7547
• @tam-i13 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7485
• @littlemilkwu made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7271

Full Changelog: https://github.com/seaweedfs/seaweedfs/compare/4.00...4.01

Happy Thanksgiving! This release has a major architectural change.

Before:

S3 Client → S3 API → Filer → Volume Server

After (Direct Access):

S3 Client → S3 API → Volume Server
                ↓
              Filer (metadata only)

This change is benchmarked in test cases from user @yahooguntu

• reduces write latency by 30%
• reduce read latency by 5%

• S3

  • S3: prevent deleting buckets with object locking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7434
  • do delete expired entries on s3 list request by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7426
  • S3: Perf related by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7463
  • S3: Enforce bucket policy by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7471
  • S3: add context aware action resolution by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7479
  • Refactor data structure by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7472
  • S3: Lazy Versioning Check, Conditional SSE Entry Fetch, HEAD Request Optimization by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7480
  • S3: Directly read write volume servers by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7481
  • S3: Add tests for PyArrow with native S3 filesystem by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7508
  • Account Info by @digitalinfobr in https://github.com/seaweedfs/seaweedfs/pull/7507
  • S3: JWT generation for volume server authentication by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7514
  • S3: S3 Object Retention API to include XML namespace support by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7517
  • S3: adds FilerClient to use cached volume id by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7518
  • test read write by s3fs and PyArrow native file system for s3 by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7520
  • S3: list owned buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7519
  • S3: set identity to request context, and remove obsolete code by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7523
  • S3: Add Vary header for non-wildcard AllowOrigin by @llc1123 in https://github.com/seaweedfs/seaweedfs/pull/7547
  • S3: Fix encrypted file copy with multiple chunks (#7530) by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7546
  • fix copying for paused versioning buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7548
  • S3: Auto create bucket by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7549
  • Support multiple filers for S3 and IAM servers with automatic failover by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7550
  • Filer, S3: Feature/add concurrent file upload limit by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7554
  • fix(s3api): fix AWS Signature V2 format and validation by @qzhello in https://github.com/seaweedfs/seaweedfs/pull/7488
  • S3: pass HTTP 429 from volume servers to S3 clients by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7556
  • Metrics: Add Prometheus metrics for concurrent upload tracking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7555
  • s3api: Fix response-content-disposition query parameter not being honored by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7559

• Misc

  • Nit: use time.Durations instead of constants in seconds. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7438
  • Fix date string parsing bug for the SQL Engine. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7446
  • chore: fix the diagram in RDMA sidecar readme by @dimaqq in https://github.com/seaweedfs/seaweedfs/pull/7503
  • add build info metrics by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7525
  • hide millseconds in up time by @trim21 in https://github.com/seaweedfs/seaweedfs/pull/7553

• Docker, Helm Charts

  • Don't make nginx the default ingress controller by @facorazza in https://github.com/seaweedfs/seaweedfs/pull/7436
  • docker: fix /data ownership and permission by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7451
  • Add explicit IP and binding parameters in Docker Compose by @gophlet in https://github.com/seaweedfs/seaweedfs/pull/7533

• Master

  • master: fix negative active volumes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7440

• Shell

  • Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7450
  • Rework parameters passing for functions within volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7448
  • Rework parameters passing for functions within ec.rebuild by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7445
  • backup: handle volume not found when backing up by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7465
  • Split logic for volume.check.disk into writable and read-only volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7476
  • Parallelize ec.rebuild operations per affected volume. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7466
  • volume.check.disk: add support for uni- or bi-directional sync between volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7484
  • feat(volume.fix): show all replica locations for misplaced volumes by @CodeRusher in https://github.com/seaweedfs/seaweedfs/pull/7560

• Volume Server

  • Volume Server: avoid aggressive volume assignment by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7501
  • fix: pass proxied query param by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7477
  • Bootstrap logic to fix read-only volumes with volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7531

• Filer

  • de/compress the fs meta file if filename ends with gz/gzip by @shaofeng66 in https://github.com/seaweedfs/seaweedfs/pull/7500
  • filer store: add foundationdb by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7178
  • Add error list each entry func by @tam-i13 in https://github.com/seaweedfs/seaweedfs/pull/7485
  • fix(filer-ui): support folder creation with JWT token in URL by @littlemilkwu in https://github.com/seaweedfs/seaweedfs/pull/7271
  • fix(tikv): replace DeleteRange with transaction-based batch deletes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7557
  • fix(tikv): improve context propagation and refactor batch delete logic by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7558

• Java, Hadoop Compatible File System

  • HDFS: Java client replication configuration by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7526

• @dimaqq made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7503
• @shaofeng66 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7500
• @digitalinfobr made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7507
• @gophlet made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7533
• @llc1123 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7547
• @tam-i13 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7485
• @littlemilkwu made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7271

Full Changelog: https://github.com/seaweedfs/seaweedfs/compare/4.00...4.01

Happy Thanksgiving! This release has a major architectural change.

Before:

S3 Client → S3 API → Filer → Volume Server

After (Direct Access):

S3 Client → S3 API → Volume Server
                ↓
              Filer (metadata only)

This change is benchmarked in test cases from user @yahooguntu

• reduces write latency by 30%
• reduce read latency by 5%

• S3

  • S3: prevent deleting buckets with object locking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7434
  • do delete expired entries on s3 list request by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7426
  • S3: Perf related by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7463
  • S3: Enforce bucket policy by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7471
  • S3: add context aware action resolution by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7479
  • Refactor data structure by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7472
  • S3: Lazy Versioning Check, Conditional SSE Entry Fetch, HEAD Request Optimization by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7480
  • S3: Directly read write volume servers by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7481
  • S3: Add tests for PyArrow with native S3 filesystem by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7508
  • Account Info by @digitalinfobr in https://github.com/seaweedfs/seaweedfs/pull/7507
  • S3: JWT generation for volume server authentication by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7514
  • S3: S3 Object Retention API to include XML namespace support by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7517
  • S3: adds FilerClient to use cached volume id by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7518
  • test read write by s3fs and PyArrow native file system for s3 by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7520
  • S3: list owned buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7519
  • S3: set identity to request context, and remove obsolete code by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7523
  • S3: Add Vary header for non-wildcard AllowOrigin by @llc1123 in https://github.com/seaweedfs/seaweedfs/pull/7547
  • S3: Fix encrypted file copy with multiple chunks (#7530) by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7546
  • fix copying for paused versioning buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7548
  • S3: Auto create bucket by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7549
  • Support multiple filers for S3 and IAM servers with automatic failover by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7550
  • Filer, S3: Feature/add concurrent file upload limit by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7554
  • fix(s3api): fix AWS Signature V2 format and validation by @qzhello in https://github.com/seaweedfs/seaweedfs/pull/7488
  • S3: pass HTTP 429 from volume servers to S3 clients by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7556
  • Metrics: Add Prometheus metrics for concurrent upload tracking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7555
  • s3api: Fix response-content-disposition query parameter not being honored by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7559

• Misc

  • Nit: use time.Durations instead of constants in seconds. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7438
  • Fix date string parsing bug for the SQL Engine. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7446
  • chore: fix the diagram in RDMA sidecar readme by @dimaqq in https://github.com/seaweedfs/seaweedfs/pull/7503
  • add build info metrics by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7525
  • hide millseconds in up time by @trim21 in https://github.com/seaweedfs/seaweedfs/pull/7553

• Docker, Helm Charts

  • Don't make nginx the default ingress controller by @facorazza in https://github.com/seaweedfs/seaweedfs/pull/7436
  • docker: fix /data ownership and permission by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7451
  • Add explicit IP and binding parameters in Docker Compose by @gophlet in https://github.com/seaweedfs/seaweedfs/pull/7533

• Master

  • master: fix negative active volumes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7440

• Shell

  • Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7450
  • Rework parameters passing for functions within volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7448
  • Rework parameters passing for functions within ec.rebuild by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7445
  • backup: handle volume not found when backing up by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7465
  • Split logic for volume.check.disk into writable and read-only volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7476
  • Parallelize ec.rebuild operations per affected volume. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7466
  • volume.check.disk: add support for uni- or bi-directional sync between volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7484
  • feat(volume.fix): show all replica locations for misplaced volumes by @CodeRusher in https://github.com/seaweedfs/seaweedfs/pull/7560

• Volume Server

  • Volume Server: avoid aggressive volume assignment by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7501
  • fix: pass proxied query param by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7477
  • Bootstrap logic to fix read-only volumes with volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7531

• Filer

  • de/compress the fs meta file if filename ends with gz/gzip by @shaofeng66 in https://github.com/seaweedfs/seaweedfs/pull/7500
  • filer store: add foundationdb by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7178
  • Add error list each entry func by @tam-i13 in https://github.com/seaweedfs/seaweedfs/pull/7485
  • fix(filer-ui): support folder creation with JWT token in URL by @littlemilkwu in https://github.com/seaweedfs/seaweedfs/pull/7271
  • fix(tikv): replace DeleteRange with transaction-based batch deletes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7557
  • fix(tikv): improve context propagation and refactor batch delete logic by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7558

• Java, Hadoop Compatible File System

  • HDFS: Java client replication configuration by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7526

• @dimaqq made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7503
• @shaofeng66 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7500
• @digitalinfobr made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7507
• @gophlet made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7533
• @llc1123 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7547
• @tam-i13 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7485
• @littlemilkwu made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7271

Full Changelog: https://github.com/seaweedfs/seaweedfs/compare/4.00...4.01

Happy Thanksgiving! This release has a major architectural change.

Before:

S3 Client → S3 API → Filer → Volume Server

After (Direct Access):

S3 Client → S3 API → Volume Server
                 ↓
              Filer (metadata only)

This change is benchmarked in test cases from user @yahooguntu

• reduces write latency by 30%
• reduce read latency by 5%

• S3

  • S3: prevent deleting buckets with object locking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7434
  • do delete expired entries on s3 list request by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7426
  • S3: Perf related by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7463
  • S3: Enforce bucket policy by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7471
  • S3: add context aware action resolution by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7479
  • Refactor data structure by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7472
  • S3: Lazy Versioning Check, Conditional SSE Entry Fetch, HEAD Request Optimization by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7480
  • S3: Directly read write volume servers by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7481
  • S3: Add tests for PyArrow with native S3 filesystem by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7508
  • Account Info by @digitalinfobr in https://github.com/seaweedfs/seaweedfs/pull/7507
  • S3: JWT generation for volume server authentication by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7514
  • S3: S3 Object Retention API to include XML namespace support by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7517
  • S3: adds FilerClient to use cached volume id by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7518
  • test read write by s3fs and PyArrow native file system for s3 by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7520
  • S3: list owned buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7519
  • S3: set identity to request context, and remove obsolete code by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7523
  • S3: Add Vary header for non-wildcard AllowOrigin by @llc1123 in https://github.com/seaweedfs/seaweedfs/pull/7547
  • S3: Fix encrypted file copy with multiple chunks (#7530) by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7546
  • fix copying for paused versioning buckets by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7548
  • S3: Auto create bucket by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7549
  • Support multiple filers for S3 and IAM servers with automatic failover by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7550
  • Filer, S3: Feature/add concurrent file upload limit by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7554
  • fix(s3api): fix AWS Signature V2 format and validation by @qzhello in https://github.com/seaweedfs/seaweedfs/pull/7488
  • S3: pass HTTP 429 from volume servers to S3 clients by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7556
  • Metrics: Add Prometheus metrics for concurrent upload tracking by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7555
  • s3api: Fix response-content-disposition query parameter not being honored by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7559

• Misc

  • Nit: use time.Durations instead of constants in seconds. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7438
  • Fix date string parsing bug for the SQL Engine. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7446
  • chore: fix the diagram in RDMA sidecar readme by @dimaqq in https://github.com/seaweedfs/seaweedfs/pull/7503
  • add build info metrics by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7525
  • hide millseconds in up time by @trim21 in https://github.com/seaweedfs/seaweedfs/pull/7553

• Docker, Helm Charts

  • Don't make nginx the default ingress controller by @facorazza in https://github.com/seaweedfs/seaweedfs/pull/7436
  • docker: fix /data ownership and permission by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7451
  • Add explicit IP and binding parameters in Docker Compose by @gophlet in https://github.com/seaweedfs/seaweedfs/pull/7533

• Master

  • master: fix negative active volumes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7440

• Shell

  • Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7450
  • Rework parameters passing for functions within volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7448
  • Rework parameters passing for functions within ec.rebuild by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7445
  • backup: handle volume not found when backing up by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7465
  • Split logic for volume.check.disk into writable and read-only volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7476
  • Parallelize ec.rebuild operations per affected volume. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7466
  • volume.check.disk: add support for uni- or bi-directional sync between volume replicas. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7484
  • feat(volume.fix): show all replica locations for misplaced volumes by @CodeRusher in https://github.com/seaweedfs/seaweedfs/pull/7560

• Volume Server

  • Volume Server: avoid aggressive volume assignment by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7501
  • fix: pass proxied query param by @kmlebedev in https://github.com/seaweedfs/seaweedfs/pull/7477
  • Bootstrap logic to fix read-only volumes with volume.check.disk. by @proton-lisandro-pin in https://github.com/seaweedfs/seaweedfs/pull/7531

• Filer

  • de/compress the fs meta file if filename ends with gz/gzip by @shaofeng66 in https://github.com/seaweedfs/seaweedfs/pull/7500
  • filer store: add foundationdb by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7178
  • Add error list each entry func by @tam-i13 in https://github.com/seaweedfs/seaweedfs/pull/7485
  • fix(filer-ui): support folder creation with JWT token in URL by @littlemilkwu in https://github.com/seaweedfs/seaweedfs/pull/7271
  • fix(tikv): replace DeleteRange with transaction-based batch deletes by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7557
  • fix(tikv): improve context propagation and refactor batch delete logic by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7558

• Java, Hadoop Compatible File System

  • HDFS: Java client replication configuration by @chrislusf in https://github.com/seaweedfs/seaweedfs/pull/7526

• @dimaqq made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7503
• @shaofeng66 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7500
• @digitalinfobr made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7507
• @gophlet made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7533
• @llc1123 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7547
• @tam-i13 made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7485
• @littlemilkwu made their first contribution in https://github.com/seaweedfs/seaweedfs/pull/7271

Full Changelog: https://github.com/seaweedfs/seaweedfs/compare/4.00...4.01

For new features, improvements, and bug fixes released in v8.5.4 for TiDB, see TiDB v8.5.4 release notes.

See the difference from the issue perspective:

• pingcap/tidb#62608
• pingcap/tidb#59863
• pingcap/tidb#62602
• pingcap/tidb#61550
• pingcap/tidb#63089
• pingcap/tidb#57608
• pingcap/tidb#63260
• pingcap/tidb#58600
• pingcap/tidb#63260
• pingcap/tidb#62799
• pingcap/tidb#62740
• pingcap/tidb#63369
• pingcap/tidb#63216
• pingcap/tidb#60093
• pingcap/tidb#62682
• pingcap/tidb#62861
• pingcap/tidb#63075
• pingcap/tidb#62665
• pingcap/tidb#63358
• pingcap/tidb#61715
• pingcap/tidb#61246
• pingcap/tidb#62069
• pingcap/tidb#61822
• pingcap/tidb#58650
• pingcap/tidb#59939
• pingcap/tidb#62465
• pingcap/tidb#63613
• pingcap/tidb#59822
• pingcap/tidb#60978
• pingcap/tidb#45077
• pingcap/tidb#61792
• pingcap/tidb#62617
• pingcap/tidb#62438
• pingcap/tidb#63638
• pingcap/tidb#62672
• pingcap/tidb#62347
• pingcap/tidb#63346
• pingcap/tidb#51116
• pingcap/tidb#58829
• pingcap/tidb#61083
• pingcap/tidb#60869
• pingcap/tidb#59502
• pingcap/tidb#61441
• pingcap/tidb#58871
• pingcap/tidb#61085
• pingcap/tidb#57691
• pingcap/tidb#60264
• pingcap/tidb#63483
• pingcap/tidb#63348
• pingcap/tidb#63353
• pingcap/tidb#63201
• pingcap/tidb#62424
• pingcap/tidb#62856
• pingcap/tidb#58068
• pingcap/tidb#63176
• pingcap/tidb#63609
• pingcap/tidb#62960
• pingcap/tidb#63663
• pingcap/tidb#63320
• pingcap/tidb#63653
• pingcap/tidb#61327
• pingcap/tidb#63466
• pingcap/tidb#60047
• pingcap/tidb#63525
• pingcap/tidb#63370
• pingcap/tidb#63729
• pingcap/tidb#63650
• pingcap/tidb#63768
• pingcap/tidb#60951
• pingcap/tidb#63873
• pingcap/tidb#63896
• pingcap/tidb#63927
• pingcap/tidb#63895
• pingcap/tidb#63092
• pingcap/tidb#64070
• pingcap/tidb#63870
• pingcap/tidb#63326
• pingcap/tidb#64062
• pingcap/tidb#64144
• pingcap/tidb#64137
• pingcap/tidb#64249
• pingcap/tidb#64282
• pingcap/tidb#64250
• pingcap/tidb#64263
• pingcap/tidb#64495
• pingcap/tidb#64509
• pingcap/tidb#64566
• pingcap/tidb#64572
• pingcap/tidb#64629
• pingcap/tidb#64671

Compared to the previous version 5.3.3, this is a bug-fix release that includes numerous improvements to enhance stability, performance, and correctness. Key highlights include critical dependency upgrades (e.g., Netty, OpenTelemetry), significant stability enhancements like the TimerWheel snapshot mechanism for reliable recovery, and a wide range of bug fixes across modules such as Broker, Proxy, and Client. Thank you for your attention, If you have any suggestions please reply to this email.

• [ISSUE #9337] Upgrade Netty to 4.1.119.Final by @weihubeats in https://github.com/apache/rocketmq/pull/9338
• [ISSUE #9253] Make the message visible earlier when response process encounters errors by @redlsz in https://github.com/apache/rocketmq/pull/9255
• [ISSUE #8651] Fix revive topic process lag metric calculation by @redlsz in https://github.com/apache/rocketmq/pull/8652
• [ISSUE #8629] Add proxy RocketmqTraffic logger appender to separate traffic logs by @redlsz in https://github.com/apache/rocketmq/pull/8630
• [ISSUE #8232] Fix the issue that proxy remoting thread pool isolation does not take effect by @redlsz in https://github.com/apache/rocketmq/pull/8233
• [ISSUE #9414] Updated the Quick Start version in README to the 5.3.3 by @mxsm in https://github.com/apache/rocketmq/pull/9415
• [ISSUE #9412] Flush bak metrics file before deleting the original file by @GenerousMan in https://github.com/apache/rocketmq/pull/9413
• [ISSUE #9403] Fix the issue of metadata synchronization failure between master and slave when the topic has attributes by @RongtongJin in https://github.com/apache/rocketmq/pull/9404
• [ISSUE #9416] Fix batch send messages have the same message id when consumed by @qianye1001 in https://github.com/apache/rocketmq/pull/9417
• [ISSUE #9379] Set compact timestamp when timeStoreTable empty by @lizhimins in https://github.com/apache/rocketmq/pull/9419
• [ISSUE #9418] Fix ColumnFamilyHandle not close in PopConsumerRocksdbStore by @lizhimins in https://github.com/apache/rocketmq/pull/9420
• [ISSUE #9352] Fix wrong value in topic-group cache by @golden-yang in https://github.com/apache/rocketmq/pull/9424
• [ISSUE #9441] Add file length check in IndexService by @lizhimins in https://github.com/apache/rocketmq/pull/9442
• [ISSUE #9443] Fix add wrong value to topicGroupTable in ConsumerManager by @yx9o in https://github.com/apache/rocketmq/pull/9444
• [ISSUE #9254] Add CombineConsumeQueue to support CQ migration by @qianye1001 in https://github.com/apache/rocketmq/pull/9256
• [ISSUE #9439] Add escape for win in the method returning broker configuration by @Crazylychee in https://github.com/apache/rocketmq/pull/9440
• [ISSUE #9451] Optimize back pressure default value to 1024. by @GenerousMan in https://github.com/apache/rocketmq/pull/9452
• [ISSUE #9454] Make sure topic can be deleted when using RocksDB CQ by @qianye1001 in https://github.com/apache/rocketmq/pull/9457
• Fix IllegalStateException during app shutdown caused by premature shutdown trigger by @hepengju in https://github.com/apache/rocketmq/pull/9408
• [ISSUE #9335] Use MixAll.ROCKETMQ_HOME_DIR for home directory by @WJ66880 in https://github.com/apache/rocketmq/pull/9474
• [ISSUE #9481] Fix Issue 4050 didn't fix completely by @WJ66880 in https://github.com/apache/rocketmq/pull/9482
• [ISSUE #9489] Fix clean pull offset in #removeOffset by @3424672656 in https://github.com/apache/rocketmq/pull/9490
• [ISSUE #8920] Refactor SSL context loading process to support multiple protocols dynamic loading by @EnableAsync in https://github.com/apache/rocketmq/pull/9483
• [ISSUE #9491] Remove the volatile declaration for the two variables requestsWrite and requestsRead in the GroupCommitService class by @huanzhikang in https://github.com/apache/rocketmq/pull/9492
• [ISSUE #9486] Fix missing of valueOfCurrentMinusBorn equals checkImmunityTime by @WJ66880 in https://github.com/apache/rocketmq/pull/9487
• [ISSUE #9497] Fix IndexOutOfBoundsException in getEarliestMessageTime when running in IPv6 environment by @woaishixiaoxiao in https://github.com/apache/rocketmq/pull/9498
• [ISSUE #9479] Relax topic length constraint when creating retry and dlq topics by @Allenhuangrui in https://github.com/apache/rocketmq/pull/9480
• [ISSUE #9476] Set queueId to 0 in DefaultTransactionalMessageCheckListener#toMessageExtBrokerInner to improve performance by @mxsm in https://github.com/apache/rocketmq/pull/9477
• [ISSUE #9447] Avoiding possible resource leaks in InputStream by @AROP-REPLICATION in https://github.com/apache/rocketmq/pull/9459
• [ISSUE #9467] Implement paged retrieval of Topic and SubscriptionGroup information by @ltamber in https://github.com/apache/rocketmq/pull/9468
• [ISSUE #8920] Make TLS certificate watch interval configurable by @EnableAsync in https://github.com/apache/rocketmq/pull/9513
• fix CONTRIBUTING.md spelling mistake by @datagramsocket in https://github.com/apache/rocketmq/pull/9511
• [ISSUE #9501] Correcting mismatched comments by @yx9o in https://github.com/apache/rocketmq/pull/9502
• [ISSUE #9515] Return origin handle to consumer when changeInvisibleDuration encounters flow limit by @qianye1001 in https://github.com/apache/rocketmq/pull/9516
• [ISSUE #9254] Fix combineCQ extra search commitLog files for recovery by @qianye1001 in https://github.com/apache/rocketmq/pull/9526
• [ISSUE #9309] Avoid the generation of dirty data in #assignResetOffset by @3424672656 in https://github.com/apache/rocketmq/pull/9310
• [ISSUE #9530] Reduce unnecessary memory allocation in adaptiveBackOffSpinLock by @3424672656 in https://github.com/apache/rocketmq/pull/9530
• [ISSUE #9520] Remove unnecessary topicConfig data version change when enable split registration by @RongtongJin in https://github.com/apache/rocketmq/pull/9521
• [ISSUE #9254] Optimize the logs of the message store by @qianye1001 in https://github.com/apache/rocketmq/pull/9528
• [ISSUE #9537] MQClientAPIFactory Implement NameServerUpdateCallback interface by @qianye1001 in https://github.com/apache/rocketmq/pull/9538
• [ISSUE #9544] Must correct file size when init the file segment by @lizhimins in https://github.com/apache/rocketmq/pull/9545
• [ISSUE #9539] Fix compare policy entry logic in ACL 2.0 by @1782935682 in https://github.com/apache/rocketmq/pull/9540
• [ISSUE #9534] Remove the meaningless brokerVersion by @RongtongJin in https://github.com/apache/rocketmq/pull/9533
• [ISSUE #9555] Fix the npe in the hashCode method when subscriptionDataSet is null by @1782935682 in https://github.com/apache/rocketmq/pull/9556
• [ISSUE #9553] Improve performance by avoiding repeated get(key) by @yx9o in https://github.com/apache/rocketmq/pull/9554
• Limit group max length to 120. by @lollipopjin in https://github.com/apache/rocketmq/pull/9563
• [ISSUE #9254] Refactor notifyMessageArriveInBatch in RocksDBConsumeQueueStore to adapt to CombineConsumeQueueStore by @qianye1001 in https://github.com/apache/rocketmq/pull/9566
• [ISSUE #9572] Unnecessary code comment cleanup by @yx9o in https://github.com/apache/rocketmq/pull/9573
• Fix typos by @co63oc in https://github.com/apache/rocketmq/pull/9560
• [ISSUE #9254] Optimize innerConsumeQueueStoreList order in CombineConsumeQueueStore by @qianye1001 in https://github.com/apache/rocketmq/pull/9599
• [ISSUE #9596] Optimize log in invokeSync when addr is null by @1782935682 in https://github.com/apache/rocketmq/pull/9597
• [ISSUE #9605] Fixed a potential resource leak by @yx9o in https://github.com/apache/rocketmq/pull/9606
• [ISSUE #9589] Optimize broker metrics initialization by @qianye1001 in https://github.com/apache/rocketmq/pull/9598
• [ISSUE #9611] Should exec callback in the Pop based on rocksdb impl by @lizhimins in https://github.com/apache/rocketmq/pull/9612
• [ISSUE #9600] Remove wrong logic in the callback for sending messages in rpc by @3424672656 in https://github.com/apache/rocketmq/pull/9608
• [ISSUE #9626] Prevent premature offset commit before consumer record flush by @lizhimins in https://github.com/apache/rocketmq/pull/9627
• [ISSUE #9542, #9609] Fix bazel CI and reduce dependencies by @xuhongxu96 in https://github.com/apache/rocketmq/pull/9610
• [ISSUE #9652] Enhance test for MQClientInstance by @GabrielDUX in https://github.com/apache/rocketmq/pull/9653
• [ISSUE #9648] Fix getOffsetInQueueByTime missing boundaryType in tieredMessageStore by @GabrielDUX in https://github.com/apache/rocketmq/pull/9649
• Fix start() and shutdown() of DefaultMessagingProcessor by @redlsz in https://github.com/apache/rocketmq/pull/7838
• [ISSUE #9663] Adopt Maven's revision mechanism (requires Maven >= 3.5.0) by @cvictory in https://github.com/apache/rocketmq/pull/9665
• [ISSUE #9650] Unified FAQ related URLs by @yx9o in https://github.com/apache/rocketmq/pull/9651
• [ISSUE #9620] Placeholder replacement string concatenation by @yx9o in https://github.com/apache/rocketmq/pull/9621
• [ISSUE #9581] Optimize the resource bloat of pollingMap and topicCidMap in LMQ scenarios by @RongtongJin in https://github.com/apache/rocketmq/pull/9579
• [ISSUE #9764] Fix bazel-compile (ubuntu-latest) error by @yx9o in https://github.com/apache/rocketmq/pull/9675
• [ISSUE #9666] Add accelerated startup recovery feature for RocksDB store with SYNC_FLUSH by @RongtongJin in https://github.com/apache/rocketmq/pull/9667
• [ISSUE #9634] Improve BrokerContainer extensibility and module structure by @RongtongJin in https://github.com/apache/rocketmq/pull/9635
• [ISSUE #9547] Fix sync group if attributes isn't null by @1782935682 in https://github.com/apache/rocketmq/pull/9679
• [ISSUE #9682] Increase initial delay for scanning inactive brokers to 5000 milliseconds by @mxsm in https://github.com/apache/rocketmq/pull/9683
• [ISSUE #9680] Improve RocksDB compaction filter factory resource management by @RongtongJin in https://github.com/apache/rocketmq/pull/9681
• [ISSUE #9677] Resolve metrics static variable conflicts in BrokerContainer mode by @RongtongJin in https://github.com/apache/rocketmq/pull/9678
• [ISSUE #9695] Not use pull offset when use pop orderly consume by @lizhimins in https://github.com/apache/rocketmq/pull/9696
• [ISSUE #9693] Add writeWithoutMmap configuration to prevent JVM crash when device becomes read-only by @guyinyou in https://github.com/apache/rocketmq/pull/9694
• [ISSUE #9701] Synchronize metrics shutdown to prevent JVM crashes during broker shutdown by @guyinyou in https://github.com/apache/rocketmq/pull/9702
• [ISSUE #9703] Remove pull offset and reset offset when deleting topic by @qianye1001 in https://github.com/apache/rocketmq/pull/9704
• [ISSUE #9707] Integrate RunningFlags with MappedFile system for better error handling and state management by @guyinyou in https://github.com/apache/rocketmq/pull/9708
• [ISSUE #9713] Improve data consistency in DefaultMappedFile flush method by @guyinyou in https://github.com/apache/rocketmq/pull/9714
• [ISSUE #9699] Optimize shutdown process and resource management by @RongtongJin in https://github.com/apache/rocketmq/pull/9700
• [ISSUE #9705] Improve persist() method reliability to prevent broker startup failure after power outages by @guyinyou in https://github.com/apache/rocketmq/pull/9706
• [ISSUE #9709] Add enableRunningFlagsInFlush configuration switch for CommitLog by @guyinyou in https://github.com/apache/rocketmq/pull/9710
• [ISSUE #9716] Refactor: replace RandomAccessFile with FileChannel by @RongtongJin in https://github.com/apache/rocketmq/pull/9715
• [ISSUE #9717] Fix RaftBrokerHeartBeatManager#scanNotActiveBroker was not actually executed by @redlsz in https://github.com/apache/rocketmq/pull/9718
• [ISSUE #9721] Fix TimerDequeueGetService thread not exiting after shutdown by @RongtongJin in https://github.com/apache/rocketmq/pull/9722
• [ISSUE #9689] Fix the issue that master transfer epoch was not updated in time by @redlsz in https://github.com/apache/rocketmq/pull/9690
• [ISSUE #9719] Fix Static Variable Conflicts in Multi-Broker Metrics Management by @RongtongJin in https://github.com/apache/rocketmq/pull/9720
• Fix/update metadata bug in v2 config by @cvictory in https://github.com/apache/rocketmq/pull/9724
• [ISSUE #9726] Abstract StoreMetricsManager interface to support non-DefaultMessageStore implementations by @RongtongJin in https://github.com/apache/rocketmq/pull/9727
• [ISSUE #9687] Cleanup BrokerHeartbeatManager code by @majialoong in https://github.com/apache/rocketmq/pull/9688
• [ISSUE #9732] Fix request may still be invoked when local process timeout by @qianye1001 in https://github.com/apache/rocketmq/pull/9733
• [ISSUE #9658] Replace JUnit 5 imports with JUnit 4 by @yx9o in https://github.com/apache/rocketmq/pull/9659
• [ISSUE #9743] Fix ReceiptHandleProcessorTest by @yx9o in https://github.com/apache/rocketmq/pull/9744
• Fix typos and linguistic errors in documentation / hacktoberfest by @survivant in https://github.com/apache/rocketmq/pull/9748
• fix get dledger mod storepath by @LittleBoy18 in https://github.com/apache/rocketmq/pull/9746
• [ISSUE #9741] Optimize authentication whitelist lookup efficiency by @yx9o in https://github.com/apache/rocketmq/pull/9742
• [ISSUE #9739] Revert skipped test cases in macos environment by @lizhimins in https://github.com/apache/rocketmq/pull/9749
• [ISSUE #9750] Fix NPE when isAsyncSendEnable is false by @RongtongJin in https://github.com/apache/rocketmq/pull/9751
• [ISSUE #9756] Fix the issue where the port is always 10911 when starting broker with brokerStartup by @RongtongJin in https://github.com/apache/rocketmq/pull/9755
• [ISSUE #9752] Fix time dequeue latency unit to milliseconds by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/9753
• [ISSUE #9761] Fix get authentication and authorization metadata provider error in AuthorizationMetadataManagerImpl by @majialoong in https://github.com/apache/rocketmq/pull/9762
• [ISSUE #9758] Fix resource filter does not take effect in the listAcl by @majialoong in https://github.com/apache/rocketmq/pull/9759
• [ISSUE #9765] Improve null handling in file operations by @yx9o in https://github.com/apache/rocketmq/pull/9766
• [ISSUE #9767] Fix incorrect decode method for UnlockBatchRequestBody in DefaultAuthorizationContextBuilder by @majialoong in https://github.com/apache/rocketmq/pull/9768
• [ISSUE #9769] Add tls.ciphers and tls.protocols in system properties by @carlvine500 in https://github.com/apache/rocketmq/pull/9770
• [ISSUE #9771] Remove static from transactionalMessageCheckListener executorService to support multi-broker in brokercontainer by @RongtongJin in https://github.com/apache/rocketmq/pull/9772
• [ISSUE #9763] Fix invalid user disable status check in authorization by @majialoong in https://github.com/apache/rocketmq/pull/9764
• [ISSUE #9773] Implement Shared RocksDB Instance for Broker Configs by @qianye1001 in https://github.com/apache/rocketmq/pull/9774
• [ISSUE #9776] Make SharedByteBuffer size configurable via MessageStoreConfig.maxMessageSize by @RongtongJin in https://github.com/apache/rocketmq/pull/9775
• [ISSUE #9773] Fix core dump when shutdown broker twice by @qianye1001 in https://github.com/apache/rocketmq/pull/9783
• Correct method name from updateDateVersion to updateDataVersion by @majialoong in https://github.com/apache/rocketmq/pull/9788
• [ISSUE #9784] Fix the policy comparator to prioritize DENY over ALLOW by @majialoong in https://github.com/apache/rocketmq/pull/9785
• [ISSUE #9798] Reduce unnecessary thread switching in sending message via proxy by @qianye1001 in https://github.com/apache/rocketmq/pull/9799
• [ISSUE #9811] Fixed the issue of excessive loop iterations in reput service by @lizhimins in https://github.com/apache/rocketmq/pull/9812
• [ISSUE #9816] Fix concurrent modify opentelemetry record in calculate consumer lag by @lizhimins in https://github.com/apache/rocketmq/pull/9817
• [ISSUE #9813] Fix buffer fetch thread pool starvation in tiered storage by @lizhimins in https://github.com/apache/rocketmq/pull/9818
• Fixed error message of CommandUtil.fetchMasterAddrByBrokerName by @TheRealHaui in https://github.com/apache/rocketmq/pull/9825
• [ISSUE #9802] Fix the issue of delete logic in tiered storage index service by @lizhimins in https://github.com/apache/rocketmq/pull/9803
• [ISSUE #9819] Upgrade OpenTelemetry to version 1.44.1 by @RongtongJin in https://github.com/apache/rocketmq/pull/9820
• [ISSUE #9735] Add TimerWheel snapshot mechanism for reliable recovery by @guyinyou in https://github.com/apache/rocketmq/pull/9736
• Add query message offset by timestamp to abstract class by @lizhimins in https://github.com/apache/rocketmq/pull/9836
• [ISSUE #9821] Fix getBrokerConfig no response issue for empty parameters by @yx9o in https://github.com/apache/rocketmq/pull/9822
• [ISSUE #9794] Change JDK prerequisite from 1.7+ to 1.8+ in documentation by @yx9o in https://github.com/apache/rocketmq/pull/9795
• fix controller mode scanControllerAddress by @LittleBoy18 in https://github.com/apache/rocketmq/pull/9845
• [#ISSUE 9841] Improve Resource Management in TimerWheel to Prevent File Handle Leaks by @guyinyou in https://github.com/apache/rocketmq/pull/9842
• [ISSUE #9632] Fix: Pop Long-polling Not Awakened for V1 Retry Messages by @qianye1001 in https://github.com/apache/rocketmq/pull/9828
• [ISSUE #9838] IndexStoreService use forceShutdown when disk is not writable by @RongtongJin in https://github.com/apache/rocketmq/pull/9839
• [ISSUE #9834] Support writeWithoutMmap in IndexStoreFile by @RongtongJin in https://github.com/apache/rocketmq/pull/9835
• Fix resource leak in IndexStoreFile shutdown by @RongtongJin in https://github.com/apache/rocketmq/pull/9840
• [ISSUE #9593] Optimize Deflater resource release by @yx9o in https://github.com/apache/rocketmq/pull/9594
• [ISSUE #9847] Reduce lock contention on the HandleData object to prevent threads from hanging by @qianye1001 in https://github.com/apache/rocketmq/pull/9848
• [ISSUE #9857] Optimize the performance of the notification by removing unnecessary retry checks by @qianye1001 in https://github.com/apache/rocketmq/pull/9858
• [ISSUE #9807] Optimize log.warn Time Consumption in the send Method by @yx9o in https://github.com/apache/rocketmq/pull/9808
• [ISSUE #9737] Fix client memory leak on connection failure by @qianye1001 in https://github.com/apache/rocketmq/pull/9740
• [ISSUE #9849] Correct ACL cache loader error log in ClusterMetadataService by @majialoong in https://github.com/apache/rocketmq/pull/9850
• [ISSUE #9741] Optimize authorization whitelist lookup efficiency by @majialoong in https://github.com/apache/rocketmq/pull/9760
• [ISSUE #9809] Fix NPE in getAcl when subject is null by @majialoong in https://github.com/apache/rocketmq/pull/9810
• [ISSUE #9789] LitePullConsumer supports manually adding subscription reported in Heartbeat by @qianye1001 in https://github.com/apache/rocketmq/pull/9790
• [ISSUE #9862] [Bug] Fix the issue of missing bornTime in POP requests when broker ACL is enabled by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/9863
• [ISSUE #9870] Ensure metadata provider cache executors are shutdown correctly by @majialoong in https://github.com/apache/rocketmq/pull/9871
• [ISSUE #9779] Fix the promblem of 'DirectBuffer will cause error at 9+ JDK version'. by @sinberCS in https://github.com/apache/rocketmq/pull/9801
• [ISSUE #9663] Add flatten-maven-plugin plugin by @lizhimins in https://github.com/apache/rocketmq/pull/9866
• [ISSUE #9816] Fix calculate consumer lag with opentelemetry by @lizhimins in https://github.com/apache/rocketmq/pull/9873
• [ISSUE #9875] Optimize the RocksDB config shutdown logic when useSingleRocksDBForAllConfigs is set to true to prevent JVM crashes. by @RongtongJin in https://github.com/apache/rocketmq/pull/9874
• [ISSUE #9734] Prepare to release Apache RocketMQ 5.3.4 by @lizhimins in https://github.com/apache/rocketmq/pull/9861

• @hepengju made their first contribution in https://github.com/apache/rocketmq/pull/9408
• @EnableAsync made their first contribution in https://github.com/apache/rocketmq/pull/9483
• @huanzhikang made their first contribution in https://github.com/apache/rocketmq/pull/9492
• @woaishixiaoxiao made their first contribution in https://github.com/apache/rocketmq/pull/9498
• @Allenhuangrui made their first contribution in https://github.com/apache/rocketmq/pull/9480
• @AROP-REPLICATION made their first contribution in https://github.com/apache/rocketmq/pull/9459
• @datagramsocket made their first contribution in https://github.com/apache/rocketmq/pull/9511
• @1782935682 made their first contribution in https://github.com/apache/rocketmq/pull/9540
• @co63oc made their first contribution in https://github.com/apache/rocketmq/pull/9560
• @xuhongxu96 made their first contribution in https://github.com/apache/rocketmq/pull/9610
• @GabrielDUX made their first contribution in https://github.com/apache/rocketmq/pull/9653
• @cvictory made their first contribution in https://github.com/apache/rocketmq/pull/9665
• @majialoong made their first contribution in https://github.com/apache/rocketmq/pull/9688
• @survivant made their first contribution in https://github.com/apache/rocketmq/pull/9748
• @TheRealHaui made their first contribution in https://github.com/apache/rocketmq/pull/9825
• @sinberCS made their first contribution in https://github.com/apache/rocketmq/pull/9801

Full Changelog: https://github.com/apache/rocketmq/compare/rocketmq-all-5.3.3...rocketmq-all-5.3.4

This release focuses on enhancing the MCP and A2A (Agent-to-Agent) capabilities of Nacos, improving system stability, and refining security and configuration management. Key additions include support for batch registration of agent endpoints, merging versions during MCP server import, and enhanced flexibility in updating MCP servers with an overrideExisting option. The release also introduces default security configuration support for both inbound and outbound communication, laying the foundation for secure AI agent interactions.

Detail changes in this version are listed below:

[#13867] Support override existing configuration when updating MCP server. [#13922] Add thread pool configuration to allow core thread timeout. [#13946] Refactor AgentEndpoint model and add new fields to support advanced protocol and queries. [#13953] Add support for default security configurations, including selection and setup for inbound and outbound security schemes. [#13957] Support batch registration of agent endpoints. [#13951] Add configurable CORS filter for console module.

[#13822] Separate responsibilities between client executor and login scheduled executor to prevent thread pool exhaustion. [#13863] Optimize shutdown.sh script to combine multiple PIDs into a single line for cleaner process termination. [#13911] Enhance importToolsFromMcp API to support streamable protocol servers. [#13914] Upgrade checkStyle plugin version to improve code linting and maintainability. [#13945] Support merging versions when importing MCP server specifications. [#13963] Refactor List config apis to support return configTags and desc.

[#13140] Fix consistent namespace handling for ConfigChangeNotifyRequest when client namespace is empty. [#13860] Fix potential null pointer exception risk in ConfigMigrateService. [#13866] Fix inability to edit imported MCP servers in the console. [#13869] Fix error creating bean 'consoleHealthController' due to missing dependency injection. [#13877] Fix namespaceId issue in McpRemoteHandler causing incorrect context propagation. [#13882] Fix synchronization lock and add volatile keyword for TpsControlManager to ensure thread safety. [#13884] Fix JSON parsing error in MCP server validation API during import. [#13902] Fix deprecated Log4j2 package scanning warnings by updating logger initialization logic. [#13948] Fix failure when updating agent card through the console interface. [#13959] Fix Password update fails when deployed with independent console/server. [#13964] Fix when nacos.extension.ai.enabled is false, nacos start failed.

[#13879] Add unit tests for AI-related components. [#13881] Add utility APIs for development and testing purposes.

Module	Java Required
Nacos-Server/Nacos-Console	Java 17
Nacos-Client	Java 8
Nacos-Maintainer-Client	Java 8

• @bee0511 made their first contribution in https://github.com/alibaba/nacos/pull/13858
• @Alioth4J made their first contribution in https://github.com/alibaba/nacos/pull/13864
• @five111 made their first contribution in https://github.com/alibaba/nacos/pull/13861
• @herodotus-ecosystem made their first contribution in https://github.com/alibaba/nacos/pull/13872
• @wanna280 made their first contribution in https://github.com/alibaba/nacos/pull/13882
• @zhan7236 made their first contribution in https://github.com/alibaba/nacos/pull/13966
• @aihai made their first contribution in https://github.com/alibaba/nacos/pull/13965

Full Changelog: https://github.com/alibaba/nacos/compare/3.1.0-bugfix...3.1.1