Слияние кода завершено, страница обновится автоматически
If you are using a CubeFS version earlier than v3.4.0, please refer to the UPGRADE NOTICE in version v3.4.0 for detailed upgrade steps and upgrade to v3.4.0 first.
If you deployed the cluster based on an older version, please refer to the documentation upgrade 3.5.0 for upgrade steps to version 3.5.+
all: Supports the capability to manage different storage media. （#3603，@bboyCH4, @true1064, @Victor1319 ）master/lcnode: Supports automatic migration of cold data via lifecycle management . (#3604 @bboyCH4, @honeyvinnie, @Victor1319)master/client: Support querying all client versions and IP information. (#3606 , @Victor1319)datanode: Support for direct I/O read operations with volume. (#3630, @Victor1319)bcache: bcache supports configuration switches that work for non-SSD types. (#3607, @longerfly)sdk: Support the ability of the tool to count the size of the directory by access time. (#3608, @longerfly)sdk: Use distributed locks to prevent concurrent deletions from multiple clients in the recycle bin. (#3610, @Victor1319)sdk/datanode: Optimize data read path performance for storage and compute separation scenarios. (#3631, @Victor1319)sdk/metanode: Support metadata reads in a leaderless environment using quorum mode and meta follower mode. (#3632, @Victor1319)raft: Conflicts between Raft metadata and WAL logs (#3605, @Victor1319）sdk: Deleting files can cause client panic when the recycle bin is enabled (#3609, @bboyCH4)all: Fix bugs related to historical version faults and anomalies. ( @Victor1319）UPGRAGDE NOTICE If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. if you CubeFS version is v3.3.2 or before, and need to upgrade to v3.3.*,you must follow these upgrade steps:
If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.
Note: Provides basic capabilities of the snapshot version in v3.4.0(beta version, disabled by default)
master: Auto decommission for bad disk. （#3494，@bboyCH4）master: Meta for automatic repair of data replicas. (#3495, @bboyCH4)meta: Synchronize accessTime between replicas for meta. (#3496, @bboyCH4)cli: Cli support configuration and querying for decommission operations. (#3497, @NaturalSelect)master/datanode/cli: Datanode bad disk report to master adds IoErrPartititionCnt and TotalPartititionCnt. (#2679, @true1064)master,datanode:Retain the data of replicas that have been deleted by RaftForce for a period of time. (#3499, @bboyCH4)master: Atomicity of data partition decommission operation. (#3500, @bboyCH4)master: Cli display the repair progress of data replica. (#3502, @bboyCH4)master: Querying the decommission progress of datanode is too slow. (#3504, @bboyCH4)master: Simplify the return results of the datanode/disk query decomission status interface. (#3507, @bboyCH4)master: Add a new interface to query the status of decommission tokens. (#3509, @bboyCH4)master: The decommission progress can display data partitions have not undergone the decommission operation. (#3512, @bboyCH4)master/datanode: Audit log for dp decommission (#3513, @NaturalSelect)master/data/meta/client: Add version information to the metrics reported by each subsystem of CubeFS. (#3516, @NaturalSelect)data: The ReloadSnapshot process of the data partition is taking too long. (#3517, @Victor1319)meta: Under heavy pressure from deleting a large number of extents, the deletion efficiency is too low. (#3518, @NaturalSelect)master: Simplify the return results of the datanode/disk query decomission status interface. (#3507, @bboyCH4)client: when volume is deleted, cfs-client need to exit. (#3512, @longerfly)master: Add config to control http pool size. (#3567, @Victor1319)metanode: Persist access time for inode in meta node . (#3565, @bboyCH4)client: Support libsdk to determine whether it is a file or directory based on mode.(#3566 , @longerfly)master: Decommission both replicas of the dp results in both replicas failing to decommission.. (#3498, @bboyCH4）master/data:After the cluster performs decommission operations, many data partitions are backed up to old directories with dates on the datanode. (#3501, @bboyCH4)master/data:The replica repair process is continuously retried . (#3503, @bboyCH4)data: Decommission operations often fail due to nodes being inactive. (#3506, @bboyCH4)master: Using RaftForce to decommission two replicas is stuck. (#3510, @bboyCH4)master: After executing the cancellation of the decommission operation, the disk remains in a disabled state. (#3511, @bboyCH4)master/data:The interface for recovering bad disk is timing out. (#3514, @bboyCH4)data: Encountering an I/O error while writing WAL logs can lead to a panic in the data service.(#3515, @bboyCH4)master: Only choose from the specific zone in function canWriteForNode. (#3519, @true1064)client: client update extents from meta and drop extents in cache leadto ek conflict. (#3520, @longerfly)client: if volume name not match regexp when mounted, return failed immediately. (#3522, @longerfly)master: The data node decommission has no response. (#3528, @bboyCH4)UPGRAGDE NOTICE If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*, you must follow these upgrade steps:
If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.
objectnode: ObjectNode support post object. （#2695，@yhjiango）objectnode: S3 sts and signature auth. (#2488, @yhjiango)libsdk/meta: Support dir lock. (#3344, @longerfly)master/data/meta: Support write disable option for volume. (#2537, @NaturalSelect)master: Support freeze volume and delayed deletion of volume. (#2922, @shuqiang-zheng)master: Support compressing client/partitions api response data to save bandwidth。 (#3314, @lily-lee)objectnode: Concurrent multipart upload security mechanism (#2928, @tangdeyi)master: enable rename atomic operation by default (#2956, @Victor1319)master: Optimize the display of capacity when "used" is greater than "total". (#3315, @true1064)libsdk: Support more api function for libsdk. (#3316, @longerfly)data: Little io error only set current datapartition as broken. (#3317, @true1064)client: Write to tiny extent when file size less than 1M. (#3318, @leonrayang)master: Optimize CPU cost when invoking the volStat api (#3319, @true1064)master: Optimize the speed of applying raft snapshot for master module. (#3320, @NaturalSelect)meta: Support rotating and deleting inode files to avoid excessive size. (#3321, @NaturalSelect)data: Speed up the startup speed of the datanode. (#3322, @NaturalSelect)master: Optimize the volume deletion process. (#3323, @Victor1319)master: Add a rdonly flag to the client/partitions api to indicate whether a volume is read-only. (#3324, @true1064)data: Support asynchronous deletion of expired datapartitions. (#3325, @Victor1319)data: Prioritize listening to external service port when starting the datanode. (#3326, @Victor1319)data: Synchronously persist dp meta information when creating dp.(#3335, @bboyCH4)client: Support retrying requests for failed extent writes when the client encounters limit io error. ( #3338, @NaturalSelect)objectnode: PartNumber of multipart must be greater than 0. (#2885, @yhjiango）objectnode: Listobjectv1 with delimiter may lead to panic. (#2931, @tangdeyi)client: When the recycle bin is enabled, avoid deleting non-empty directories. (#3328, @bboyCH4)libsdk: Fix some issues encountered during the usage of the libsdk. (#3329, @longerfly)data: Optimize the blocking issue during data partition migration in snapshot synchronization. (#3330, @bboyCH4)meta: Fix the issue of possible duplication caused by the unpersisted meta unique id. (#3331, @true1064)master: Master follower replicas only request the leader replica to obtain the partition list.(#3332, @NaturalSelect)client: Fix the issue that interruption causes failure to write data to ec.(#3333, @Victor1319)data: Fix the issue of memory leaks caused by blocking during data transmission between datanode leader and follower. (#3336, @Victor1319)data: During the process of reusing network connections, there is a possibility of using incorrect network response messages. (#3337, @Victor1319)UPGRAGDE NOTICE If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. if you CubeFS version is v3.2.1 or before, and need to upgrade to v3.3.*, you must follow these upgrade steps:
If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.
client: Add a trash feature to client (#2291, @bboyCH4)datanode: Support cleaning up garbage data on the datanode (#2880, @Victor1319)metanode: Audit log in metanode (#2899, @NaturalSelect)datanode: limit datanode's disk flow and concurrent iops (#2900, @sejust)client: Too many waring errors log when use trash (#2915, @bboyCH4)client: Audit's writer maybe nil leadto error (#2901, @longerfly)gapi: fix read body entirely into memory (#2691, @tangdeyi)objectnode: Timing attack can leak user passwords and CubeFS leaks users key in logs from cncf security audit (#2781, @leonrayang)client: Optimize insecure random number generation in function util/string.go:RandomString (#2698, @true1064)docs: Add Security Best practice (#2853, @leonrayang)gapi: Timing attack can leak user passwords and CubeFS leaks users key in logs from cncf security audit (#2781, @leonrayang)objectnode: fix limiter lib deadlock (#2794, @tangdeyi)master\objectnode: some commits related with security audit (#2824, @leonrayang)master: qos.Lock of assignClientsNewQos forget release and trigger deadlock(#2861, @leonrayang)bcache: fix possible deadlock of bacahe(#2819, @longerfly)ClusterMgr: fix volume manager deadlock(#2793, @tangdeyi)blobnode: fix put shard deadlock (#2790, @mawei029)client: inode leak when use trash(#2911, @bboyCH4)client: trash encounters IO error when dealing with log file names(#2912, @bboyCH4)client: trash delete interval do not work(#2914, @bboyCH4)client: Failed to delete a large directory with trash enable(#2917, @bboyCH4)UPGRAGDE NOTICE If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.
meta\client: add quota of children dentries for directory (#1763, @wuchunhuan )master: add qps limiter for individual API on master (#1766, @wuchunhuan )client: add audit log for fuse client (#1764, @wuchunhuan )objectnode\sdk: objectnode supports docking erasure-code subsytem(blobstore) (#1765, @wuchunhuan)master\datanode: Add speed control of dp decommission for disk or datanode (#1778, @bboyCH4)master: master support leader switching by external trigger (#1772, @leonrayang )metanode\master: metanode support follower read (#1775, @leonrayang)master: Add metrics for meta-partition consistency detection (#1776, @leonrayang)master: add API statistics log for master (#1767, @wuchunhuan)datanode: Datanode startup should not be influnced by missing data partitions (#1773, @leonrayang)master: volume's capacity must be bigger than 120% used size (#1723, @Victor1319)metanode: metanode support config memRatio of the machine. (#1725, @Victor1319)datanode: datanode support config diskPathPrefix to manage disk path list. (#1727, @Victor1319)metanode: reduce extent alloc memory when metanode snapshot (#1728, @Victor1319)metanode: random write may cause metanode memory grow (#1729, @Victor1319)datanode: bad disk space is calculated in datanode's total space (#1601, @Victor1319)datanode: no permission disk can't be detected (#1602, @Victor1319)master\cli: add display of unavailable replica for cfs-cli datapartition check (#1771 ,@true1064)datanode: datanode should compute datapartition used size right after loading extents from disk (#1782 ,@true1064)client:enhance support nfs access operation (#1798, @leonrayang )master: add some metircs (#1785, @liubingxing @litao)master\datanode\client\cli: fix the typos and format code (#1787, @liubingxing @litao)datanode: add lack datapartitions monitor (#1789, @guojunhao)master: create data partition exclude decommissioned disk (#1790 , @guojunhao)datanode: Optimize the qos for random write (#1791, @litao)metanode: speed up metanode startup (#1792, @liubingxing)cli: add more info to cli/fsck cmd (#1793, @liubingxing @litao)master: master snapshot recover not reset local rocksdb info (#1522, @wuchunhuan )master:Memory cost too fast during restart in case of data partition‘s count is magnity (#1774 , @leonrayang)client: Readonly dp can still accept write request from client (#1779, @bboyCH4)metanode: Metanode should not establish connection to blobstore for cold volume (#1781, @bboyCH4)client: blockcache service may be oom when the client caches many large files concurrently (#1783, @zhangtianjiong)datanode: too may delete requests may cause a lot of tcp connections. (#1724, @Victor1319)master: when master creating data partition, should use vol.dataPartitionSize as datapartition.total (#1777 ,@true1064)master: In addition to all replica status is readwrtie, still need to consider volume used space before check set datapartition status as readwrite (#1780 ,@true1064)master: master auto creates dataparitions when volume goes from full to not full (#1784 ,@true1064)unit test: Fix the pipeline cubefs-ci error problem (#1795, @baijiaruo)metanode\client:do rename on the file which is soft link not worked well (#1797 , @leonrayang)meta\master: fix some bug to improve cluster stability (#1786, @liubingxing @litao)If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
If your Blobstore version is v1.1.0 or before which built with cubefs-blobstore (https://github.com/cubefs/cubefs-blobstore) , please refer to UPGRADE to v3.2.0 following these steps #1556.
blobstore: Merge some service modules #1563blobstore: ClusterMgr support simple key-value persistent storage #1566blobstore: Remove blobstore's dependency on monogdb #1493blobstore: Remove blobstore's dependency on consul #1507blobstore: Support audit log filtering #1506blobstore: Blobnode use the same erasure encoder and buff-pool libraries as Access #1498blobstore: Separate flow control for repair and migration traffic #1508blobstore: Retry http body under rpc request #1567If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
master: add leaderSize param in create partition req #1456master: support decommission broken replica for 2-replica no leader dp #1456datanode: in case disk full, repair size is also thought as used size #1456raft: raftstore.monitor can be optimized #1543If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
metanode: Show metapartition id correctly #1554metanode: remove redundant memcopy when reading raft snapshot #1552master: update metanode or datanode id need drop old one before #1555metanode: add metric mpDentryCount #1553datanode: fallocate return interrupt err try again #1551If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
master\client\datanode: ：Provide QoS services to enhance multi-tenant isolation #1447client: Caching acceleration #1446master\datanode: Support two replicas data storage #1389master: Add vol label in cfs_metanode_mpInodeCount #1482client: Support config posixAcl for volmue #1485master: Add datapartition count limit for datanode. #1481datanode: TinyDeleteRecord file will be very big when one replica is down #1433client: EnablePosixACL is not vaild in centos system #1393blockcache:Blockcache memory overflow and be killed #1422yum source: Yum install script fails due to missing packages #1410unit test: S3test random path busy to produce data and timeout #1515master: Zone name check not strict enough while crossZone enabled #1479datanode\master: Avoid writing operation failed because of disk full #1519master: Enable volume update from 3 to 2 replics #1516UPGRAGDE NOTICE If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
master: support to delete data replica with force #1258master: revert the deletion process in the tiny extent to avoid deletion failure caused by different logical and physical sizes #1439master: add metapartition replica check to fix replicas status still writeable while metanode unavalible #1440If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
This is a significant release based on master branch with a lot of code mainly to support multi layer data layer including blobstore (Erasure-Code) storage, blobstore supports erasure coding storage method(https://github.com/cubeFS/cubefs-blobstore) to support low frequency volume and reduce cost .
Before this we released v3.0.0-beta based on release v2.4.0 (https://github.com/cubeFS/cubefs/releases/tag/v3.0.0-beta) , which make volume can be create as standard(3 replica) or low frequency access type(Erasure-Code), but not support multi data layer.
Release formal version release-v3.0.0 which based on the newest master commit, below is a list of the main features.
Please refer to the documentation for details,the content already updated.
log: support write ebs log to file and update ebs log level #212client\datanode\master: support one replica of data storage #1227master: support get master's raft status info #1227master: support preload data from blobstore to cubefs #212client: client local read cache #212client: Time-consuming statistics of client sdk interfaceclient: client support blobstore #212client: support ebs index #212master: support different type dataparitition(normal dp,cache dp,preload dp) #212master: support mulit volume type(replcias volume,ec volume) #212project: enable go mod and update vendor #1343datanode: intruduce metrics degrade level to datanode #1438master: not do ttl & updateSize for hot vol #212compile: change go mod version from 1.14 to 1.16 and consistent with compile #1343compile: enable go mod and update vendor #1343project: change from chubaofs to cubefs #1343master: single replica create volume should set followeread with true #1227raft: raft snapshot still running while peer remove happened which lead to the snapshot gorutine cann't end. #1227master: Refactor finding partitins to be offlined #1189master: Avoid race syncing node's info when setting rdonly #1189client: follow recover handlers when closing open handler #1316master: remove 5 GB left limit for disk to create datapartition #1189server: if server recive kill-signal before being running-state, server will can't be killed graceful #1261master: not do ttl & updateSize for hot vol #212master: support disk, datanode, metanode decommissionclient: add cli-command for cold volume #212master: ec-volume can't be deleted if size not equal 0 #212client: add cfs-cli request parameters #212client: add scan cli #212master: add volume type to newVolCreateCmd #212master: not support force delete cold volume #212datanode: add cfg diskRdonlySpace for datanode #1353datanode: use os.ReadDir instead of ioutil.ReadDir #1438datanode: improve performance of write. data allocated didn't put back to buffer pool #1438datanode: reduce datanode cpu usage #1438datanode: remove unnecessary json unmarshal when getting local extents #1438datanode: reduce datanode cpu usage by eliminate file seeks #1438client: mitigate the pain of extents fragmentation #1318If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
datanode feat: intruduce metrics degrade level to datanode [#1234]datanode remove 5 GB left limit for disk to create datapartition[#1345]fuse client follow recover handlers when closing open handler[#1344]datanode improve performance of write. data allocated didn't put back to buffer pool [#1346]datanode remove unnecessary json unmarshal when getting local extents [#1346]If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
fuse client:support auto push data to push gatewaymaster:reduce flow from interface of "client/partiton" that follower support it and will not redirect to leaderraft:the receive buffer channel size of raft support to be configurablemetanode:implement the content summarymaster:domain for cross zonefuse client:fix: currAddr of stream conn is emptyfuse client:fix: update extent cache no matter whether eh is dirty or notmetanode:when kill metanode(mean while mp stoped firstly) and if apply snapshot happen at the same time, snapshot will be block, causeing metanode can't be killedmetanode:meta node migration be recorded in badmetapartitions concurrently without lock which lead to mp id missmetanode:return file nodes for statfsmetanode:meta not use warn when consulMeta not setmetanode:makes AppendExtentKeyWithCheck request idempotentfuse client:push addr shadowed if export port is not setobjectnode:readdir gets insufficient dentries if prefix larger than markerraft:remove redundant memcopy when reading raft snapshotraftstart tcp listen before starting raft ltp test:broken test case ftest01，update and unlock ltp test cases ltp test:fix ltptest ci bugdocker:finstall killall command in the docker imagecli:fix cli tool typomaster: support disk, datanode, metanode decommission and assign target nodemaster:return response cache directly, not copy againmaster:add vol usedRatio warn log when usage > 90%master:add master metrics vol_meta_count to export vol dp/mp/inode/dentrymaster:del replica with force if decommission hangmetanode:add log for loading meta partitionsfuse client:increase client retry times to avoid mp raft election timeout.grafna: change grafana volume used size rate from rate to derivgrafna:support push monitor data to gatewayobjectnode: add subdir authorized check for objectnodeobjectnode: tracking objectnode modification for cfs-serverraft:check nil when get leader term inforaft: check size when read data from heartbeat portraft: add getRaftStatus for metanodecli:update gitlab-cicli:Enhancement: add ci check for ltptest resultcli:enhance: upload docker_data when ci tests finishcli:Create and Update ci.ymlstyle:rename cfs to cbfs & optimize the libsdk moduledocker:change: use ghcr instead of dockerhubbuild: add version information and rules for building fsckbuild:change: update build status badge1. Purpose In the cross zone scenario, the reliability need to be improved. Compared with the 2.5 version before, the number of copysets in probability can be reduced. The key point is to use fault domains to group nodesets between multiple zones.
Reliable papers https://www.usenix.org/conference/atc13/technical-sessions/presentation/cidon
Chinese can refer to https://zhuanlan.zhihu.com/p/28417779
2. configuration
1) Master
Config file：master.json Enable faultDomain set item "faultDomain": true
Zone count to build domain faultDomainGrpBatchCnt，default count:3，can also set 2 or 1
If zone is unavaliable caused by network partition interruption，create nodeset group according to usable zone Set “faultDomainBuildAsPossible” true, default is false
The distribution of nodesets under the number of different faultDomainGrpBatchCnt 3 zone（1 nodeset per zone） 2 zone（2 nodesets zone，1 nodeset zone，Take the size of the space as the weight, and build 2 nodeset with the larger space remaining） 1 zone（3 nodeset in 1 zone）
Ratio 1) The use space threshold of the non-fault domain（origin zone） After the upgrade, the zone used by the previous volume can be expanded, or operated and maintained in the previous way, or the space of the fault domain can be used, but the original space usage ratio needs to reach a threshold, that is, the current configuration item The proportion of the overall space used by meta or data Default：0.90 UpdateInterface： AdminUpdateZoneExcludeRatio = "/admin/updateZoneExcludeRatio"
2) the use space threshold of the nodeset group in the domain Nodeset group will not be used in dp or mp allocation default：0.75 Update interface： AdminUpdateDataUseRatio = "/admin/updateDomainDataRatio"
2) Datanode && metanode
After the fault domain is enabled, a minimum configuration of the fault domain is constructed under the default configuration: Each zone contains 1 datanode and 1 metanode, and the zone name needs to be specified in the configuration file There are 3 datanodes and 3 metanodes in 3 zones
For example, three datanodes (metanode) are configured separately: "zoneName": "z1", "zoneName": "z2", "zoneName": "z3", Start after configuration, the master will build a nodeset for z1, z2, and z3, and component a nodesetgrp
3. Note 1) After the fault domain is enabled, all devices in the new zone will join the fault domain 2) The created volume will preferentially select the resources of the original zone 3) Need add configuration items to use domain resources when creating a new volume according to the table below. By default, the original zone resources are used first if it’s avaliable
| Cluster:faultDomain | Vol:crossZone | Vol:defaultPriority | Rules for volume to use domain | h| ------ | ------ | ------ |------ | | N | N/A | N/A | Do not support domain | | Y | N | N/A | Write origin resources first before fault domain until origin reach threshold | | Y | Y | N | Write fault domain only | | Y | Y | Y | Write origin resources first before fault domain until origin reach threshold |
Note: the fault domain is designed for cross zone by default. The fault domain of a single zone is considered as a special case of cross zone, and the options are consistent
example : curl "http://10.177.200.119:17010/admin/createVol?name=vol_cross5&capacity=1000&owner=cfs&crossZone=true&defaultPriority=true"|jq .
1. Purpose In order to query the content summary information of a directory efficiently, e.g. total file size, total files and total directories, v2.5 stores such information as the parent directory’s xattr.
The parent directory stores the files, directories and total file size of the current directory. Then only need to make recursive of the sub directories, and accumulate the information stored by the directories to query the content summary information of a directory.
2. Configuration Client config file: fuse.json 1) Enable XAttr ”enableXattr”:”true” 2) Enable Summary ”enableSummary”:”true” Both of xattr and summay have to be set if you want to mount a volume to the local disk. Set summary is enough if you want to access the volume via libsdk.so.
3. How to use There are two different ways to get the content summary of a directory. 1) Fuse mount getfattr -n DirStat yourDirPath getfattr can be installed by: yum install attr or apt install attr 2) libsdk.so cfs_getsummary (libsdk/libsdk.go)
4. Note 1)The incremental files’ summary information will be held by their parent directories. But the old files will not. Use cfs_refreshsummary (libsdk/libsdk.go) interface to rebuild the content summary information. 2)The files, directories and total file size are updated asynchronously in the background. Users are not aware of these operations, but it does increase the requests to meta servers (usually doubled). You are recommended to evaluate the impact to your cluster before using this feature.
If your CubeFS version is v2.3.x or before, please refer to the UPGRADE NOTICE in v2.4.0 for upgrading steps. And also please make sure that your fuse client or objectnode version is equal to or older than the servers, i.e. master, metanode and datanode. In another word, newer versioned client can not be used in a cluster with older versioned servers.
meta&object introduce ReadDirLimit interface to retrieve partial results #1234
fuse client use ReadDirLimit in fuse client #1244
sdk makes AppendExtentKeyWithCheck request idempotent #1224
meta start tcp listen before starting raft 1256
raft remove redundant memcopy when reading raft snapshot to avoid snapshot hanging 1264
object handling range read request in a behavior compatible with S3 #1286 #1298
sdk add version in the http requests issued to master to collect client info 1262
sdk mitigate the pain of extents fragmentation 1282
If your CubeFS version is v2.3.x or before, and need to upgrade to v2.4.x , please follow the following upgrade steps:
first of all, firewall open two more port (17710 & 17810) in your machine to support tcp multiplexing；
17710 is calculated by 17210, the port listened by metanode, add 500
17810 is calculated by 17310, the port listened by datanode, add 500
then add item '"enableSmuxConnPool"=false' in your datanode's config file, upgrade all datanodes; detail steps as follow:
a. using new dataNode bin file replace the old version
b. add '"enableSmuxConnPool"=false' in your datanode's config file
c. restart datanode process
d. update all the datanodes in the custer
update '"enableSmuxConnPool"=true' in your datanode's config file, restart all datanode again;
notes: before set enableSmuxConnPool as true, you must finsh step 2.
upgrade all metanode bin file and restart meta process;
upgrade all master bin file and restart master process;
upgrade all client bin file and restart client process;
The key point of upgrading from previous versions to v2.4.0 is to make sure ALL datanodes are upgraded to the new version with "enableSmuxConnPool" set to "false" before upgrading other components, such as metanode, master and client. Then datanode can be configured to enable smux conn pool, i.e. "enableSmuxConnPool" set to "true".
dataNode smux support #1124
metaNode smux support #1124
datanode metanode support TickInterval configurable #1117
master remove token-based volume read-write permission control #1119
datanode fix build repair task panic #1124
metanode too many open/close stream when delete limit #1124
master do split on metanode partion if inode count larger than maxinodecnt #1124
datanode dp.raftPartition nil pointer #1124
datanode fix datanode repair log print error #1124
master partition be split can be write if not full and less then on step #1124
metanode new a gauge instance every time to avoid a gauge instance modified concurrently #1124
master fix conflict for histogram data when report data concurrently #1124
datanode fix bug when packet opCode=OpExtentRepairRead and network is unstable #1124
client export init should be after init log in fuse client #1124
util flush log before os.exit #1124
datanode format datanode log when delete #1124
metanode update log level to info if mp is not exist #1124
Note that when upgrading from versions prior to v2.3.0 to version v2.3.0 or beyond, servers(i.e. resource manager, metanode and datanode) must be upgraded before client(i.e. fuse client and objectnode) due to #1098.
client: support multiple subdir permissions for an individual user. #1051
sdk: introducing libsdk so applications can embed the use of cubefs to a single binary and no additional process is required at runtime. #1082
datanode: improve the shut down process of datanode so the raft apply id is persisted. #1030
raft: auto fix crc-mismatch raft log crc. #1039
objectnode: fix key encoding issue of S3 ListObjects API. #953
datanode: fix data partition decommission timeout. #972
metanode: change mtime of the directory when creating or deleting dentry. #1000
datanode: fix statfs deviation when using EXT4 as the underlying local filesystem for datanode. #1031
client: func sortHostsByDistance change dp's hosts. #1106
datanode: Introducing data partition selector which allow users to customize the client's selection logic for data partition when reading and writing, and can switch at any time. #853
client: Add enablePosixACL configuration to enable POSIX ACL feature. #906
datanode: Improved data recovery speed. #899
datanode: Fix the issue that the information in the memory is not released immediately after deleting the extent, which causes the status information of the data partition to be incorrect. #938
datanode: Fix the issue that repeatedly sending requests to the failed data partitions. #937
objectnode: Fix the issue that recursive making directory concurrent with sample multi-layer path prefix. #904
master: Fix the concurrency safe issue when removing raft member. #893
sdk: Fix the panic issue while remove data partition concurrently. #894
object: Implemented S3 api 'DeleteBucketPolicy' for bucket deletion. #757
master: Introducing management API for volume capacity expanding and shrinking. New capacity must be set more than 20% lager than used. #764
master: Introducing management API for updating node address. #813
master: Introducing management API for checking nodes. #813
metanode: Add header and checksum verification for EXTENT_DEL files. #813
cli: Add logging for CLI tool, and added several commands to call master API. #764 #801
compile:Support direct compile or docker cross compile on Arm64 platform. #779
client: Introducing nearRead option that allow client read data priority from the nearest DataNode to improve reading performance. #810
client: Update volume follower read config from master periodicity. #837
object: Improved compatibility for ListObjects and ListObjectsV2 interfaces. #769
master: Check whether the used space between the replicas is consistent when performing the load partition operation. #813
metanode datanode: Optimize batch delete Extent; add autoRepairLimitRater on DataNode. #781
datanode: Introducing autoRepair option for limit data repair speed on DataNode. #842
datanode: The data repair task will skip process extent file which has been already deleted. #842
master: When loading metadata of cluster, using the current ID instead of the loaded ID. #821
metanode: Fix the deadlock problem while MetaNode deleting dentry. #785
metanode: Fixes #760 Add inode to freeList when NLink is 0, and delete inode 7 days later. #767
datanode metanode: When DataNode and MetaNode start, the partitions not exist in cluster view will be renamed to expiredPartition and skip loading. #824
datanode metanode: Fixes issue#698, Raft instance delete itself by applying ConfChange raft log. #866
metanode: MetaNode may not free space. #838
client: Fix the problem that client opening file with outdated extents information. #783
client: Batch inode get mechanism is out of service. #847
client: Fix the problem that makes authorized user mount volume failure. #828
datanode: Fix several issue in automatic repair process for tiny extent. #855 #857
datanode: Fix the problem that the client still applies to create a new extent to a data partition that has no space. #867
console: CubeFS Console is a web application, which provides management for volume, file, s3, user, node, monitoring, and alarm.
The CubeFS Console makes it much easier to access services provided by CubeFS. #728metanode: Provide compatibility verification tool for meta partition. #684
object: CORS access control. #507
fuse: Introduce fsyncOnClose mount option. Choose if closing system call shall trigger fsync. #494
Release2.1.0 did a lot of work to optimize memory usage.
master: Add set/get deleteBatchCount interfaces. #608
master: Delete partitions on DataNode/MetaNode concurrently in decommission action. #724
master: Added api for getting & setting parameters for batch deleting extents. #726
metanode: Filter inode candidates that will be sent to a partition in batch-inode-get to save memory. #481
metanode: Batch delete inode, unlink&evict dentry. #586
datanode: Prioritize healthy data node when sending message to data partition. #562
metanode datanode: Check expired partition before loading. #624
object: Implemented several conditional parameters in GetObject action. #471
object: Making S3 credential compatible with earlier version of CubeFS. #508
object: Totally support presigned url; Partial support user-defined metadata; Making ETag versioned. #540
object: CopyObject across volumes; Support coping directory; Support coping xattr. #563
object: Parallel downloading. #548
object: Add 'Expires' and 'Cache-Control' in putObject/copyObject/getObject/headObject. #589
object: Modify part system metadata. #636
fuse: Decrease request channel size to save memory. #512
fuse: Let the client daemon inherit all the environment variables but not just PATH. #529
fuse: Introduces a debug interface /debug/freeosmemory to trigger garbage collection manually. #539
fuse: Add configuration MaxCPUs. #546
fuse: View client log through http proto. #552
fuse: Support modifying ModifyTime of inode by setAttr. #733
deployment: Add log map in docker-compose.yaml. #478
deployment: Introduce nginx to improve compatibility. #534
test: Add testing script for S3 APIs in python. #514 #595
monitor: Add detailed vol info in grafana dashboard. #522
cli: Some new features for CLI tool: bash completions, configuration setting, decommission, diagnose etc. #555 #695
metanode: Accelerate deletion. #582 #600
datanode: Limit replica number of data partition to at least 3. #587
datanode: Set if auto-repair by http interface. #672
metanode datanode fuse: Optimizations on port checking. #543 #531
metanode datanode: Validate creating partition request from master. #611
object: Refactor copy object function. #563
fuse: Replace process of token validation from meta wrapper to client. #498
fuse: Adjust rlimit config for client. #521
metanode: When overwriting a file, if the inode in a dentry is updated successfully, ignore unlink and evict inode errors. #500
metanode: Fix incorrect metrics collection for volume usedGB in monitor system. #503
metanode: VolStat can correctly update when force update metaPartitions frequently. #537
metanode: The MaxInodeID of meta partition is not synchronized when recovered from snapshot. #571
metanode: Free Inodes by raft protocol. #582
metanode: Painc with deleteMarkedInodes. #597
object: Fix the parsing issue of the two preconditions of If-Match and If-None-Match. #516
object: Change the time format in list buckets API to UTC time. #525
object: Fix xml element of DeleteResult. #532
object: Empty result in list operation when run in parallel with delete. #509
object: Change from hard link to soft link in CopyObject action. #563
object: Solved parallel-safety issue; Clean up useless data on failure in upload part. #553
object: Fixed a problem in listing multipart uploads. #595
object: Solve the problem that back-end report “NotExistErr” error when uploading files with the same key in parallel. #685
fuse: Evict inode cache when dealing with forget. #523
followerRead a client specific option. #382
dentry-cache to client option. #453
iget error due to metapartition split. #446
data partition and meta partition. #467
df. commit
reservedSpace parameter invalid when gt 30GB. commit
dentry. commit
opResponseLoadPartition removes duplicate locks. commit
loadSnapshotSign command about inodeCount and dentryCount. commit
rack to cell. commit
MaxInode and DentryCnt. commit
The initial release version.
Вы можете оставить комментарий после Вход в систему
Неприемлемый контент может быть отображен здесь и не будет показан на странице. Вы можете проверить и изменить его с помощью соответствующей функции редактирования.
Если вы подтверждаете, что содержание не содержит непристойной лексики/перенаправления на рекламу/насилия/вульгарной порнографии/нарушений/пиратства/ложного/незначительного или незаконного контента, связанного с национальными законами и предписаниями, вы можете нажать «Отправить» для подачи апелляции, и мы обработаем ее как можно скорее.
Опубликовать ( 0 )