Node in the Network

Node's entire lifetime

This section describes the flow a node will follow during its entire lifetime.
When they join the network, they won't yet be a member of any section.
First,they will have to bootstrap with their proxy node, receive a RelocateInfo and attempt to join the section that this RelocateInfo is pointing to.
Once they have a full section, they will be able to operate as a full section member.

OWN_SECTION refers to this node's own section.
It is an Option<Prefix>.
While a node is being relocated, the value will be none. Once they get accepted into a section, it becomes Some(that_section_s_prefix)

This function gets called when a node just joined the Network.
At this stage, we are connected to a proxy node and they indicate to us which section we should join.

Once a node has joined a section (indicated by OWN_SECTION.is_some()), they will be able to perform as a member of that section until they are relocated away from it.
See StartTopMember graph for details.

First Create new identity with public key-pair
The node connects to a proxy with the new identity to use for joining the new section as a full node.

Once a node knows where to be relocated, they will follow this flow to become a full member of the section.
This covers resource proof from the point of view of the node being resource-proofed.
The output of this function is an Option. If we fail resource proof, it is none, which means we will have to bootstrap again. If it is Some, it contains the RelocateInfo we need to join this section.
See JoiningRelocateCandidate graph for details.

graph TB Start --> LoopStart style Start fill:#f9f,stroke:#333,stroke-width:4px LoopStart --> HasSection HasSection(("Check?")) HasSection --"OWN_SECTION.is_none()"--> BootstrapAndRelocate BootstrapAndRelocate["BootstrapAndRelocate:
Get RelocateInfo"] BootstrapAndRelocate --> ReBootstrapWithNewIdentity HasSection --"OWN_SECTION.is_some()"--> StartTopMember StartTopMember["StartTopMember
"] style StartTopMember fill:#f9f,stroke:#333,stroke-width:4px StartTopMember --> ReBootstrapWithNewIdentity ReBootstrapWithNewIdentity["Rebootstrap
with new relocated identity
output: RelocateInfo"] style ReBootstrapWithNewIdentity fill:#f9f,stroke:#333,stroke-width:4px ReBootstrapWithNewIdentity --> JoiningRelocateCandidate JoiningRelocateCandidate["JoiningRelocateCandidate(RelocateInfo)
output: JoiningApproved"] style JoiningRelocateCandidate fill:#f9f,stroke:#333,stroke-width:4px SetNodeApproaval["OWN_SECTION=JoiningApproved"] JoiningRelocateCandidate --> SetNodeApproaval SetNodeApproaval --> LoopEnd LoopEnd --> LoopStart

Node as a member of a section

Once a node has joined a section, they need to be ready to take on multiple roles simultaneously:

All of these flows are happening simultaneously, but they share a common event loop. At any time, either all flows or all but one flows must be in a "wait" state.
If our section decides to relocates us, we will have to stop functioning as a member of our section and go back to the previous flow where we will "Rebootstrap" so we can become a member of a different section.

graph TB style StartTopMember fill:#f9f,stroke:#333,stroke-width:4px style EndRoutine fill:#f9f,stroke:#333,stroke-width:4px StartTopMember --> InitializeNodeInternalState InitializeNodeInternalState["intialise_node_internal_state()

(Parsed, Routing table...)"] InitializeNodeInternalState --> ConcurentStartElder ConcurentStartElder{"Concurrent
start elder"} ConcurentStartElder --> StartTopLevelDst style StartTopLevelDst fill:#f9f,stroke:#333,stroke-width:4px ConcurentStartElder --> StartTopLevelSrc style StartTopLevelSrc fill:#f9f,stroke:#333,stroke-width:4px ConcurentStartElder --> StartCheckAndProcessElderChange style StartCheckAndProcessElderChange fill:#f9f,stroke:#333,stroke-width:4px ConcurentStartElder --> WaitFor WaitFor(("Wait for 0:")) Rpc((RPC)) WaitFor --> Rpc Rpc -- "RelocatedInfo RPC"--> EndRoutine EndRoutine["EndRoutine: Kill all sub routines"]

Destination section

As a member of a section, our section will sometimes receive a node that is being relocated. These diagrams are from the point of view of one of the nodes in the section, doing its part to handle the node that is trying to relocate to this section.

Deciding when to accept an incoming relocation

This flow represents what we do when a section contacts us to relocate one of their nodes to our section.
The process starts as we receive an ExpectCandidate RPC from this node.
We vote for it in PARSEC to be sure all members of the section process it in the same order.
Once it reaches consensus, we are ready to process that candidate by letting them perform the resource_proof (see AcceptAsCandidate).
There are some subtleties, such as the fact that we only want to process one candidate at a time, but this is the general idea.

We receive this RPC from a section that wants to relocate a node to our section.
The node is not communicating with us yet, only once we sent RelocateResponse RPC to the originating section.
On receiving it, we vote for ParsecExpectCandidate to process it in the same order as other members of our section.
It kickstarts the entire chain of events in this diagram.
Note that we could also see consensus on ParsecExpectCandidate before we ourselves voted for it in PARSEC, as long as enough members of our section did.

These RPCs should generally be handled during resource proof, in AcceptAsCandidate.
It could be that they reach us before we are ready to receive them. For instance, because although enough other nodes got consensus on ParsecExpectCandidate, I'm still a bit behind.
It's OK to discard the RPC in this case as the node who aims to be relocated here will perceive this as a timeout from our end and resend later.

These consensus outputs were voted for in AcceptAsCandidate.
The same node could be accepted by some nodes who would vote ParsecOnline, but also time out for some other nodes who would vote for ParsecPurgeCandidate.
If it's the case, we only want to process the first of these two events and discard the other one.
Because the "Wait for 2" loop in AcceptAsCandidate has higher precendence than the "Wait for 1" loop in this one, this is exactly what will happen.

If we know of a section that has a shorter prefix than ours, we prefer for them to receive this incoming node rather than ourselves as it will help keep the Network's sections tree balanced.
This shorter_prefix_section is a function that will return None if we are the shortest of any section we know, Some if there is a better candidate.
If it is Some, we will relay the RPC to them instead of accepting the rellocation to our own section.

We want to accept at most one incoming relocation at a time into our section.
The local variable: PROCESSING_CANDIDATE indicates whether someone is already passing the resource proof process, so we can't accept anyone else for now.
When PROCESSING_CANDIDATE is true and we reach consensus on ParsecExpectCandidate, we send a RefuseCandidate RPC to the would-be-incoming-node so they can try another section or try again later.

If we are ready to accept an incoming node and we reach consensus on ParsecExpectCandidate, we will execute this flow to make them pass resource proof.
See AcceptAsCandidate diagram for details.

graph TB Start["StartTopLevelDst:
No exit - Need Killed"] style Start fill:#f9f,stroke:#333,stroke-width:4px Start --> LoopStart LoopEnd --> LoopStart LoopStart --> WaitFor WaitFor((Wait for 1:)) WaitFor --RPC--> RPC WaitFor --Parsec
consensus--> ParsecConsensus RPC((RPC)) DiscardRPC[Discard RPC] RPC --ExpectCandidate--> VoteParsecExpectCandidate RPC --ResourceProofResponse
CandidateInfo--> DiscardRPC DiscardRPC --> LoopEnd ParsecConsensus((Parsec
consensus)) DiscardParsec["Discard
Parsec
event"] ParsecConsensus --ParsecExpectCandidate--> Balanced ParsecConsensus --ParsecOnline
ParsecPurgeCandidate
--> DiscardParsec DiscardParsec --> LoopEnd VoteParsecExpectCandidate["vote_for(
ParsecExpectCandidate)
to handle the RPC consistently"] VoteParsecExpectCandidate --> LoopEnd Balanced(("Check?
(shared state)")) Balanced -- "shorter_prefix_section(
).is_some()" --> Resend Balanced -- "shorter_prefix_section().is_none()
PROCESSING_CANDIDATE==yes" --> SendRefuse Balanced -- "shorter_prefix_section().is_none()
PROCESSING_CANDIDATE==No" --> SetCandidateYes Resend["send_rpc(
ExpectCandidate)
to shorter_prefix_section()"] Resend --> LoopEnd SendRefuse["send_rpc(
RefuseCandidate)
to source section"] SendRefuse --> LoopEnd SetCandidateYes["PROCESSING_CANDIDATE=yes
(shared state)"] SetCandidateYes-->Concurrent0 Concurrent0{"Concurrent
paths"} Concurrent0 --> AcceptAsCandidate Concurrent0 --> LoopEnd AcceptAsCandidate["AcceptAsCandidate
(shared state)"] style AcceptAsCandidate fill:#f9f,stroke:#333,stroke-width:4px AcceptAsCandidate --> SetCandidateNo SetCandidateNo["PROCESSING_CANDIDATE=no
(shared state)"] SetCandidateNo-->LoopEnd

Resource proof from a destination's point of view

In the previous diagram, we ensured an incoming candidate would only be processed for resource_proof if we are the best fit we know and we are not currently processing another node.
This leads us here: to the resource proof.
We will first add this node to our peer_list with the sate ResourceProof and send a RelocateResponse to the source section.
This will let them know we are ready to process them.
They will then send their CandidateInfo to everyone in our section.
On receiving a valid CandidateInfo, I will send them a ResourceProof RPC. This gives them the "problem to solve". As they solve it, they will send me ResourceProofResponses. These will be parts of the proof. On receiving valid parts, I must send a ResourceProofReceipt. Once they finally send me the last valid part, they passed their resource proof and I vote for ParsecOnline (essentially accepting them as a member of my section).
At any time during this process, they may timeout, in which case I will decided to reject them and vote for ParsecPurgeCandidate.
This process ends once I reach consensus on either accepting or the candidate (ParsecOnline) or refusing them (ParsecPurgeCandidate).
These payloads are Opaque, which means that consensus may be reached without a quorum on either of these. For that reason, it is possible that both reach consensus, in which case the second one will be discarded in the flow above. It won't cause issues as consistency is the only property that matters here: if we accept someone who then went Offline, we will be able to detect they are Offline later with the standard Offline detection mechanism.

We only want to receive CandidateInfo once and we can't verify a ResourceProofResponse without knowing the CandidateInfo.
The local variable: GOT_CANDIDATE_INFO helps us stay on top of this.

Once we've voted a node online, we don't care to handle further ResourceProofResponses from them.
This local variable helps us with this.

graph TB AcceptAsCandidate["Accept As Candidate
Shared state all peers proceed"] EndRoutine["End of AcceptAsCandidate
(shared state)"] style AcceptAsCandidate fill:#f9f,stroke:#333,stroke-width:4px style EndRoutine fill:#f9f,stroke:#333,stroke-width:4px AcceptAsCandidate --> SendRelocateResponse SendRelocateResponse["add_node(NodeState=ResourceProof)

send_rpc(RelocateResponse)
to source section

schedule(TimeoutAccept)"] WaitFor(("Wait for 2:

Only current
candidate
events")) VoteParsecPurgeCandidate["vote_for(
ParsecPurgeCandidate)"] VoteParsecOnline["vote_for(
NetworkEvent::Online)

VOTED_ONLINE=yes"] RequestRP["send_rpc(
ResourceProof)
to candidate

GOT_CANDIDATE_INFO=yes"] SendProofReceit["send_rpc(
ResourceProofReceipt)
for proof"] SendRelocateResponse --> LoopStart LoopStart-->WaitFor WaitFor -- Consensus--> Consensus Consensus((Consensus)) Consensus -- ParsecPurgeCandidate
consensused --> RemoveNode Consensus -- ParsecOnline
consensused --> MakeOnline MakeOnline["set_node_state(
Candidate,
OnlineState)

send_rpc(
NodeApproval)"] RemoveNode["purge_node_info(
candidate node)"] RemoveNode --> EndRoutine MakeOnline --> EndRoutine WaitFor -- ResourceProofResponse

GOT_CANDIDATE_INFO==yes
VOTED_ONLINE==no --> ProofResponse((Proof)) ProofResponse -- Valid Part --> SendProofReceit ProofResponse -- Valid End --> VoteParsecOnline WaitFor -- CandidateInfo

GOT_CANDIDATE_INFO==no --> Info((Info)) Info -- Valid
CandidateInfo --> RequestRP Info -- Invalid
CandidateInfo --> VoteParsecPurgeCandidate WaitFor -- TimeoutAccept
expire --> VoteParsecPurgeCandidate RequestRP --> LoopEnd SendProofReceit-->LoopEnd VoteParsecOnline --> LoopEnd VoteParsecPurgeCandidate --> LoopEnd LoopEnd --> LoopStart

Source section

As members of a section, each node must keep track of how many "work units" other nodes have performed.
Once a node has done accumulated enough work units to gain age, the section must work together to relocate that node to a new section where they can become 1 age unit older.
These diagrams detail how this happens.

Deciding that a member of our section should be relocated away

In these diagrams, we are modeling the simple version of node ageing that we decided to implement for Fleming:
Work units are incremented for all nodes in the section every time a timeout reaches consenus. Because a quorum of elders must have voted for this timeout, the malicious nodes can't arbitrarily speed up the ageing of their nodes.
Once a node has accumulated enough work units to be relocated, if they're a non-elder adult node, we will enter TryRelocating to attempt relocating them away from our section. If they are an elder node, we will first wait for the Adult/Elder promotion/demotion flow to kick in and demote them to an adult. This will happen because we changed their state from Online to Relocating, which means that they will get demoted.
Also of note: we will only be actively relocating at most one node away from our section at a time.

In the context of Fleming, nodes (especially adults) aren't doing meaningful work such as handling data.
As a proxy, we use a time based metric to estimate how much work nodes have done (i.e: how long they remained Online and responsive).
A local timeout wouldn't do here as it would allow malicious nodes to artificially age nodes in their sections faster. However, by reaching quorum on the fact a timeout happened, we ensure that at least one honest node has voted for the timeout.
All nodes start the WorkUnitTimeout. On expiry, they vote for a WorkUnitIncrement in PARSEC and restart the timer.

This function increments the number of work units for all members of my peer_list (remember that n_work_units is a member of the PeerState struct).

RELOCATING is an Option<PublicId>. It represents the node that we decided to relocate next, if any. It could be an elder, but if it is, we will only change the state of that node rather than immediately relocate it.
If we reach consensus on a WorkUnitIncrement while RELOCATING.is_some() - or in other words, while we are already relocating a node away from this section - we simply discard the PARSEC event. It will be dealt with later, after at least one more WorkUnitTimeout.

This function will return the best candidate for relocation, if any.
First, if any member of our peer_list has the state: RelocatingState, Some(them) will be returned.
This helps ensure we aren't relocating multiple nodes at the same time if we have to wait for an elder to be demoted.
If none of the nodes in our peer_list have that state, it can return for instance the node with the largest number of work units for which the number of work units is greater than 2^age.

This function mutates our peer_list to set the state: RelocatingState for the node: RELOCATING.
inputs:
- RELOCATING
- state
side-effect:
- mutates peer_list

The flow a section follow to relocate one of its adult nodes away from itself.
See next diagram for details.

Exactly one of these RPCs will be sent to us from the destination section as a response to our section's ExpectCandidate RPC (see TryRelocating for more context).
These must be handled here as it is possible that one of the nodes sees this event reach consensus before they were aware that an ExpectCandidate RPC was sent from our section to the destination.
When this happens, we will immediately vote for it in PARSEC as we need to act in the same order as anyone else in our section.

Exactly one of these events should reach consensus, and it must be handled in TryRelocating as the "Wait" in there has a higher priority than the one in this flow.
If we ever see consensus for one of these here, we know there is a bug; so we can panic.

graph TB Start["StartTopLevelSrc:
No exit - Need Killed"] style Start fill:#f9f,stroke:#333,stroke-width:4px Start --> StartWorkUnitTimeOut StartWorkUnitTimeOut["schedule(WorkUnitTimeOut)"] StartWorkUnitTimeOut --> LoopStart LoopEnd --> LoopStart LoopStart --> WaitFor WaitFor((Wait for 3:)) WaitFor --Event--> Event WaitFor --RPC--> RPC WaitFor --Parsec
consensus--> ParsecConsensus Event((Event)) Event -- WorkUnitTimeOut
Trigger --> VoteParsecRelocationTrigger VoteParsecRelocationTrigger["vote_for(WorkUnitIncrement)
schedule(WorkUnitTimeOut)"] VoteParsecRelocationTrigger --> LoopEnd RPC((RPC)) RPC --RefuseCandidate--> VoteParsecRefuseCandidate RPC --RelocateResponse--> VoteParsecRelocateResponse VoteParsecRefuseCandidate["vote_for(
ParsecRefuseCandidate)
to handle the RPC consistently"] VoteParsecRefuseCandidate --> LoopEnd VoteParsecRelocateResponse["vote_for(
ParsecRelocateResponse)
to handle the RPC consistently"] VoteParsecRelocateResponse --> LoopEnd ParsecConsensus((Parsec
consensus)) DiscardParsec["Discard
Parsec
event"] Bug["BUG: assert fail:

Section can only send either
one of these RPC, and only for
the candidate we are relocating"] ParsecConsensus -- WorkUnitIncrement consensused --> IncrementWorkUnit IncrementWorkUnit["increment_nodes_work_units()"] IncrementWorkUnit-->AlreadyRelocating AlreadyRelocating(("Check?")) AlreadyRelocating --"RELOCATING.is_none()"--> SetRelocating AlreadyRelocating --"RELOCATING.is_some()"--> DiscardParsec ParsecConsensus --"Any:

ParsecRefuseCandidate

ParsecRelocateResponse"--> Bug Bug DiscardParsec --> LoopEnd SetRelocating["RELOCATING=Some(get_node_to_relocate())"] SetRelocating --> SetRelocatingNodeState SetRelocatingNodeState["set_node_state(RELOCATING, RelocatingState)"] SetRelocatingNodeState --> isRelocatingElder isRelocatingElder(("Check?")) isRelocatingElder--"is_elder(RELOCATING)"-->ResetRelocating isRelocatingElder--"!is_elder(RELOCATING)"-->Concurrent0 Concurrent0{"Concurrent
paths"} Concurrent0 --> TryRelocating Concurrent0 --> LoopEnd TryRelocating["TryRelocating
(shared state)"] style TryRelocating fill:#f9f,stroke:#333,stroke-width:4px TryRelocating-->ResetRelocating ResetRelocating["RELOCATING=None
(shared state)"] ResetRelocating --> LoopEnd

Relocating a member of our section away from it

At this stage, we have decided that one of our section members should be relocated. The process is quite simple: we send an ExpectCandidate RPC to the destination section. That section will eventually either pass the node to a section with a shorter prefix, send us a RefuseCandidate RPC or eventually send us a RelocateResponse RPC. If the destination picked another section with a shorter prefix, that section will send us one of these RPCs (or suggest another section with a shorter prefix, but this recursion will end at some point).
All in all, we have the certainty that eventually, exactly one of these two RPCs will make it to our section.
When it does, it will be voted in PARSEC, regardless of the order of operations (see previous diagram), so it will eventually make it through PARSEC.
If they refused our candidate, our job ends there: the candidate will simply be a candidate for relocation again later, so we will try to relocate it then.
If they accepted the node and sent us a RelocateResponse RPC, so the ParsecRelocateResponse event reached consensus, we will send to the node that is being relocated the RelocatedInfo the will need.
At this point, we may purge their information since this node isn't a member of our section anymore.

graph TB TryRelocating["TryRelocating
Shared state all peers proceed"] EndRoutine["End of TryRelocating
(shared state)"] style TryRelocating fill:#f9f,stroke:#333,stroke-width:4px style EndRoutine fill:#f9f,stroke:#333,stroke-width:4px TryRelocating --> SendExpectCandidate SendExpectCandidate["send_rpc(
ExpectCandidate)"] SendExpectCandidate --> WaitFor WaitFor((Wait for 4:

Only current
candidate
events)) WaitFor --Parsec
consensus--> Consensus Consensus((Consensus)) Consensus --"ParsecRefuseCandidate"--> EndRoutine Consensus --"ParsecRelocateResponse"--> SendPovableRelocateInfo SendPovableRelocateInfo["send_rpc(RelocatedInfo)
to node

Includes RelocationResponse
content and consensus
Node may be already gone"] SendPovableRelocateInfo-->PurgeNodeInfos PurgeNodeInfos["purge_node_info(
node)"] PurgeNodeInfos--> EndRoutine

Elder-only

Process for Adult/Elder promotion and demotion including merge

This flow updates the elder status of our section nodes if needed.
Because it is interlinked, it also handles merging section: When merging, no elder change can happen.
However other flows continue, so relocating to and from the section is uninterupted:
We have to be careful that the section follows up on relocation once merged, so we may want to avoid active relocation when merging.

As for incrementing work units, we want to update the eldership status of all nodes in a section on a syncrhonised, regular basis.
For this reason, it makes sense to have a timer going through Parsec.
Note that this timer has to be only as fast as needed so that it remains highly unlikely that 1/3 of the elders in any section would go offline within one timer's duration.

A section sends a Merge RPC to their neighbour section when they are ready to merge at the given SectionInfo digest.
In this flow, we handle both situations:

We vote for this Parsec event on receiving a Merge RPC from our neighbour section.
It contains the information about them that we need for merging. When this PARSEC event reaches consensus in PARSEC, we store that information by calling store_merge_infos.

This function is used to store the merge information from a neighbour section locally.
Once it has been stored, has_merge_infos will return true and we will be ready to enter the ProcessMerge flow.

This function indicates that we received sufficient information from our neighbour section needing a merge, and reached consensus on it.
We are ready to start the merging process with them.

This function indicates that we need merging (as opposed to a merge triggered by our neighbour's needs).
The details for the trigger are still in slight flux, but here are some possibilities:

If any of our elders is not Online, they must be demoted to a plain old adult.
If this happens, the oldest adult must be promoted to the elder state as a replacement.
Alternatively, if any of our Online adult nodes is older than any of our elders, the youngest elder must be demoted and this adult must be promoted.
Note that elder changes are only processed when the section is not in the middle of handling a merge.

graph TB CheckAndProcessElderChange["StartCheckAndProcessElderMergeChange:
No exit - Need Killed"] style CheckAndProcessElderChange fill:#f9f,stroke:#333,stroke-width:4px CheckAndProcessElderChange --> StartCheckElderTimeout StartCheckElderTimeout["schedule(
CheckElderTimeout)"] StartCheckElderTimeout --> LoopStart WaitFor(("Wait for 5:")) LoopStart --> WaitFor WaitFor -- Event --> Event Event((Event)) Event-- CheckElder
Timeout--> VoteCheckElderTimeout VoteCheckElderTimeout["vote_for(
ParsecCheckElderTimeout)"] VoteCheckElderTimeout--> LoopEnd RPC((RPC)) WaitFor -- RPC --> RPC RPC --Merge--> VoteParsecNeighbourMerge VoteParsecNeighbourMerge["vote_for(
ParsecNeighbourMerge)"] VoteParsecNeighbourMerge --> LoopEnd Consensus((Consensus)) WaitFor-- Parsec
consensus --> Consensus Consensus -- "ParsecNeighbourMerge" --> SetNeighbourMerge SetNeighbourMerge["store_merge_infos(ParsecNeighbourMerge info)"] SetNeighbourMerge-->LoopEnd Consensus--"ParsecCheckElderTimeout"-->CheckMergeNeeded CheckMergeNeeded(("Check")) CheckMergeNeeded--"!merge_needed()
and
!has_merge_infos()"-->CheckElderChange CheckElderChange(("Check")) CheckElderChange --"check_elder_change()

Has elder changes: elder first ordered by:
State=Online then age then name."--> Concurrent0 Concurrent0{"Concurrent
paths"} Concurrent0 --> ProcessElderChange Concurrent0 --> LoopEnd ProcessElderChange["ProcessElderChange(changes)"] style ProcessElderChange fill:#f9f,stroke:#333,stroke-width:4px ProcessElderChange -->RestartTimeout CheckElderChange -- "No
changes" --> RestartTimeout RestartTimeout["schedule(
CheckElderTimeout)"] RestartTimeout-->LoopEnd CheckMergeNeeded --"merge_needed()
or
has_merge_infos()"-->Concurrent1 Concurrent1{"Concurrent
paths"} Concurrent1 --> ProcessMerge Concurrent1 --> LoopEnd ProcessMerge["ProcessMerge"] style ProcessMerge fill:#f9f,stroke:#333,stroke-width:4px ProcessMerge --> RestartTimeout LoopEnd --> LoopStart

Process Adult/Elder promotion and demotion needed from last check

Vote for Add for new elders, Remove for no longer elders and NewSectionInfo
This handles any change, it does not care whether one or all elders are changed, this is decided by the calling function.

graph TB ProcessElderChange["ProcessElderChange
(Take elder changes)"] style ProcessElderChange fill:#f9f,stroke:#333,stroke-width:4px EndRoutine["End of ProcessElderChange
(shared state)"] style EndRoutine fill:#f9f,stroke:#333,stroke-width:4px ProcessElderChange --> MarkAndVoteSwapNewElder MarkAndVoteSwapNewElder["vote_for(Add) for new elders
vote_for(Remove) for now adults nodes
vote_for(NewSectionInfo)

WAITED_VOTES.insert(all votes)"] MarkAndVoteSwapNewElder --> LoopStart WaitFor(("Wait for 5:")) LoopStart --> WaitFor Consensus((Consensus)) WaitFor-- Parsec
consensus --> Consensus Consensus -- "WAITED_VOTES.contains(vote)" --> OneVoteConsensused OneVoteConsensused["WAITED_VOTES.remove(vote)"] OneVoteConsensused --> WaitComplete WaitComplete(("Check?")) WaitComplete--"WAITED_VOTES
.is_empty()
(Wait complete)"-->MarkNewElderAdults MarkNewElderAdults["For all nodes:
set_elder_status(node, is_elder)"] MarkNewElderAdults--> EndRoutine WaitComplete--"!WAITED_VOTES
.is_empty()
(Wait not complete)"--> LoopEnd LoopEnd --> LoopStart

ProcessMerge Sub-routine

Vote for ParsecOurMerge, and take over handling any ParsecNeighbourMerge.
Complete when one merge has completed, and a NewSectionInfo is consensused.
If Multi stage merges are required, they will require calling this function again

Store the neighbour's merge info, may not be sibling in case of multi merge

Did we store the neighbour's merge info for our sibling

Remove the stored sibling's merge info and return the NewSectionInfo.

graph TB ProcessMerge["ProcessMerge
"] EndRoutine["End of ProcessMerge
(shared state)"] style ProcessMerge fill:#f9f,stroke:#333,stroke-width:4px style EndRoutine fill:#f9f,stroke:#333,stroke-width:4px ProcessMerge --> VoteOurMerge VoteOurMerge["vote_for(
ParsecOurMerge)"] VoteOurMerge --> LoopStart WaitFor(("Wait for 5:")) LoopStart --> WaitFor Consensus((Consensus)) WaitFor-- Parsec
consensus --> Consensus Consensus -- "NewSectionInfo" --> CompleteMerge CompleteMerge["Complete_merge()
(Start parsec with new genesis...)"] CompleteMerge --> MarkNewElderAdults MarkNewElderAdults["For all nodes:
set_elder_status(node, is_elder)
based on new section info"] MarkNewElderAdults--> EndRoutine Consensus -- "ParsecNeighbourMerge" --> SetNeighbourMerge SetNeighbourMerge["store_merge_infos(ParsecNeighbourMerge info)"] SetNeighbourMerge --> CheckMerge CheckMerge((Check)) CheckMerge -- "OUR_MERGE
and
has_sibling_merge_info()" --> VotForNewSectionInfo VotForNewSectionInfo["OUR_MERGE=false
merge_sibling_info_to_new_section()
vote_for(
NewSectionInfo)"] VotForNewSectionInfo--> LoopEnd CheckMerge -- "!OUR_MERGE
or
!has_sibling_merge_info()" --> LoopEnd Consensus--"ParsecOurMerge"-->SendMergeRpc SendMergeRpc["OUR_MERGE=true
send_rpc(Merge)"] SendMergeRpc -->CheckMerge LoopEnd --> LoopStart

CheckOnlineOffline Sub-routine

graph TB CheckOnlineOffline["CheckOnlineOffline:
No exit - Need Killed"] style CheckOnlineOffline fill:#f9f,stroke:#333,stroke-width:4px CheckOnlineOffline --> LoopStart WaitFor(("Wait for 5:")) LoopStart --> WaitFor LocalEvent((Local
Event)) WaitFor --event--> LocalEvent LocalEvent -- Node detected offline --> VoteNodeOffline VoteNodeOffline["vote_for(
ParsecOffline)"] VoteNodeOffline --> LoopEnd LocalEvent -- Node detected back online --> VoteNodeBackOnline VoteNodeBackOnline["vote_for(
ParsecBackOnline)"] VoteNodeBackOnline --> LoopEnd Consensus((Consensus)) WaitFor-- Parsec
consensus --> Consensus Consensus--"ParsecOffline"-->SetOfflineState SetOfflineState["set_node_state(
node,
OfflineState)"] SetOfflineState -->LoopEnd Consensus -- "ParsecBackOnline" --> SetRelocating SetRelocating["set_node_state(
node,
RelocatingState)"] SetRelocating --> LoopEnd LoopEnd --> LoopStart

Elders and adults

Becoming a full member of a section

This is from the point of view of a node trying to join a section as a full member.
This node is going to perform the resource proof until it receive a NodeApproval RPC to complete this stage successfully.
If the node is not accepted, after time out, it will try another section.
graph TB JoiningRelocateCandidate --> InitialSendConnectionInfoRequest JoiningRelocateCandidate["JoiningRelocateCandidate
(Take destination section nodes)"] style JoiningRelocateCandidate fill:#f9f,stroke:#333,stroke-width:4px EndRoutine["End of JoiningRelocateCandidate
"] style EndRoutine fill:#f9f,stroke:#333,stroke-width:4px InitialSendConnectionInfoRequest["For all nodes in destination section:
MembersProofRequest(node) = None
and send_rpc(ConnectionInfoRequest)

schedule(TimeoutResendInfo)
schedule(TimeoutRefused)"] InitialSendConnectionInfoRequest-->LoopStart LoopStart --> WaitFor WaitFor(("Wait for 6:")) LocalEvent((Local
Event)) WaitFor --> LocalEvent LocalEvent -- ResourceProofForElderReady --> SendFirstResourceProofPartForElder SendFirstResourceProofPartForElder["send_rpc(
ResourceProofResponse)
with first part for elder"] SendFirstResourceProofPartForElder --> LoopEnd LocalEvent--"TimeoutResendInfo triggered
Some elders have not responsed
with ResourceProof"--> ResendConnectionInfoRequest ResendConnectionInfoRequest["For all nodes with
MembersProofRequest(node).is_none():
send_rpc(ConnectionInfoRequest)

schedule(TimeoutResendInfo)"] ResendConnectionInfoRequest --> LoopEnd LocalEvent--"TimeoutRefused
triggered"--> EndRoutine Rpc((RPC)) WaitFor --> Rpc Rpc -- NodeApproval --> EndRoutine Rpc -- ConnectionInfoResponse --> ConnectAndSendCandidateInfo ConnectAndSendCandidateInfo["Connect and send
CandidateInfo"] ConnectAndSendCandidateInfo-->LoopEnd Rpc -- ResourceProofReceipt --> SendNextResourceProofPartForElder SendNextResourceProofPartForElder["send_rpc(
ResourceProofResponse)
part for elder"] SendNextResourceProofPartForElder --> LoopEnd Rpc -- ResourceProof --> StartComputeResourceProofForElder StartComputeResourceProofForElder["start_compute_resource_proof(source elder)"] StartComputeResourceProofForElder --> LoopEnd Rpc -- "ExpectCandidate
RefuseCandidate
RelocateResponse
..." --> VoteParsecRPC VoteParsecRPC["vote_for(the parsec rpc)
(cache for later)"] VoteParsecRPC --> LoopEnd LoopEnd --> LoopStart

Node relocation overview

Original flow

sequenceDiagram Src->>+Dst: ExpectCandidate Dst-->>-Src: RelocateResponse Src->>Node: RelocatedInfo Node->>+Dst: ConnectionInfoRequest Dst-->>-Node: ConnectionInfoResponse Node->>Dst: CandidateInfo Dst->>Node: ResourceProof loop ResProof Node->>Dst: ResourceProofResponse Dst->>Node: ResourceProofReceipt end Dst->>Node: NodeApproval