Deploy a Geographically Redundant Replica Set

Replica sets achieve high availability (HA) through failover. However HA does not imply the replica set remains writeable in all cases. Particularly with a 2 DC setup there is a chance the replica set will become read-only if it cannot elect a primary depending on which DC is unavailable. All of this is and more is described in mongodb documentation.

Please have a look: https://docs.mongodb.com/manual/core/replica-set-architecture-geographically-distributed/

What is the best way configuring the 5 node replica set across two data centers if one data center down immediately we need to move the replica primary to secondary data center

The failover case should be automatic with no manual intervention however if the DC with the majority nodes goes down then the remaining DC will be read-only. To overcome this the replica set in the read-only DC can be reconfigured to replace all unavailable nodes with an arbiter.

The following document describes the steps required: https://docs.mongodb.com/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

Another possibility would be to replace one of the nodes in the DC with the most nodes with an Arbiter node in a separate DC (such as Cloud Azure/AWS) which has visibility to the other two DCs. During a failover the Arbiter would break the possible tie and keep the replica set writeable regardless of which DC goes down. Moreover, the Arbiter node need not be a powerful host as it stores no data.
The following document describes the arbiter node and details the steps required to add it to you replica set:https://docs.mongodb.com/manual/tutorial/add-replica-set-arbiter/

A variation on the above which increases the availability of the cluster is to increase your replica set count to 7 nodes and distribute them as follows:

1. DC1: P, S, S
2. DC2: S, S, S
3. DC3: A

Recommendations to maintain a writable replica set (order of preference)

1. Increase replica set to 7 nodes with an Arbiter in a third DC.
2. Replace one of the nodes with an Arbiter node on a low-end host in a separate DC with visibility to the other two DCs. For example: DC1 (P,S) DC2 (S,S) DC3(A) (PSSSA 5 node replica set)
3. Reconfigure the replica set to remove all unavailable nodes and introduce an arbiter in the remaining DC at the time of failure. For example: DC1(P,S,S) DC2(S,S) (PSSSS 5 node replica set). Assume DC1 is lost. Reconfigure DC2 to remove all nodes from DC1 and add an arbiter node to DC2 so a new primary can be elected.

Distributing replica set members across geographically distinct  data centers adds redundancy and provides fault tolerance if one of the data centers is unavailable.

There are different approaches are  there two configure the replica sets across the geographically  data centers to achieve the HA, Based on BCP plan we need to consider the suitable setup for our environment.

To achieve the maximum availability of the databases  we recommended to set up the mongodb replica set  across three  geographical data centers, below are the replica set configuration.

OUR GOALS: 

1) Database should be up and writable even if a complete datacenter goes down
2) Database failover should be automatic in case of server/datacenter failure
3) A single server failure should not cause the primary to switch to a different datacenter

Five-member Replica Set configurations 

For a replica set with 5 members, some possible distributions of members include:

• Three data centers: two member to Data Center 1, two members to Data Center 2, and one member to site Data Center 3.
  • If any Data Center goes down, the replica set remains writeable as the remaining members can hold an election.

Design

In order to satisfy our goals we came up with a three datacenter design using 4+1 replica set

1) Datacenter 1: Primary (Priority 5), Secondary 0 (Primary 4)
2) Datacenter 2: Secondary 1 (Priority 3), Secondary 2(Priority 2)
3) Datacenter 3: Arbiter No data available in this node it used for election process to elect the primary node.

We place 2 full members in each of the first two datacenters and an arbiter in the third datacenter. We also configured the priority for each server so that we can control which member becomes a primary in case of server failure.

For example, the following 5 member replica set distributes its members across three data centers.

Seven-member Replica Set :

In order to achieve the high throughput (read date from secondary data center causes the network latency ) from  primary data center we proposed the below architecture .

For a replica set with 7 members, some possible distributions of members include:

• Three data centers: three member to Data Center 1, two members to Data Center 2 and two member to site Data Center 3.
  • If any Data Center goes down, the replica set remains writeable as the remaining members can hold an election.
    
• Design
  In order to satisfy our goals we came up with a three datacenter design using 5+2  replica set
  1) Datacenter 1: Primary (Priority 5), Secondary 0 (Priority 4) ,Secondary 1 (Priority 3)
  2) Datacenter 2: Secondary 2 (Priority 2), Secondary 3(Priority 2)
  3) Datacenter 3: Arbiter 1 and Arbiter 2 ( No data available in this node it used for election process to elect the primary server )
• We place 3 full members in  the  first  datacenter , two members in second datacenter and two  arbiter in the third datacenter.
•  We also configured the priority for each server so that we can control which member becomes a primary in case of server failure.