Use pam_access to restrict SSH connection

# Open file /etc/security/access.conf
vi /etc/security/access.conf

# Append following line:
-: ALL EXCEPT root username:123.123.123.123
# only accept login access from root/username from IP address 123.123.123.123

# Save the file and Open /etc/pam.d/sshd file :
vi /etc/pam.d/sshd

# Append following entry
account required pam_access.so


Other examples:

-:root:ALL EXCEPT LOCAL
# allow any users except root from anywhere, and root only from localhost.
-:root:ALL EXCEPT 123.123.123.123
# allow any users except root from anywhere, and root only from 123.123.123.123.

-:ALL EXCEPT root username:ALL
# Deny network and local login to all users except for user root and username

+ : root : 192.168.1.0/24
# Only allow root user login from 192.168.1.0/24 network:


Linux - SSH basic security setting